Wednesday, December 6, 2006

The Blazing Trail of Open Source Development

It's often said that open source Latest News about open source doesn't innovate. It imitates. That's certainly what the proprietary software industry would have you believe. Looking at the activity in some of the most prominent open source projects in use in enterprises today, it's tempting to agree.

For example, although open source databases are incredibly popular for all kinds of mission-critical applications, neither MySQL or PostgreSQL is really doing anything that IBM (NYSE: IBM) Latest News about IBM, Microsoft (Nasdaq: MSFT) Latest News about Microsoft, Oracle (Nasdaq: ORCL) Latest News about Oracle, and Sybase (NYSE: SY) Latest News about Sybase haven't been doing for years. Similarly, the productivity suite is an impressive example of community-driven development, and yet its only real purpose is to create a free, standards-based clone of Microsoft Office. Even Linux itself is an attempt to rewrite Unix as free software.

Apache Leads the Way

None of this discounts the fact that open source has blazed a trail or two of its own rather than simply following the pack. The Apache Web server is a prime example. Apache has been the Web server of choice since it was forked from the National Center for Supercomputing Applications' httpd in 1995. According to Netcraft, the Web site and server barometer, 62 percent of all Internet Free How-To Guide for Small Business Web Strategies - from domain name selection to site promotion. Web sites run Apache today, compared with 31 percent running Microsoft IIS (Internet Information Server).

As Microsoft's Internet Explorer demonstrates, however, mere popularity isn't a measure of first-rate performance. Apache is most often the default Web server installed on most Linux servers, but its success isn't simply due to familiarity or lack of competition. Rather, it offers stability, high performance, a stellar security record, and an impressive array of features and extensions that give it many more capabilities than its commercial competitors.

Apache effectively created the market for Web server software and continues to lead the way. In five years' time, IIS will no doubt be around in one form or another, but Apache will still lead the pack, pushing the envelope of what a Web server can do.

So was Apache a fluke? Is it the lone case where the open source community was able to anticipate demand for a product before proprietary vendors could implement it? Hardly.

Across the software industry, countless developers, individuals and companies are experimenting with open source methods. One reason is that community-driven development allows a software product to grow organically. As Eric S. Raymond observed in his seminal work, The Cathedral and the Bazaar, "Every good work of software starts by scratching a developer's personal itch." When a group of developers begins to collaborate in an open fashion, each one scratching a unique personal itch, the result is software that expands to fill those functional areas not addressed by proprietary offerings.

Examples of this kind of collaborative innovation can be found across the IT landscape. To prove it, we marshaled a handful of essayists to showcase the abundant examples of how open source software is not only matching the capabilities of proprietary software, but going it one better.

Open Source Breathes Life Into Java

There can be no doubt that open source has been a tremendous boon to Java Latest News about Java. The JCP (Java Community Process), by which the Java language and platform moves ahead, seems to inch forward at a glacial pace. Committee review and approval are slow, thoughtful processes, but they're conducted at a pace that cannot be wholly condemned. Java, after all, is the leading platform for enterprise applications and, as such, it should evolve slowly, even when needs are pressing. Resolving one set of problems by creating another is never a good solution.

At the other end of the spectrum, however, is the open source community, which has been galloping ahead with numerous important innovations. The fundamental tools of enterprise Java today are all open source: Ant (for building applications), Hibernate (for persisting data to disk), JUnit (for running unit tests), and Maven (for continuous integration); not to mention the highly regarded Eclipse and NetBeans development environments and the PMD source code validator.

Java developers also benefit from several open source frameworks, such as JSF (JavaServer Faces), Spring, and Struts. Then there are the numerous containers: Apache Tomcat, Geronimo, Jetty, Jonas, and Resin -- to say nothing of many smaller containers for embedded use. Open source has been a tremendous font of imagination and productivity for the Java world.

Between these two streams -- the JCP and open source -- stand the three major providers of enterprise Java technology: BEA (Nasdaq: BEAS) Latest News about BEA Systems, IBM and Sun Microsystems (Nasdaq: SUNW) Latest News about Sun Microsystems. They all straddle the line between open source and closed tools. Although many IBM tools are proprietary, it has been a major contributor to Java itself and to the open source community. BEA has become very active of late with its "blended" strategy, in which its proprietary tools actively support open source products. In addition, BEA also has donated code and bodies to several open source projects.

Sun remains something of a holdout. Zealots have been hectoring Sun to open source Java, in hopes of accelerating innovation in the platform. Sun, however, has dragged its feet. There is some logic to its action -- or lack thereof -- namely, that opening the Java code could splinter Java into differing versions, thereby diminishing its vaunted portability, but the community has stridently attacked this stance.

This puts Sun in a position to which it is unaccustomed, in view of its long-standing contributions to open source. Sun remains the only company to have open sourced its own operating system and a complete office productivity suite. Indeed, it has open sourced many Java technologies. So, perhaps grudgingly, at LinuxWorld in August, Sun made explicit the time frame in which it intended to open the Java libraries and Java Virtual Machine, in addition to launching a community Web site where developers can track its ongoing progress.

Will all this vigorous energy in the direction of open source benefit Java in the long term? Its slow pace of development aside, part of Java's success lies in the stewardship Sun has shown. Taking that role out of Sun's hands by opening the Java source code could result in brilliant enhancements to the platform, or it might end up damaging it.

Devices Gain an Edge With Linux

Linux is finding success in much smaller devices than the servers and workstations that have traditionally been its mainstays. For embedded systems developers, the advantage of Linux over proprietary OSes lies as much in its flexibility and openness as in its low cost.

The consumer electronics market has an insatiable demand for new technologies, but developers working with proprietary embedded platforms from the likes of Microsoft and Symbian Latest News about Symbian must depend on a single vendor to deliver device drivers and support for the latest hardware. Their developer counterparts in the open source community, however, often begin tinkering with technology as soon as it's released. The code they produce may be raw, but it appears quickly, which in turn allows them to produce prototypes faster, get to market sooner, and gain first-mover advantage over their competitors.

Linux also makes it easier to build complex embedded applications. Traditional RTOS (real-time operating systems) for embedded devices are single-purpose platforms designed for running one task in a single processor thread. Embedded Linux resembles its heavyweight cousin, in that it's a general purpose OS designed to run multiple applications. This versatility helps developers to serve an increasingly sophisticated customer base that demands ever more of digital devices. What's more, the absence of complex proprietary licensing terms frees developers to customize every level of their applications, choosing the components they want while ignoring others.

Competition among vendors is inevitably good for customers. In the embedded Linux market, top vendors such as MontaVista and Wind River compete not only against proprietary OS vendors, but also against each other. In addition, some device manufacturers choose to go it alone, producing their own flavors of embedded Linux, independent of the OS vendors. Because Linux is open source, each variant becomes part of a broad ecosystem of developers, customers and partner companies, all contributing to the larger whole.

This atmosphere of "competitive collaboration" leaves still other companies free to build upon the Linux base with additional tools. For example, Trolltech offers Qtopia, a complete environment for rapid development of embedded Linux applications and user interfaces. The core Qtopia technologies are available under the GNU GPL Latest News about GPL (General Public License), giving developers the same visibility into Qtopia code that they enjoy with the Linux kernel itself. Similarly, Nokia (NYSE: NOK) Latest News about Nokia has released its own open source application platform, Maemo, which can be found on embedded devices such as the Nokia 770 Internet Tablet.

These toolkits offer an additional advantage to developers: familiarity. Qtopia is based on Trolltech's well-established Qt toolkit for desktop Linux systems; Maemo is based on similar technologies from the Gnome project. The result is that application developers can use their existing skills to build software for embedded Linux, unlike proprietary OSes.

These advantages add up to an ever-increasing range of devices powered by Linux. From Sharp PDAs Latest News about PDAs to TiVo (Nasdaq: TIVO) Latest News about TiVo set-top boxes, Linksys wireless Get the Facts on BlackBerry Business Solutions routers to Motorola (NYSE: MOT) Latest News about Motorola mobile phones, more companies are betting on Linux as their springboard to success in the fast-paced and highly competitive device market.

New Frontiers for Multimedia

It's easy to assume that source and multimedia are mutually exclusive. A common criticism of free desktop Linux distributions such as Ubuntu is that they lack support for multimedia playback, even for common formats. Don't blame the distributions' packagers, though. A maze of patents has accumulated around multimedia through the years, covering every aspect of playback and encoding. Even the MP3 format is restricted by patents that conflict with the requirements of free software licenses.

Independent developers haven't been ignoring multimedia, however -- far from it. A number of open source projects manage not only to skirt the restrictions of existing technologies, but also to improve on them.

For example, Ogg Vorbis is an open source "lossy" audio compression technology. As does MP3, it doesn't retain every bit of data but reproduces a signal that sounds more or less like the original. However, Vorbis uses advanced psychoacoustic modeling to deliver better sound quality than MP3 at a similar level of compression. What's more, it is not encumbered by patents and is completely royalty free.

For video, look no further than the Dirac project, sponsored by BBC Research. Although still experimental, Dirac uses wavelet compression, an advanced mathematical technique that, in theory, should deliver better quality video than current methods. BBC owns some patents on the Dirac code, but it has granted perpetual royalty-free licenses to anyone who wishes to use them. Similarly, On2 Technologies has granted a perpetual license to its VP3 compression codec, which has become the basis of another open source project, Theora.

While development of open source multimedia technologies proceeds apace, adoption is another matter. Vorbis is a mature codec that delivers superior performance, but few device manufacturers support it, despite the fact that it requires no licensing fees.

The problem is content. Hardware makers will support the formats that content providers offer, and, increasingly, that means formats that support DRM (digital rights management). Unfortunately, whether it's Apple (Nasdaq: AAPL) Latest News about Apple, Microsoft, Sony (NYSE: SNE) Latest News about Sony, or another company, proprietary DRM effectively makes the technology provider the gatekeeper for the entire multimedia stack. This, too, is a challenge that open source can answer.

The Open Media Commons, sponsored by Sun Microsystems, is an attempt to develop DRM technology through community participation. At its core is open source code that Sun has taken pains to ensure does not run afoul of any existing DRM patents -- an area nearly as treacherous as multimedia itself. The hope is that a fully open DRM scheme will allow customers to use DRM-restricted content in approved ways across a range of interoperable software and devices from various manufacturers.

The common theme of all these projects is that cooperation and collaboration will accelerate adoption of digital audio and video, even as they push the technologies forward. Patent restrictions and proprietary code have held back multimedia long enough; the way ahead is open source.

When 'Wide Open' Means Secure

There's a reason nearly every security appliance vendor uses open source tools, and it has little to do with licensing. The vast majority of these devices -- ranging from spam Latest News about spam and spyware filters to network scanners to intrusion detection and prevention systems -- are not only built on an open source platform such as Linux or FreeBSD, but they also actively use other open source products to accomplish their given tasks.

On any platform, the most popular network port scanner and OS fingerprinting application is Nmap -- either on its own or integrated into dozens of other applications. Nmap, an open source tool, provides a quick and accurate method of determining open ports on any given IP address or subnet; it can also determine the OS of a particular device by examining the way IP packets are constructed.

Another example is Nessus, a popular open source vulnerability scanner. Nessus can use Nmap to scan a host, but goes much further by attempting to trigger potential exploits on the target system to verify its integrity. The plugins available for Nessus number in the thousands, and more are added on a regular basis as exploits are discovered -- thanks again to the fact that the source code is readily available.

Snort, the widely used IDS (intrusion detection system), is also notable. As is Nessus, Snort is available packaged as a commercial product, but the open source version is still going strong. Other open source tools, such as OpenSSH and OpenSSL, are industry standards -- in use everywhere by thousands of different products on dozens of platforms.

Last, there's the operating system layer itself. Although no OS is truly secure, security tools offered on a Windows platform are immediately suspect due to well-documented security issues of the underlying OS. Linux, FreeBSD, NetBSD or OpenBSD-based products have a much better security track record. OpenBSD claims to have had only one remote hole in the default install in more than eight years.

To the uninitiated, the concept of open source may seem at odds with high security. The prevailing thought is that making the source code available makes it easier for hackers and virus writers to exploit bugs in the code. You only have to look at Microsoft -- one of the most staunchly proprietary software vendors around -- to see that this really isn't true; in fact, the opposite is true.

A commercial product may have a quality assurance team of a few dozen people, but most significant open source projects -- those that are served by vibrant and growing communities -- effectively have QA teams numbering in the thousands. Many of those folks know the code intimately, as they've contributed to the project. This leads to a more secure product, as bugs are exposed quickly and the fixes are subject to review by many more pairs of eyes than are available in a commercial setting. As a result, a great many security professionals view open source as being more secure than proprietary code, and this impression is finally beginning to penetrate throughout the industry.

Languages Thrive on Community Scripting

Scripting languages, sometimes called "dynamic" languages, have become all the rage, in part because they let developers get a lot of work done with comparatively little code. This "bang for the buck" derives from new approaches that push more of the work onto the compiler and runtime environment -- such as deriving a variable's type by its value -- in addition to special shortcuts for frequently performed actions.

Scripting languages, including Perl, PHP, and Python, have another aspect in common: They are almost universally open source. In many areas, the difference between open and closed source might not be important, but it appears to be critical in scripting languages. Curiously, this aspect isn't due to open source's agility in fixing flaws, but rather to its capability to create community -- that is, a group of users who rely on the language and promote it actively.

A dramatic example of the importance of open source can be seen by comparing the fates of two scripting languages loosed on the world at roughly the same time: Ruby and NetRexx. NetRexx was designed by IBM, based in part on Big Blue's successful mainframe scripting tool, Rexx. It runs interpretively or can be compiled to Java bytecodes. As a language, NetRexx is highly productive and easy to learn, with numerous elegant constructs. Because it compiles to Java bytecodes, all Java libraries can be accessed from NetRexx, plus it enjoys the performance benefits of the Java Virtual Machine.

Ruby, on the other hand, is purely interpretive; it has no virtual machine and so runs slowly. The language itself has elegant points, but much of its advantage comes from the use of nonintuitive syntactical elements, derived in part from Perl's shorthand syntax. NetRexx and Ruby have another important aspect in common: They are both available for free.

So, which of these two languages is more successful: the fast, elegant NetRexx or the less intuitive and far slower Ruby? Contrary to what one may expect, Ruby is flying high today -- NetRexx is nearly dead.

The key reason, I believe, is that Ruby is open source and NetRexx is not. The result is that Ruby was able to build an active developer community -- especially in Japan, where it was created and has been popular for a decade. Ruby gathered steam until David Heinemeier Hansson wrote "Ruby on Rails," a wildly popular Web application framework. Rails is Ruby's killer app. It is a paragon of design elegance and has guaranteed Ruby an enduring role. Meanwhile, NetRexx limps along untended and friendless, despite its great merit. Had IBM open sourced it, scripting today might be wholly different.

Open source alone is certainly no guarantor of a language's success. Indeed, there are many open source languages with no hope of success. Although open source alone is not sufficient, it is necessary. Without it -- and lacking the resources of a Microsoft to create community -- few new languages will likely ever reach the needed critical mass of users and evangelists.

More Options for Enterprise Messaging

Open source tools and e-mail share a long history. Mail servers such as Exiin, Postfix, and Sendmail enjoy widespread use, to say nothing of a healthy assortment of open source mail clients, from Mozilla Latest News about Mozilla Foundation Thunderbird to Pine. But e-mail isn't the be-all and end-all of enterprise messaging. For advanced features such as group calendaring, shared address books, and IM integration, enterprise customers typically have had just two options: IBM's Lotus Notes and Microsoft Exchange.

This is beginning to change. Although the Big Two of enterprise messaging are likely to dominate for the foreseeable future, a number of attractive alternatives have begun to appear, particularly for small to mid-size enterprises. Not surprisingly, many of these promising newcomers hail from the world of open source.

Perhaps the most ambitious such project, Open-Xchange, replicates the functionality of traditional groupware servers in an open source package. OpenXchange integrates a variety of open source tools -- including the Apache Web server, the Tomcat Java servlet engine, and the PostgreSQL database, among others -- to offer full e-mail, calendaring, and collaboration capabilities through a Web-based UI. Commercial add-ons allow Open-Xchange to interoperate with Microsoft Outlook and Palm (Nasdaq: PALM) Latest News about Palm handheld clients.

Still other open source messaging servers take interoperability a step further. Zimbra, for example, uses AJAX (Asynchronous JavaScript and XML) technology to integrate messages with outside data sources, from address book lookups to Google (Nasdaq: GOOG) Latest News about Google Maps. What's more, Zimbra was designed from the ground up as an interoperable system of components, many of which are exposed as Web services. Third-party developers can connect to these components' APIs to gain direct access to Zimbra calendar items, address books and e-mail.

Because Zimbra is open source, developers don't have to navigate around proprietary pitfalls. They can get right into the meat of the code and integrate their own software directly. Open source messaging systems also impose no legacy communications interfaces. In place of proprietary Microsoft Exchange protocols, OpenXchange and Zimbra take advantage of a suite of open standards including WebDAV, LDAP, iCal and HTTP.

The significance of this open approach to messaging is remarkable. Because companies rely so heavily on e-mail as an essential business tool, critical enterprise data ends up stored on e-mail servers. When those servers are based on proprietary software, companies are at the mercy of a single vendor to orchestrate e-mail storage, retrieval, search, archiving, backup, integration and management. A messaging system based on open protocols and open code gives customers assurance that they'll be able to access their data when they need it, over the long term. Further, active development communities will increasingly allow those systems to compete with the proprietary stalwarts, not just on price and availability, but also on features.