Wednesday, January 10, 2007

Reshaping IT Project Delivery Through Extreme Prototyping

Extreme Prototyping is an architectural process for developing applications, especially web applications, in terms of increasingly functional prototypes. At a high level, it breaks down web development into three distinct phases. The first phase is the static prototype, consisting of HTML pages and possibly a logical data model supporting those pages. The second phase is a coding process in your chosen web framework whereby the screens are fully functional using a simulated services layer. The third phase is where the services are implemented. The process is called Extreme Prototyping to draw attention to the second phase of the process, where a fully-functional UI is developed with very little regard to the services other than their contract. This deliverable, much like the static HTML prototype, can be considered a prototype, albeit a dynamic one.

By using Extreme Prototyping, you will be able to substantially shorten the requirement cycle and also the development cycle. Extreme Prototyping will allow you to consistently estimate and deliver your projects faster, cheaper, and better. Extreme Prototyping achieves these results primarily by breaking down the dependencies between multiple teams and thereby allowing parallel development. Extreme Prototyping continues to put extra emphasis on executable deliverables.

The Three Bricks of Extreme Prototyping

If you imagine system development as a set of concrete deliverables, you can imagine an "Extreme Prototype" to consist of three bricks, as depicted in Figure 1.

The three bricks of Extreme Prototyping


Figure 1. The three bricks of Extreme Prototyping

The first brick is the Static Prototype. Based on this prototype, the UI brick and the Service brick can be developed fairly independently and brought together toward the end of the project for an integration alignment. The service contracts (or blades) should allow for a perfect alignment in any typed language.

The Anticipated Advantages of Extreme Prototyping

Iterative approaches such as Extreme Programming recommend building and delivering your working programs in small steps and doing so often, especially in terms of executable code that is delivered daily and weekly. Nevertheless, there are assumptions for this to work well. The team has to be small enough to handle the continuous feedback. If projects have a tendency to be cancelled, the partially-baked project is another reason to take that step; Extreme Programming works best when all the teams are fully committed to carry out the project to the end.

Extreme Prototyping with its three phase approach is orthogonal yet symbiotic to the iterative process. The static prototyping phase ensures project commitment from the business. The dynamic prototyping phase ensures speedy delivery without depending on a service layer implementation. Delivery and demo of the dynamic prototype to the users will provide feedback on a midpoint of the project. The service implementation phase, which can start in parallel with the dynamic prototype, has a clear service contract to fulfill. Iterations are can be employed within each phase as appropriate.

Many IT projects start with a requirement phase that delivers use-cases. Business users are expected to understand these use-cases and sign off. What happens more frequently in the field instead is that the business team gets lost in sticky figures, activity diagrams, and so on. So much time is spent elaborating on these use-cases that in the end, all the allocated budget (and more) is spent before writing a single line of code. The key is to wrap up the use-cases very quickly and use them to derive an HTML prototype that is as complete as possible. It is easier for business teams to validate and agree on a prototype. The clarity of prototypes will work as a catalyst to complete the use-cases and act as an antidote to the analysis paralysis.

Extreme Prototyping via static prototypes can also aid in budgeting and planning. Very often a business seeks an estimate for a project based on just a vision document. The estimate would include all the phases of a project including build and delivery. Estimates based on this document are subjective and will have a lot of variability. The anticipated cost may close the project down. Instead of putting an request for proposal (RFP) for the whole project scope, the business team could put together an RFP just for the Static Prototype. This will yield an estimate that has a mostly fixed cost, and the deliverable is readily reused in an eventual estimate.

Subsequent phases can then be budgeted and planned.

Static Prototypes can help avoid technology risks. Static Prototypes are developed with well-known and less risky technologies such as HTML, CSS, and JavaScript. As these technologies are proven and low risk, it is unlikely that the Static Prototype will be delayed or cost more than expected. In contrast, directly moving into an iterative phase with the end technology from the start could pose learning issues or even substantial road blocks. Dealing with these challenges during the delicate, uncertain, and crucial requirements phase is unwise.
Overview of the Complete Development Process

Here is an outline of an Extreme Prototyping process:

* Static Prototype phase
o Static Prototype
o Master or background pages
o CSS, JavaScript
o Business rules, use-cases
* Extended Static Prototype phase
o All of the above
o Logical data model to support the screens
* Dynamic prototype (or Extreme Prototype phase)
o UI recoded/adjusted for the chosen web framework
o Working executable code
o Field validations work
o Navigation of screens will work
o Service signatures solidified
o A complete working UI with no implementation behind
* Service Implementation Phase
o API document
o Each service implemented by calling databases or other resources
o Integration

Extreme Prototyping starts out like any other well-planned project with a Static Prototype, but distinguishes itself in the coding phase of the UI. Extreme Prototyping strives to see the UI deliverable as another prototype with a thoroughly spec'ed service layer. In this sense, the UI layer is defining an API for the service implementers. This is similar to how various JSRs are developed. One body defines the standard and the API, and number of companies implement that standard using the common API. The UI implementation will then be followed by the service implementation.

Static Prototyping Phase in Detail

The Static Prototype starts with a vision first. From the vision you recognize and write down a series of users that would be using the system. You write down goals for the system. The analysis can take the form of actors and use-cases, or a word-processing document outlining each need. The key to Extreme Prototyping is to minimize the details in these documents so that most of the requirements are captured primarily in the prototype.

One of the early resources that needs to be engaged is the prototyper. You want to hire a prototyper that can grasp your vision for the project after a few sittings with the users and can lead them to a final solution ably and quickly. The key for the prototyper is to understand the system well enough, either by talking to the users or reading documents, whichever is faster and more efficient. You want this prototyper to have the following skills:

* Ability to imagine the site from start to finish
* Ability to design style and content
* HTML, CSS, and JavaScript
* Communication, understanding, and empathy
* Ideally, would have developed a few websites before
* At least 5 to 10 years in website design would be a plus
* Experience with Flash etc., if the site is more interactive
* Experience with portals
* Experience with marketing in that space

I expect web-based tools to emerge rapidly in the next few years to aid in Static Prototyping. I am especially fond of project portals where a project is started in a company by allocating a website to it. Everything that happens to the project is captured with ease on the project web site. I have used this very effectively for a couple of projects, as described in How to Publish Multiple Websites Using a Single Tomcat Web Application. All project-related communication and assets are portrayed via web presence. For instance, users can enter their wishlists for the system as a series of documents online with out any additional tools. Discussions and elaborations can take place for each posted item. The items can be classified into modules and searched if required. The prototyper can take notes on the project website while in meetings and capture the tacit knowledge. All emails related to the project can be tagged and made available as conversations on the website. This early knowledge aggregation will help the developers downstream to quickly decide what needs to be done on a particular screen or in a given situation.

Prototypers typically use Dreamweaver, FrontPage, or just plain HTML-enabled editors for developing the actual prototype. Whichever tool you choose, the following support will be useful:

* Good support for CSS
* Good support for background or master pages
* HTML tag completion
* HTML tag or in general code assistants
* Optionally some kind of content management support
* Ability to quickly publish over the internet or intranet so that users could quickly see results

I expect online web tools to emerge in this space as well. Recently, I have been experimenting with knowledgefolders to do the following:

* Create/start a prototyping account for a particular project.
* Design background pages for this project to nail the look and feel.
* Have the background pages include or define the CSS, JavaScript, etc.
* Free prototypers to focus on each page without worrying about the background in which it sits, much like "master pages" in PowerPoint or other presentation apps. This will give the prototyper a nimble ability to quickly prototype online.
* Design screens in this online form to make readily available for users to critique.
* Crosslink in the system to a critique page so that users can type their feedback directly about the screen in question.
* Crosslink the prototype screen to another document that will talk about field validations and specifications.
* Provide every possible bit of information needed by the developers to turn the prototype in to a working system without ever leaving the online prototype.
* Publish the online prototype account as its own web domain, such as mysystem-prototype.mycompany.com.
* Publish the service signatures, particularly the API, needed by these screens.
* Crosslink the prototype screen to a set of test cases that the developer has to test the screen with.

The conclusion of this phase will be a clickable website that will reproduce the user experience as close to the real thing as possible. Usually the following would have gone into this effort:

* A Static Prototype with navigation
* Proper background pages
* Proper CSS
* Detailed specifications to accompany the prototype (optional)

Idea of an Extended Static Prototype

A Static Prototype that is fairly complete has hidden assets that can be mined. The primary one is the logical data model that is necessary to support the prototype. Screens, much like use-cases, will make evident the main topics or logical entities or tables in a database. The summary and detail screens will reveal the relationships between tables. Various form elements and display elements will clarify field-level attributes of a table. In short, it is possible to derive the logical data model from the screens. The logical data model can further be converted into a physical data model. At the end of this, it is possible to produce a database ready for development by the developers. I call this Static Prototype, when accompanied with an instantiated database, is an Extended Static Prototype. The Extended Static Prototype is a logical unit that can play a role in division of duties.

Companies often evaluate to see if certain software is best developed outside due to expertise or time and resource constraints. The main drawback to outsourced development is how the delivered software will fit into the internal architectures and resources, not to mention to the coding standards and the rigor expected internally. An Extended Static Prototype can solve this problem. A company can choose to outsource an Extended Static Prototype and the variability of an Extended Static Prototype is small. This means there are very few places that the outsourcing company can go wrong. The technologies are known and what is delivered is conceptual. On the other hand, the source code for a website has a substantial variability. So by outsourcing elements of least variability you will come out more satisfied. Under this scheme, an outsourced company that specializes in websites and can give a working prototype along with the logical data model to the in-house development team. The development team can then develop the concept using any of the chosen frameworks. It is like a tag team, ready to run from the start.

Another asset that can be directly derived from the prototype is the set of test cases for each screen. Together the additional deliverables of an Extended Prototype can be summarized as:

* Logical and physical data models
* Test cases

Dynamic prototype

The primary goal of a dynamic prototype, abstractly speaking, is to separate the known and unknown so that progress can be made in the known space. UI programmers, on the receipt of a Static Prototype, will proceed to create a UI layer that is completely functional. UI layer will talk to the service implementation through one or more of service interfaces. It is possible to complete this working UI layer without ever implementing a single line of the real implementation. The finished UI layer will be complete in the following respects:

1. The HTML of the static prototype will be transported to the UI elements of the respective frameworks.
2. All the CSS and the look and feel will be in place.
3. Client-side validations will be functional.
4. Server-side validations will be functional.
5. A domain object model will be in place to support the screens.
6. Service calls will be made for all the updates or actions on the pages.
7. Navigation of the screens will be in place.
8. A simulated service implementation will provide the data for screens either in terms of Java objects or XML.

Despite the seemingly hard exterior of typed languages, they provide the basics needed to successfully execute a dynamic prototype. The key is enforcing the contracts in terms of well-defined objects and the service API.

The one question that begs to be asked is: if you're going this far, why not implement both layers together? The primary reason is to break down the dependencies between the service implementation team and the UI team so that both teams can be productive simultaneously. It is very common for the UI team to sit paralyzed until a database instance, domain objects, or a service layer is in place. How can they code unless they know all these details? The goal of dynamic prototyping is to answer this question with simulated APIs and allow UI to proceed without service implementations. By having conceptual domain models for screens and screen-level objects it is possible to envision the system in terms of these objects and then let the service layer contracts translate these objects into proper implementations. By taking this approach both UI and services teams can start their work simultaneously.

While following this process UI programmers often question about what goes into the UI layer and what goes into the services layer. How far should they go in the UI layer? How do you draw the boundaries? Some of this is clear enough where data access, storage, and business logic that is devoid of screen influence is relegated to the service layers. Sometimes another way of looking at this can help you. If you know how to execute or perform a certain function clearly with the information you have in the UI layer, then do so. The moment you run into a roadblock where you can not proceed in the UI without knowing some piece of information, make that unknown a service and simulate it, like everything else, allowing you to postpone that unknown while you proceed with the rest of the UI. In this sense the interfaces are treated like the boundaries between known and unknown.

The Idea of a PageData or Page Object

If you examine closely the dynamic prototype that is being suggested here you will see that it is advantageous to represent an HTML page as a Page object in memory. This Page object parallels the physical HTML page: it will have data on it, it will have actions on it, etc. Consider the following code:

public class MyWebPage
{
//Concentrate your data in one place
MyWebPageData;

//ui controls
uiControl1;
...

//Population etc
onload(){}
postback(){}

//Actions on the page
action1(){}
...
}

public class MyWebPageData
{
//data only
field1;
list1;
object1;
...
}

All the data on a web page is encapsulated into a page object. This approach has the following benefits:

1. Provides a conceptual model for developing the UI, irrespective of how the data is organized in domain objects.
2. The page data object can be streamed in a variety of formats including XML in response to incoming URLs.
3. Allows UI developers to complete their contract without heavily depending on the backend systems.

Architecture for a Dynamic Prototype

Consider the following architecture in Figure 2, where an action on a web page is directly accessing the DAO objects and manipulating them for a certain business purpose.

Actions directly using DAOs


Figure 2. Actions directly using DAOs

In this scenario, the Action class is responsible for two things. On one side, it interacts with the UI of which it is part of. On the other side, it interacts with business logic. There is a distinct advantage in separating these two concerns. If the action class can postpone the business concern through a stub or a service, then the UI can be built independently of the business service. This will allow division of labor and hence quick deliverables by throwing more people at it.

With this thought in mind, let me re-architect Figure 2 into Figure 3:

Actions using services


Figure 3. Actions using services

The key features of this design are:

1. Service methods are encapsulated into one or many service interfaces.
2. Each interface will be implemented by an implementation.
3. Implementation will deal with business logic and also the persistence of the business objects.

This can be further enhanced by introducing a static wrapper for the services, as in Figure 4:

Actions using services via a static wrapper


Figure 4. Actions using services via a static wrapper

1. IServiceInterface will define services. There may be a couple of these interfaces in the whole project.
2. Inputs and outputs to these methods are plain domain objects.
3. An implementation of this interface will call DAOs.
4. A static wrapper will front the implementation by hiding any construction details of one or more implementations.
5. Static wrapper can intercept services for value added services such as:
1. Transactions
2. Logging
3. Simulation

The key concept in dynamic prototyping is to build the system (as much as possible) as if there is a services layer implementation from the start fulfilling the contract. Initially, the implementation is going to be using some mechanism to simulate the contract. The static wrapper or an equivalent scheme can nicely package the interfaces into some sort of a distributor, where multiple implementations can co-exist. The article Server-Side Typed Event Distributors further explores these ideas and points to an architecture where each method of an interface can be gradually refined for a final implementation. Under this architecture, the static wrapper will look more like Figure 5:

Design of static distributor of services


Figure 5. Design of static distributor of services

The finer points of this architecture are:

1. Static wrapper can hold multiple implementations.
2. The default service implementation will provide a filler for each method.
3. Each implementation can pick and choose which method to implement.
4. Static wrapper will use a certain hierarchy to pick an implementation, usually starting with the real implementation.
5. This means, when the real implementation has a valid method, it will be called and not the other simulated or secondary ones.


Service Implementation Phase

In this phase or layer you will accomplish the following:


1. Consolidate the logical data model
2. Create the database
3. Implement service contracts

The Static Prototype will directly lead to a logical data model. The initial logical data model can be worked out by a development lead based on these screens. The approach is not that different from coming up with domain objects based on use cases in the rational process. The logical data model can be further refined as the service contracts take shape.

Once the logical data model is in place it makes sense to describe the relationships of tables in an annotated document called an annotated logical data model. Each page will have four or five tables and describe how they are related. This is in contrast to a single logical diagram that spans a wall in an office.

There is one document that I have found most useful in the execution of these projects. I call this the API doc. This essentially documents the service contracts. The table of contents of this document may look like this:

1. A list of all the screens in the system.
2. Each screen contains:
1. A prototype picture of the screen
2. A bulleted list of responsibilities or how this screen works (especially the not-so-obvious ones)
3. A list of APIs for that screen
3. Each API contains:
1. Name
2. What is the responsibility or functionality of the API
3. Input objects
4. Output objects

Using this document service implementers can write the APIs to satisfy the contracts. This document can also be used by the UI developers to know what the screen looks like and what needs to be done slightly better than the use case document. Or UI developers can help write this document as they start coding each UI screen.

Conclusion

Extreme Prototyping allows you to make a steady progress while postponing the lesser known issues to the rear which gets worked when their turn comes instead of tackling the entire problem once. In the process, it offers the following benefit:

1. Early risk avoidance.
2. Early savings in cost and resources.
3. Guaranteed system quality as the prototype clearly lays out the system.
4. Reduced dependency on the skills of Java developers for designing the UI.
5. Increased the number of UI developers to distribute the screens during the dynamic prototype. This is possible because you have reduced (or postponed) the dependencies through interfaces.
6. Parallel development of services and UI at the same time by allowing simulated services.
7. Faster, cheaper, and better product in the end.