[UPDATE this article was originally published here, I included it on this blog to get my story on factories in one place.]
Vincent Hoogendoorn, Architect, Macaw
This article details a case on how the Software Factories vision is being pursued and realized by a Microsoft Systems Integrator specialized in .NET technology. A careful approach is detailed to gradually realize the vision, avoiding large investments and high risks while still reaping immediate and significant benefits. A detailed overview of the key first elements of the software factory is provided: a generic .NET architecture style, project templates, and a comprehensive collection of both standard and custom development tools. The experienced benefits are detailed and the potential is identified both for other systems integrators to apply this approach and for further development towards realizing the full software factory vision.
As a Systems Integrator specialized in advanced Microsoft web technology, our larger projects reached a scale that really required standardizing and automating key aspects of our development process. Without this, scaling the development teams beyond a certain size made the development environments and -process unpredictable and inefficient. This particularly proved true for projects based on (combinations of) complex server products such as SharePoint, Commerce Server and BizTalk. To remedy this, we implemented processes, guidance, a standard architecture style, project templates and custom development tools. This approach proved very successful to stabilize and accelerate large projects. We soon realized that our approach could be generalized and made scalable, so it would benefit both small and large projects involving arbitrary combinations of Microsoft products. While studying key publications on the Software Factories vision [Greenfield04], we recognized that we in fact – in a modest way – were realizing part of that vision. We decided to embrace the wider vision and to devise a careful approach that would allow at least 80% of our development effort as a Microsoft Systems Integrator to benefit from the software factory approach.
We wanted a careful approach to realizing Software Factories, based on proven success, without taking large risks or making heavy investments. This led us to stage our approach in two phases.
The first stage is based on the proven success of our approach for large projects. The objectives we set for this stage are:
- Accelerate and scale development through standardization and automation.
- Create an optimal environment for harvesting and deploying reusable assets.
- Prove success in both small and large projects.
- Facilitate 80% of our development effort by the end of 2006.
This first stage focuses on reusing software structure, not functionality. We first want to create an environment of many products with a standardized software structure, which by itself has proved to provide a lot of value. However, such an environment will also be an optimal breeding ground for effectively reusable functionality.
The second phase of our approach takes off when sufficient functionality reuse has been recognized, proving the business case for reuse. At that point, we define one or more software product lines that – based on the common software structure –provide packaged and configurable functionality.
A key element of our approach is that we only package functionality for reuse after the potential for reuse has been demonstrated for two or three occasions within our domain. We do not want a big upfront design approach; reuse has to emerge.
To realize our approach, we specified a standard architecture style for the software structure of our products. The family of systems that we target with this architecture style consists of all applications and services, built in .NET with C#, based on (combinations of, multiple versions of) Microsoft server products.
While analyzing the .NET products that we – as a Microsoft Systems Integrator – assembled since the platform became available, we recognized a single pattern for structuring software that not only proved to add significant value to our largest projects but that would also fit the vast majority of this family of systems well.
This pattern is known as the Three-Layered Services Application. Microsoft documents this pattern [MSDN05] and also provides extensive guidance on designing applications and services for .NET based on this pattern [PnP02]. Applications and Services designed according to this pattern do not only benefit from it from a stand-alone perspective, but they also integrate very well in a SOA, as detailed in the Connected Systems vision by Microsoft.
Software Structure in a Three-Layered Services Application
Since the Three-Layered Services Application pattern had proven to be valuable and durable and also appeared to apply throughout our domain as a Microsoft Systems Integrator, we decided to base our architecture style on it. To reflect that we target all of our .NET development with this style, we named it MAST (Microsoft .NET Architecture Style).
A key guiding principle for defining MAST is that functionality assumptions should be avoided. We focus on the Microsoft foundation products that we use and on the generic software structure in .NET. This is to promote a large enough scale for our (emergent) software product line(s) to be effective. Since we are a relatively small company, splitting up our economy of scope could very well reduce the advantages of the software factory approach to the point where it would become unprofitable.
The MAST standard specification was developed according to the Maturity Level for Work in Progress of the W3C Process Document [Jacobs05]. It serves both as architectural guidance and as a specification to build templates and tools for accelerating development.
We intend to publish the full MAST standard specification in the near future. However, to make this architecture style more concrete, we provide a fairly detailed overview of the specification here.
The main elements specified by the MAST standard are:
- Standard component types.
A component taxonomy based on the Three-Layered Services Application pattern, extended with specialized component types for specific Microsoft foundation products (an example of a component type is a Business Entity, see the figure above).
- A standard namespace hierarchy.
This hierarchy achieves a very good balance between consistency and flexibility in software structure. It promotes stability, reuse, low-cost refactoring and flexible deployment.
Namespace Structure in MAST MAST specifies standard names for each namespace level that indicates a technical role, but not for the other (functional) namespace levels.
- A standard structure for solutions and projects in Visual Studio.
MAST specifies that each component type maps to one Visual Studio project. The namespace level directly above the component type (either a functional area or a product dependency) maps to one Visual Studio solution. MAST specifies naming of solutions and solution items, projects, project folders and project items. Finally, MAST specifies Visual Studio project property values.
Both VS 2003 (.NET 1) and VS 2005 (.NET 2) are supported.
- A standard structure for dependencies.
Because each Visual Studio project contains components of a known type, MAST can specify all potentially valid project references within a Visual Studio solution.
Combining relaxed layering (meaning a component may directly access all lower layers) with selected design pattern implementations has resulted in a matrix that specifies all potentially valid project references within a Visual Studio Solution.
Dependencies between Visual Studio Solutions are restricted; only dependencies on the functional area named Framework are allowed. Since foundation products may have dependencies on each other (e.g. SharePoint Portal Server 2003 has a dependency on Windows SharePoint Services 2), dependencies on the framework of other product dependencies are also allowed. This results in a funnel-like structure for dependencies across visual studio solutions:
Dependencies across Visual Studio Solutions in MAST
A key benefit of this dependencies structure is that the impact of changes outside the framework is limited to a single visual studio solution. Since all solutions have the same structure (based on the three-layered services application pattern), refactoring functionality from solutions into the framework is very cheap; it can be done when the need for actual reuse in a second solution arises. This results in thin frameworks that only contain functionality that is actually used in two or more solutions.
- A standard folder structure for source.
A key aspect of this folder structure is that a product dependency (e.g. DotNet2 or Wss2) is completely self-contained.
Part of a product folder structure in MAST. Visible in the left pane are the folders for the DotNet2 product dependency with its configuration, shared assemblies and functional area’s (Framework and Templates). In the right pane the template solution with its DotNet2 specific project templates is visible.
Templates and Tools
Based on the specified standard architecture style, we created templates for visual studio solutions and projects for specific product dependencies. The templates provide software structure and only contain minimal functionality – just enough to allow actual deployment, running and testing. Separate versions of the templates were created for Visual Studio 2003 (.NET1) and Visual Studio 2005 (.NET2).
A template solution for the DotNet2 Product Dependency in MAST
As illustrated in the example above, the software structure defined by MAST is easily recognized in the templates. The product dependency (DotNet2) and the functional area (Templates) appear in the solution name. The layer and component type (e.g. Presentation.WepAppUI) appear in the project names, and the component subtypes (e.g. Pages) appear as folders within projects.
Post-build events for incremental deployment
The templates have configurable post-build events that perform quick incremental deployment to the local machine. By separating the deployment location from the editing location (e.g. for aspx pages) it is possible to do an automatic full deployment to a development machine at any time, without touching any development work (source) in progress. Developers then only need to rebuild the solution they are currently working on to integrate their changes locally. Even though we already use a continuous integration build server, we found that adding this approach significantly improves the predictability and efficiency in the development environment.
Because the .NET 2 based version of key Microsoft foundation products is still a while away (e.g. for SharePoint Portal) and we need to support 80% of our current development effort, we still need to fully support .NET 1 development with our tools (as well as .NET2). While technology currently available in Visual Studio 2005, Team System and the Guidance Automation Toolkit is very promising, we decided to start out with a toolset that supports both .NET versions. Once the volume of our .NET 2 development effort is big enough to justify the investment, we plan to create a GAT based version of the tooling. From then on, we will gradually shift our focus for further tools development towards the GAT based version.
We use the following standard tools in our automated build servers:
- CruiseControl for continuous integration build server.
- Subversion for source control.
- NAnt for automating build and deployment scripts.
- NUnit, MbUnit and IeUnit for automated testing.
- FxCop and NDepend for code analysis.
- NDoc for generating documentation.
To support and automate development using the templates and the specified architecture style, we also use the following custom tools:
- MAST Software Structure Generator.
Based on the specified target namespace of a new component, this tool selects the appropriate template and uses that to generate the new component class in the specified namespace. If a new solution and / or project is needed to contain the new class, they are also generated on the fly. All project properties and references are then set according to the standard. We plan to add a VS 2005 front-end to the tool using the GAT.
- MAST Software Structure Validator.
A tool – similar in operation to FxCop – that validates the software structure of a product by analyzing its source. All rules specified by the MAST standard are validated. As with FxCop, an online knowledgebase provides details and rationale for each rule; it also explains when each rule applies. Developers do not need to invest a lot of time in learning and remembering all details of the MAST specification – they learn from the validator’s feedback while developing the product.
- MAST Build & Deploy Script Generator.
This tool generates post-build events for quick incremental deployment to the local development station (we separate the edit location from the deployment location for all artifacts). This tool also generates scripts for the build server. These scripts create builds, package them, and can be run on any machine in any environment to automatically deploy the full build. The tool uses machine-specific deployment configurations to generate and run the scripts. The configuration allows automatic deployment to different locations and deployment of different subsets of the product functionality, based on the machine environment, role and configuration.
Both structure generation and structure validation allow templates to be maintained and specialized with the product. This type of template maintenance is very quick, because the templates are just simple projects that live in the designated functional area named “Templates”, which – in a debug build – is automatically built, deployed and tested, just like any other code is.
The key benefits we experience from this approach are:
- Because of the high consistency across projects, resource management can be done in a much more flexible manner. Dynamically varying team composition, size and locations (e.g. for virtual teams and outsourcing) becomes easier.
- Deployment (both incremental and full) is almost completely automatic; no hand-written deployment scripts are needed.
- Designing and maintaining the software structure in a consistent manner soon becomes second nature for all team members. Much less time is spent on arbitrary decisions and on translating different sets of concepts / styles between (replacement) team members.
- Communication within development teams is very efficient and unambiguous. The concepts defined in the architecture style become common language. This proves particularly valuable for communication between architects and developers.
- Because the architecture style avoids functionality assumptions, the software structure remains very stable when requirements change (over time).
- Because the same three-layered service application pattern is repeated in all functional areas (including frameworks), refactoring becomes cheaper. Refactoring now becomes a more lightweight, on-demand activity.
- Last but not least: because of the consistent structure across projects, recognizing, harvesting and generalizing functionality for reuse can be done very effectively.
The Road Ahead
As a Microsoft Systems Integrator we do not think we are unique. Our approach assumes the .NET platform and a team-based, project-based approach; however it avoids assumptions about functionality or methodology (which in our case is based on the Rational Unified Process). Our approach could therefore be valuable to other Systems Integrators as well, especially for companies that concentrate on Microsoft technology and use a team-based, project-based approach. It seems likely that good collaboration opportunities exist.
As a growing part of our development effort uses this approach, its benefits are expected to scale better than linear. Once the actual reuse of functionality reaches critical mass, we will define and implement one or more software product lines to package that generalized functionality, realizing even more benefits.
We will continuously improve and extend our factory assets (e.g. tools, templates, frameworks), moving towards the latest software factory development tooling as the market share of its supported platform grows.
The potential value of realizing the software factories vision remains compelling, although the road to get there will take some time.
Software Factories – Assembling Applications with Patterns, Models, Frameworks, and Tools
Jack Greenfield, Keith Short, Steve Cook, Stuart Kent, John Crupi
Published by Wiley, ISBN: 0-471-20284-3
W3C Process Document
Ian Jacobs, W3C
“Three-Layered Services Application” application architecture pattern
Application Architecture for .NET: Designing Applications and Services
Microsoft Patterns & Practices Group
About the Author
Vincent Hoogendoorn is an architect at Macaw, a Systems Integrator specialized in advanced Microsoft web technology, based in the Netherlands.
His professional areas of interest are Solution Architecture, Service Oriented Architecture and Software Factories for the .NET platform. He recently was involved in projects for Smurfit- Kappa, Bouwfonds, Sara Lee and Akzo Nobel. Starting 2006, he is also member of Macaw’s Knowledge Development group, where he works on realizing the Software Factories Vision for Macaw.