Factory Overview, Part 1: A Bird’s Eye View of the Macaw Solutions Factory

The Macaw Solutions Factory is quite a big thing, even though it fits both small and large solutions. In this post I will take you on a quick tour of the major systems and tools in the Factory. Later posts in the Factory Overview series will zoom in on the details of those systems and tools.

Note: there are several series of Macaw Solutions Factory posts, on multiple blogs.
You can use this feed to track all Factory posts.

In Factory Vision, Part 1: Introducing the Macaw Solutions Factory I described what the Factory does:

“Automate and standardize development and deployment on top of Microsoft Server products, targeting shared infrastructures with .NET, SQL Server, SQL BI, BizTalk and SharePoint”

and how the scope of the Factory is defined:

“All infrastructure, tooling and content used in Macaw’s primary process: Realize IT Solutions with Microsoft Technology”.

Given this it’s not really surprising that there’s too much in the Factory to describe in one post.
So without further ado, let’s start the Factory tour. I’ll present the tools in the chronological order in which they typically are first used during the application lifecycle.

Microsoft Virtual Machine Manager

One of the first things that you need to do when you start development on a new solution is to provide machines for the development and test environments. Usually you need at least as much machines as there are developers in the team, plus typically build and test servers. Since each machine logically belongs to one solution (not to a person), over time this can result in a lot of machines that must be kept available (for maintenance) but that only need to run occasionally. Virtualization offers great benefits here, allowing flexible creation and storing of machines, as well as additional benefits such as quick rollback to a checkpoint in case a machine installation gets corrupted.

In the Factory we use Microsoft’s Virtual Machine Manager, run on a powerful array of virtual machine hosts, to deliver self-servicing creation, access and management of virtual machines. In addition to fully configurable OS installations, we supply preconfigured machine templates for 32 bits and 64 bits versions of Windows Server 2003 and Windows Server 2008. New machines can be created in a matter of minutes.

The Factory Machine Installer

Virtual Machine Manager offers great flexibility through machine templates, but many Microsoft server products do not work well when they are preinstalled on such a template. This is due to the fact that these server products were not designed to handle a machine rename. Renaming machines happens when you instantiate a machine template – you need to select a unique name for the new machine for network access. At the OS level a tool named Sysprep handles this, but many Microsoft server products lack this kind of machine rename support.

In addition to that, a more flexible and less maintenance intensive mechanism is desired to deliver the desired combination of OS, server products and applications that are needed to develop or test a specific solution.

Enter Microsoft’s Deployment Toolkit. We use this toolkit to create and maintain the Factory Machine Installer, which allows users to select any desired combination of server products and applications from a single menu. With a single click all selected technologies are installed fully automatic, with all the latest updates. During the remainder of the installation (which can take up to several hours) no user interaction is needed. If a team needs to start (or is extended) tomorrow, you can create all machines simultaneously and have them ready for production work in the morning.

Automatically installing and retaining stable development and test machines saves a lot of time during the application lifecycle. In traditional situations it is not uncommon to spend several days before a new team member has a fully productive development machine for a specific solution.

For more details, see this post on the Factory Machine Installer by Maarten.

Microsoft Visual Studio

The Factory leverages the power of Microsoft Visual Studio, both its built-in functionality and its extensibility. Visual Studio Professional and Team System editions in VS 2005 and VS 2008 versions are fully supported (note that some MS Server products require using VS 2005 because they do not yet support developing in VS 2008). The Factory allows you to mix versions and editions, even within a single Factory product, as needed.

Microsoft Visual Studio Team System, including Team Foundation Server, is the application lifecycle management (ALM) platform of choice for working with the Factory. However we also support alternatives for version control (Subversion) and automated build servers (CruiseControl.NET Build Station).

The freedom of choice that the Factory provides with Visual Studio versions, editions and components, makes the Factory fit both small and large teams, solutions, and development environments.

The MAST standard

At the heart of the Factory design is the Microsoft.NET Application Style (MAST) standard. MAST standardizes both the logical design of an application built with the Factory, and how the logical design is translated to an implementation structure in a working folder. MAST is based on Microsoft Patterns & Practices guidance on application architecture, it has at its core the three-layered services application pattern:

(Note that MAST supports both the above pattern from the 1.0 guidance and the updated pattern from the 2.0 guidance, which adds a services layer and an application façade component type)

The MAST standard provides a standards-based, durable, stable yet flexible implementation structure that is consistent across Microsoft server products and technologies. MAST offers consistency, separation of concerns, alignment with underlying server products, and cost-effective automation through convention over configuration.

For solutions built with the Factory this results in very good maintainability. Experience also proves that the consistent implementation structure across solutions makes it much easier for developers to work on another solution or on multiple solutions.

The Factory IDE

The Factory Integrated Development Environment (IDE) extends Visual Studio with templates, tools and automation to create a highly productive development environment. Virtually all tools and templates leverage detailed knowledge of the standard MAST implementation structure, to achieve maximum automation. MAST significantly reduces the cost of implementing and maintaining the Factory IDE; it also increases the attainable level of automation.

Our choices on how we utilize the available Visual Studio extension mechanisms in the Factory IDE are guided primarily by pragmatism. Rather than see how far we can get with the ‘default’ technology for a specific type of extension, we consistently select the extension technology that is most effective and future proof for our purpose. This approach has resulted in some innovative and powerful Factory IDE features, such as integration with PowerShell and with the Confluence Wiki.

These are the main tools and templates in the Factory IDE:

Factory Product Wizard

In Factory terms, the thing that you build with one instance of the Factory is called a Factory Product. Since the architecture and the implementation structure of Factory products is standardized using the MAST standard, Factory products are also referred to as MAST products. Some examples of products that you can build with the Factory are:

For more details, see in this post under the heading “What can you build with the Factory?”.

Once you have a development machine installed with all required server products and applications, the next step when you start a new solution is to create its working folder. In addition to the source code, each working folder contains a copy of the Factory IDE. This allows integrated versioning of both the source and the development tools; it also makes product-specific Factory customizations easy to create and maintain. The Factory IDE is designed to run in multiple versions side-by-side, so independent versioning per Factory product is also possible in a shared development environment (see this post for more rationale on this).

The Factory Product Wizard lets you create a working folder for a new product, with (only) the tools and templates that you need for developing that product. Since the Wizard knows the MAST standardized working folder structure, which is designed in accordance to the convention over configuration principle, it requires very little user input to create a product working folder that has a consistent structure.

MAST Working Folder

Once the working folder for a Factory product has been created with the Factory Product Wizard, it will contain all the templates, tools and automation scripts that you need to start developing your product. A typical working folder created by the Wizard looks like this:

The templates in the MAST working folder are ready to be built and (automatically) deployed with a single action. This allows you to verify if your development machine is functioning end-to-end (using the automated system test suite included in the templates), even before you have written any code.

You can also automatically package and deploy the templates to other machines, e.g. a test machine. This means you get practically free deployment tests starting from day 1 of a project. No more ‘nasty surprises’ that are common when you do your first deployment much later, e.g. right before your product must be tested in a test environment, only to discover too late that something in that environment was not installed or configured correctly.

Factory MSBuild extensions

When you build a MAST Visual Studio solution, post build scripts are run via an MSBuild task that hosts PowerShell. These PowerShell post build scripts perform a rapid (incremental) deploy of the current Visual Studio solution to the local machine. Deployment is done from the working folder (where files are edited under source control) to the runtime deployment locations that are configured for the local machine:

Factory Guide Visual Studio Extension

Developers spend most of their time working in Visual Studio. To automate development tasks and offer guidance to developers while they are working in Visual Studio, we created the Factory Guide.

The Factory Guide shows actions that are relevant in the context where the developer is working in. The context consists of elements such as the part of the product they are working on (the Functional Area) and the components that are present in that part. These actions are implemented in PowerShell scripts (the Guide hosts PowerShell). These scripts have direct access to the Visual Studio object model, the working folder, and the server products running on the development machine.

This provides a very powerful and flexible automation environment. Existing scripts can be modified easily and new scripts and actions can also be added easily. PowerShell offers excellent integration with command-line tools, web services and managed code assemblies, so it is easy to leverage any type of asset in the scripts. Modifying and extending the Factory Guide is simply done in the working folder of the product being built; no additional Factory source and no Factory development environment is required, and versioning is simple.

Each action in the Guide is preceded by icons that indicate whether approved and/or community guidance on that action is available (in the central Factory Wiki):

This way, up-to-date guidance availability information is visible at a glance for all actions in the Guide – no need to click through on each action to see whether there actually is any guidance. Any Factory user can add community guidance to any action with a single click. This means that when a developer (specialist) in any project adds guidance about an action, this becomes visible to all Factory users in all projects. Since guidance is initially sparse and then is created on the fly and as needed, this leads to guidance that is both valuable (there is a proven need) and effective (directly visible to all Factory users).

We call this guidance approach: community driven guidance in context.

MAST Solution Builder

First you’ve automatically created and installed your development machines with VMM and the Factory Machine Installer, and then you have created a working folder on your new development machine for your new Factory product with the Factory Product Wizard. Finally you have automatically deployed and tested the templates that you are going to use. Due to the level of automation in these steps an experienced user will spend less than an hour on them.

At this point you are ready to define the baseline for your new Factory product – meaning that you create all Visual Studio solutions and projects that will be needed (as far as is known at this point in the project). Once you have added all Visual Studio solutions and projects to the working folder, you can add it to source control and let the team start fleshing out the various components.

We created the MAST Solution Builder to let you quickly create new solutions and projects (or add projects to existing solutions). With a single click the Solution Builder takes the selected template projects, creates a new solution (if needed) and instantiates those projects in the new solution:

Since the Solution Builder understands the standard MAST structure of solutions and projects in the Factory, it can automate many tedious and error-prone tasks when creating new projects:

• It creates the solution and project folders with the correct name
• It copies all selected template projects while removing example code
• It sets all namespaces and project properties to the correct names
• It sets all correct project and assembly references, and updates linked project items
• It creates an FxCop project file with all project assemblies added as target and with the MAST validation rules added (see MAST Validator below)
• It updates comment headers with the correct product information.

Since the MAST templates live in the same working folder as the product source, under single source control, the templates can easily be customized and maintained for a specific project. The Solution Builder will instantiate the customized templates when creating new Visual Studio projects.

The MAST Solution builder requires very little user input to do its work; you only need enter the Functional Area (~ VS Solution) name and select the project templates to add.

By applying the principle of convention over configuration, the MAST standard allows to automatically translate a logical design (consisting of machine roles, product dependencies, functional areas and component types) into an implementation structure (consisting of product- and machine- config files and Visual Studio solutions and projects).

All the above makes it possible to create a complete implementation baseline, starting from a logical (whiteboard) product design, in a matter of hours – even for complex, large solutions. At that point, the entire baseline will build and deploy automatically without errors. The team can get from logical design to fleshing out the baseline in one day, while using continuous deployment across development and test environments.

MAST Validator

MAST standardizes the logical design and many details of the implementation structure. But even though this offers a lot of value, it is not realistic to expect developers to know all details of the standard by heart, much less to apply the standard consistently. The MAST templates and the MAST Solution builder provide you with a MAST compliant baseline, but once you start fleshing out that baseline, how will you make sure that you will remain compliant to the standard?

This concern is even more pressing because the Factory IDE tools rely on MAST compliance. If your source is not MAST compliant, there is no guarantee that the Factory IDE tools (e.g. automatic deployment) will work correctly.

To address this concern we created the MAST Validator. As the name implies, the MAST Validator validates the source in the working folder against the MAST standard. We implemented the MAST Validator as an FxCop rule library:

The MAST Validator analyzes much more than typical FxCop rule libraries, which usually only analyze compiled assemblies. The MAST Validator also analyzes source files (including Visual Studio project files) and even databases.

Where applicable, MAST rules adjust themselves automatically to customized templates. E.g. in the example above, the project references in the template solution (which can be customized just as any other solution in a Factory product) are interpreted by the rule as the set of all potential valid project references. If a non-template solution contains any other project reference, the rule reports this as an error. This allows for automatically detecting unwanted dependencies, much like the Layering Diagram in VS 2010 will provide.

In case you wonder why we integrated the MAST rule library in FxCop instead of directly in Visual Studio, this is because FxCop provides us with additional UI that supports the audit process. In FxCop you can exclude rule messages with a reason, and you can see all exclusion reasons at a glance. When you want to do a quick audit of a product, you run FxCop, verify that no active MAST messages exist, and whether you agree with the rationale provided in the notes of excluded messages:

The Factory IDE will only create MAST deployment packages if there are no active MAST error messages; this effectively enforces MAST compliance throughout the application’s lifetime.

The MAST Validator helps you to keep your source MAST compliant, and to audit on MAST compliancy, with little effort. At the same time flexibility is preserved: you can ignore rule messages if you record a satisfying rationale to do so.

Factory Local Build Server

The Factory fully supports Visual Studio Team Foundation Server build servers. However as an alternative for smaller projects, a lightweight build server is also included in the Factory IDE:

The Factory Local Build Server can be run on any development machine. It offers several build types to produce a MAST Product package; with the package you can then deploy your product on all required machines in DTAP. The Factory Local Build Server is based on CruiseControl.NET Build Station; it is completely self-contained and is automatically configured for your product. It has no prerequisites apart from Visual Studio (IIS is not needed).

MAST Packages can be built from source control (so you know for sure all required files are present in SC) but also directly from the working folder. The last option allows you to create and test deployment packages even before local changes are committed to source control. This gives you the opportunity to fix deployment and integration issues before they can propagate via source control to other team members and/or environments.

The Factory Local Build Server provides the benefits of a basic build server for smaller projects without the need for additional machines, software, or configuration.

MAST Package Installer

Once you have produced a MAST Package with the TFS Build Server or with the Factory Local Build Server, you can use that single package to install your product on all machines across DTAP. Typically the debug or FileSystem packages are only deployed in development environments, while the Release SourceControl packages are deployed across test, acceptation and production environments.

The MAST package (which is a ZIP file) contains the complete Factory product as well as all information on what parts of the product should be deployed to which locations on each machine. Installing a package is simply a matter of unpacking the ZIP on each machine and then running the included Setup.cmd.

For environments with many machines deployment can be further automated to offer a central drop location for a new package to be deployed.

Since the package contains all deployment information on all machines, no user input is required. Not having to write, communicate and perform manual deployment instructions adds up to a lot of time saved during an application’s lifecycle; it also avoids many potential human errors. The deployment procedure is the same for first installs as well as updates; the logic in the package installer script automatically handles both situations correctly.

The MAST Package Installer is implemented as a set of PowerShell scripts and some integrated assemblies and command line tools. It is easy to add custom, product-specific deployment scripts by dropping in extra PowerShell scripts that follow a specific naming convention. The MAST Package Installer will then run your extension script at the appropriate moment(s) and pass it the relevant deployment information.

Factory Wiki

All written guidance in the Factory is maintained in the Macaw Solutions Factory Wiki. We set out to create an MSDN-like documentation system with both endorsed content and community content on each page. Since we want to have our guidance community-driven, we needed a Wiki with identical, rich content editing capabilities for both page content and user comments. After extensive research we selected Atlassian Confluence Enterprise.

The Wiki has good integration and extension capabilities; e.g. we used this to integrate the Factory Guide in Visual Studio to the Wiki. It also has very good import- and export capabilities (e.g. from MS Word and to PDF).

Because the Factory IDE integrates directly into one central Wiki, and any user can add rich comments on any page with very little effort, the Factory Wiki is an effective platform to quickly share knowledge amongst all Factory users.

Office Communications Server & Terminal Services Gateway

Also part of the Factory is our infrastructure to support the New World of Work. We enable Factory users to work together anytime from any location. We use Office Communications Server and Terminal Services Gateway to enable this:

A typical scenario that this infrastructure supports is:

• A Factory user needs assistance on an error that occurs in his development machine; he checks in Office Communicator which colleagues are available and then IM’s a co-worker.
• The co-worker needs to see more details so they remotely share the desktop of the user’s development machine, either using Terminal Services Remote Control on two Remote Desktop connections, or with desktop sharing in Office Communicator.
• At the same time they open a hands free voice channel via Office Communicator and/or mobile phone.
• Working on the same screen while talking, they research and resolve the issue together.

This type of ad-hoc remote pair working is incredibly more effective than documenting and communicating an issue via email (let alone traveling to the same location); it has proven a big time-saver.

Experience has learned – surprisingly enough – that working remotely together this way is actually more productive than sitting physically together at the same machine. This is because each person has his/her own display, keyboard and mouse while working together on the same virtual screen.

Factory Improvement Forum

The above Factory tools and systems help you to quickly realize IT solutions based on Microsoft server products. However, the Factory itself is also continuously developed to support new technologies and to incorporate newly learned best practices. To direct this development we created the Factory Improvement Forum, where Factory users can suggest ideas for new features and vote on them.

With this forum we can effectively prioritize new factory features. This helps to make sure that we are always working on the right new features, the ones that are most wanted and most valuable to the Factory users.

This is where the Factory both gets and delivers its value: to the people working in innovative projects, realizing real-world IT solutions.

Closing Remarks

If you are reading this, I guess you’re not put off by the substantial size of this overview and I’d like to thank you for your interest. The next posts in this series will be more focused and much shorter.

The post Factory Vision, Part 1: Introducing the Macaw Solutions Factory gives an overview of the vision that drives the Factory; this post gives an overview of the tools and systems in the Factory as it exists today.

The Factory is designed for continuous innovation, to integrate new (versions of) Microsoft technology, while gradually leaving obsolete technologies behind. Many new Factory Features are already envisioned – to support new technology, more team roles and more aspects of ALM. But also to further raise the level of abstraction and automation. The Factory is a powerful tool, whose development is guided by both pragmatism and the compelling vision as described by Jack Greenfield et al in ‘the’ Software Factories book:

The Factory approach has proven to be very valuable with today’s technology and level of effort. Even so I am convinced that in terms of attainable value, this is just the beginning.

Vincent Hoogendoorn