VincentH on .NET

When you need to deploy a solution to multiple environments (e.g. separate Development, Test, Acceptation and Production or DTAP environments), you often have a set of environment-specific settings that you need to manage.

Since version 2.1 the Macaw Solutions Factory supports environment-specific web.config, app.config and EntLib config files.

Recommendations

Managing environment specific settings is risky, because by definition you cannot test these settings anywhere else.

For this reason, the Microsoft .NET Architecture Style (MAST) recommends to minimize the number of environment specific settings that your solution needs by design.

Secondly, MAST recommends that you further reduce risk by avoiding duplication of environment specific settings in multiple config files within the same environment. Instead, try to place these settings as much as possible in one central repository for each environment, e.g. in a service, a database or a SharePoint list (the last one has the added benefit of a free management UI).

Still, some technologies require settings in configuration files. To support this, the Factory allows you to manage environment-specific web.config, app.config and MS Enterprise Library (EntLib) settings.

Note that each MAST product has separate app.config, web.config and entlib config files. However to further reduce duplication of config files when maintaining multiple Factory products in the same environment, it is possible to share config files from a single location in source control across multiple MAST products, by using e.g. Externals in Subversion source control or NTFS Junctions when using TFS source control.

To add an environment-specific app.config or web.config file:

1. Browse to the folder where your shared app.config or web.config file is located.
   
   It should be located in a subfolder under the _ConfigFiles folder for the product dependency folder that contains your Visual Studio solution.
2. Create the new app / web config file in this folder and prefix the file name with the environment name (this can be anything, e.g. ‘production’ or a specific machine name)
   
3. For each machine that should use this configuration, specify the new environment-specific web / app config file name in the machine configuration file.
   
   Note that a sample.production.machine.config file is supplied with the Factory.

Now when you deploy a deployment package on any of these machines, or when you build a solution in Visual Studio on any of these machines, the environment-specific version of the config file will be deployed to the runtime locations as the web.config or app.config file.

To add a new environment to MS Enterprise Library settings:

We leverage the built-in EntLib feature ‘Environmental Overrides’.

1. Select ‘Edit Enterprise Library product configuration’ in the Factory Guide:
2. Create a new Environment, e.g. ‘Test’:
   

   Be sure to follow the naming convention for the environment-specific files as indicated below:
   
   When you click ‘Save’, the environment delta file will be created:
   

3. Now you can specify an override for any EntLib setting, for the Test environment. First select the setting, then select the value ‘Override Properties’ for ‘Override on Test’, and finally supply the override value for the setting.

   E.g. to override a database connection string:
   

4. Now add an override value for the Test environment for the Application Setting ‘MastEnvironmentMachineNames’:
   
   The MastEnvironmentMachineNames value is a comma-separated list of names of the machines that make up this environment. Wildcards * and ? are supported. These are the same machine names that are used in the filename of the machine configuration files.

   Note that when a machine name does not match any override value of MastEnvironmentMachineNames, the base EntLib configuration without any overrides will be used.

5. Select ‘Save’ to save all override settings in the Test.dconfig file. Also right-Click the (Test) environment and select ‘Save Merged Configuration’ to create the environment-specific EntLib configuration file :
   
   
   Don’t forget to add both the <Environment>.dconfig and the Entlib-<Environment>-Product.exe.config to source control.

Now the correct environment-specific version of the EntLib configuration file will be deployed when you deploy a MAST package on a machine, or when you select the ‘Deploy Enterprise Library product configuration’ action in the Factory Guide:

To edit or add values in an existing environment-specific EntLib file:

1. First open the base EntLib configuration via the ‘Edit Enterprise Library product configuration ‘ action in the Factory Guide. It will open without any environment overrides loaded.
2. Then right-click the Environments node, select ‘Open Environment Delta’ and open the appropriate <Environment>.dconfig file:
   
3. Once you are done editing and adding new override values, save both the Entlib .dconfig file and the merged configuration file again.

I just finished setting up BizTalk 2009 (Developer Edition) on a W2008R2 x64 machine with the following components:

Initially, the BizTalk Server Configuration utility failed to install the BAM portal. The error log reported that IIS was in 64 bit mode where 32 bit mode was required.

MSDN guidance on configuring the BAM Portal said that in IIS 6 you should execute this script:

cscript c:\inetpub\adminscripts\adsutil.vbs SET W3SVC/AppPools/Enable32bitAppOnWin64 1

However, it also mentioned this:

However, the MSDN guidance did not specify what you should do if you use IIS 7.

After I found out that in IIS 7 you can specify per application pool whether you want to enable 32 bits processes, I found a clean and easy way to get the BAM Portal to install correctly:

1. In IIS Manager, select Set Application Pool Defaults:
   
2. Set the property “Enable 32-Bit Applications” to True:
   
3. Run the BizTalk Server Configuration utility and configure the BAM Portal.
4. If you want, you can now change the Application Pool Defaults back to what you prefer.

NJoy the green checkmarks.

In 64-bits versions of Windows Operating Systems, a subsystem called Windows on Windows 64 (WoW64) enables you to run 32 bits applications. Even to date, many Windows applications only exist in a 32 bits version – not the least of which is Microsoft’s own Visual Studio(!)

The purpose of WoW64 is to hide many of the structural differences between 32 bits and 64 bits Windows for 32 bits applications. This is a good thing. Most of the time, anyway…

However, when your PowerShell script runs in a 32 bits process on a 64 bits OS, and you want to integrate with 64-bits applications (e.g. you want to integrate SharePoint or SQL Server within Visual Studio), WoW64 can get in the way.

E.g., WoW64 hides the 64 bit registry and changes the environment variables for processor architecture and for the program files folder – in short, WoW64 hides that you are running on a 64 bits OS.

To detect a 64 bits OS from PowerShell on Vista, W2008 and later Windows versions, we could use WMI:

(Get-WMIObject win32_operatingsystem).OSArchitecture

However, this is not supported on W2003.

Fortunately, Microsoft provided an alternative, which is supported on all Windows versions that have Wow64: the IsWow64Process function. Since this is an unmanaged API call to Kernel32.dll, we need to do some extra work to call this in PowerShell.

We can write a simple managed C# wrapper around IsWow64Process:

using System;
using System.Runtime.InteropServices;
 
public class Kernel32
{
    [DllImport("kernel32.dll", SetLastError = true, CallingConvention = CallingConvention.Winapi)]
    [return: MarshalAs(UnmanagedType.Bool)]
    public static extern bool IsWow64Process([In] IntPtr hProcess, [Out] out bool lpSystemInfo);
 
    // This overload returns True if the current process is running on Wow, False otherwise
    public bool CurrentProcessIsWow64()
    {
        bool retVal;
        if (!(IsWow64Process(System.Diagnostics.Process.GetCurrentProcess().Handle, out retVal))) { throw new Exception("IsWow64Process() failed"); }
        return retVal;
    }
}

When we use the PowerShell InvokeCSharp function, we can put this C# inline in our PowerShell script:

Function global:CurrentProcessIsWOW64
{
    # Use some inline C# code to call the unmanaged IsWow64Process() function in Kernel32.dll and return its result:
    $code = @"
        using System;
        using System.Runtime.InteropServices;
 
        public class Kernel32
        {
            [DllImport("kernel32.dll", SetLastError = true, CallingConvention = CallingConvention.Winapi)]
            [return: MarshalAs(UnmanagedType.Bool)]
            public static extern bool IsWow64Process([In] IntPtr hProcess, [Out] out bool lpSystemInfo);
 
            // This overload returns True if the current process is running on Wow, False otherwise
            public bool CurrentProcessIsWow64()
            {
                bool retVal;
                if (!(IsWow64Process(System.Diagnostics.Process.GetCurrentProcess().Handle, out retVal))) { throw new Exception("IsWow64Process() failed"); }
                return retVal;
            }
        }
"@
    InvokeCSharp -code $code -class 'Kernel32' -method 'CurrentProcessIsWow64'
}
 
Function global:ProcessArchitecture
{
    switch ([System.IntPtr]::Size)
    {
        4 { 32 }
        8 { 64 }
        default { throw "Unknown Process Architecture: $([System.IntPtr]::Size * 8) bits" }
    }
}
 
Function global:OSArchitecture
{
    if (((ProcessArchitecture) -eq 32) -and (CurrentProcessIsWOW64)) { 64 } else { ProcessArchitecture }
    # Note that on Vista, W2008 and later Windows versions, we could use (Get-WMIObject win32_operatingsystem).OSArchitecture.
    # We don't use that because we also support W2003
}

So there you have it, some simple PowerShell functions to detect the process architecture and the OS architecture, regardless of whether or not you are running under WoW64.

P.S. I also added these functions to the global functions in the Macaw Solutions Factory, so they can be leveraged by anyone who wants to extend the Factory.

Out of the box PowerShell 1.0 makes it very easy to call any managed code in an assembly. However, one of the strong points of a scripted language such as PowerShell is that you can debug and extend it in any environment without the need for development tools or additional source code. This is especially valuable in scripts that integrate heavily with production environments, e.g. with automated deployment scripts such as we use in the Macaw Solutions Factory.

Ideally you would want the ability to call any C# source code from PowerShell, either inline or as a separate source file, effectively adding the maintainability benefits of a scripted language to C#. Based on a library created by Lee Holmes I created an improved PowerShell 1.0 function called InvokeCSharp that does just this.

Note that in PowerShell V2, this functionality is available through the built-in Add-Type cmdlet.

InvokeCSharp is completely self-contained and can be used in any PowerShell script.

Usage

Here are some simple examples on how to use InvokeCSharp:

Let’s say we have a file MyClass.cs:

using System;
 
public class MyClass
{
    public string AString;
 
    public MyClass(string aString)
    {
        AString = aString;
    }
}

Then we can invoke it like this in PowerShell:

# Create a class instance from a .cs file, passing a parameter to the class constructor:
# Note that the class name can be omitted here, since it is implied from the .cs file name
$myClassInstance = InvokeCSharp -file '.\MyClass.cs' -parameters 'A text'

Or we can put some C# code inline in the PowerShell script itself, and invoke it:

# Call some simple inline code. Note that you could also wrap the code in a namespace,
# add using statements and reference other assemblies.
$code = @"
	public class MyClass2
	{
		public static int MyStaticMethod(int a, int b)
		{
			return a * b;
		}
	}
"@
$result = InvokeCSharp -code $code -class 'MyClass2' -method 'MyStaticMethod' -parameters 3, 7 # Will return 21

Although these are very simple examples, all the rich goodness of PowerShell – .NET integration is available. Passing any type of parameters back and forth is simple and completely transparent. All C# features can be used; e.g. you can use the C# DllImport attribute to call unmanaged API’s when COM interop is not available or not preferred.

After initial compilation the performance is equivalent to calling compiled assemblies from PowerShell.

Extending the Macaw Solutions Factory PowerShell scripts with inline C#

I added InvokeCSharp to the global functions in the Macaw Solutions Factory. This makes it possible to simply extend or modify any Factory script – including the scripts that deploy a MAST Product from a package to (production) machines.

For anyone who is more adept with C# than with PowerShell this will make it easier to extend the Factory scripts. This also enables leveraging existing C# libraries from PowerShell while keeping the entire solution maintainable as a script.

Improvements

The improvements I added to InvokeCSharp when compared to the original version are:

• The C# code you specify is completely self-contained; no more C# wrapper code where your code is pasted into. This improves readability and maintainability of inline C# code.
• Simpler parameter passing; no more wrapper code needed for parameters and return values.
• Automatically un-wraps parameters that PowerShell wraps in a PSObject. This eliminates the need to explicitly cast parameters to make them strongly typed on each call to InvokeCSharp, and to always wrap them in @().
  Figuring this one out was interesting; it required diving in the innards of the PowerShell Extended Type System (ETS) which gets in the way when you work with .NET reflection classes from PowerShell. PowerShell tries to make the ETS transparent to the PowerShell user, which in general is a good thing but in this case meant it was hard to see what was happening.
• Adds support for calling static methods, including overloaded methods, without creating a class instance.
• Adds support for creating a class instance and providing constructor parameters.
• Caches compiled assemblies instead of class instances. This improves performance because it eliminates duplicate compilation when the same C# code is called more than once. In addition to assembly caching you can of course also cache the object instances using any of the standard PowerShell mechanisms; however this is better handled outside the InvokeCSharp function.
• Support for inline C# and separate C# source files; the latter allows you to easily edit the C# in Visual Studio, e.g. to use IntelliSense and compile the source to catch compile time errors.

PowerShell Source

So here are the goods – the complete PowerShell source for InvokeCSharp:

#< # #.Synopsis #   Create a class instance and/or invoke a method on C# source code. #   For details, see: #   http://vincenth.net/blog/archive/2009/10/27/call-inline-c-from-powershell-with-invokecsharp.aspx  #.Description #   This function compiles the provided C# source code (only if necessary, compiled #   assemblies are cached). The code can be supplied either inline in a string or #   in a source file. Relative source filename paths are interpreted #   relative to the parent folder of the calling script. #   This enables simple organization and reference of related .ps1 and .cs files. # #   If a method name is specified, that method is invoked with any supplied #   parameters and the function then returns the result of the method. #   Both static and nonstatic methods are supported, including overloaded #   methods. # #   If no method name is specified, an instance of the specified class is #   created, passing any supplied parameters to the class constructor, and the #   function then returns the class instance. You can then program against the #   class instance in standard PowerShell fashion. # #   If necessary for compilation, you can specify names of any referenced #   assemblies. #.Parameter code #   The complete C# source code - effectively this is an inline C# source file. #   Note that it is possible but not necessary to declare classes within a #   namespace. #   Either the code parameter or the file parameter must be specified. #.Parameter file #   Absolute or relative path to a C# source file. Relative paths (beginning #   with '.\' or '..\') are interpreted relative to the folder that contains #   the calling script (i.e. the script that calls this function). #   Either the code parameter or the file parameter must be specified. #.Parameter class #   The name of the class to be instantiated / that contains the method to #   be invoked. Include the namespace in which the class is declared, if any. #   If you specify the -file parameter and the class name is identical to the #   file name, you can omit the -class parameter. #.Parameter method #   Either the name of the method to be invoked, '()' to invoke a  #   constructor to create and resturn a class instance, or '' to return #   the cached assembly. The last option allows you to precompile assemblies #   without calling a method or creating a class instance. #     #   If the specified method is not static, an instance of the class will #   be created on the fly. #.Parameter parameters #   To pass parameters to the constructor or method, specify them as an #   array of objects (e.g. with the comma operator or @() ). #.Parameter reference #   If necessary, specify an array of assembly filenames to be added as #   assembly references when the source code is compiled. #.Parameter forceCompile #   Specify this switch to force recompilation. This is useful if you specify #   the file parameter and the file contents has changed (when specifying a #   source file name, the cache key is the file name only - changes to the file #   do not automatically cause cache invalidation). #.Returns #   Either the cached assembly, a class instance or a method result (depending  #   on the value specified for the method parameter). #.Example #   # Create a class instance from a .cs file, passing a parameter to the class constructor: #   # Note that the class name can be omitted here, since it is implied from the .cs file name #   $myClassInstance = InvokeCSharp -file '.\MyClass.cs' -parameters 'A text' #     #   # Call some simple inline code. Note that you could also wrap the code in a namespace,  #   # add using statements and reference other assemblies. #   $code = @" #       public class MyClass2 #       { #           public static int MyStaticMethod(int a, int b) #           { #               return a * b; #           } #       } #   "@ #   $result = InvokeCSharp -code $code -class 'MyClass2' -method 'MyStaticMethod' -parameters 3, 7 # Will return 21 ##>
function global:InvokeCSharp
{
    param(
        [string] $code = '',
        [string] $file = '',
        [string] $class = '',
        [string] $method = '()',
        [Object[]] $parameters = $null,
        [string[]] $reference = @(),
        [switch] $forceCompile
    )
 
    # Stores a cache of generated assemblies. If this library is dot-sourced
    # from a script, these objects go away when the script exits.
    if(-not (Test-Path Variable:\macaw.solutionsfactory.assemblycache))
    {
        ${GLOBAL:macaw.solutionsfactory.assemblycache} = @{}
    }
 
    if (($code -eq '') -and ($file -eq '')) { throw 'Neither code nor file are specified. Specify either one or the other.' }
 
    # If a source file was specified, see if it was already loaded, compiled and cached:
    if ($file -ne '')
    {
        if ($code -ne '') { throw 'Both code and file are specified. Specify either one or the other.' }
 
        # We interpret the current directory as the directory containing the calling script, instead of the currect directory of the current process.
        if ($file.StartsWith('.'))
        {
            $callingScriptFolder = Split-Path -path ((Get-Variable MyInvocation -Scope 1).Value).MyCommand.Path -Parent
            $file = Join-Path -Path $callingScriptFolder -ChildPath $file
        }
 
        # If no class name is  specified, we assume by convention that the file name is equal to the class name.
        if ($class -eq '') { $class = [System.IO.Path]::GetFileNameWithoutExtension($file) }
 
        # Use the real full path as the cache key:
        $file = [System.IO.Path]::GetFullPath((Convert-Path -path $file))
        $cacheKey = $file
    }
    else
    {
        # See if the code has already been compiled and cached
        $cacheKey = $code
    }
    if ($class -eq '') { throw 'Required parameter missing: class' }
 
    # See if the code must be (re)compiled:
    $cachedAssembly = ${macaw.solutionsfactory.assemblycache}[$cacheKey]
    if(($cachedAssembly -eq $null) -or $forceCompile)
    {
        if ($code -eq '') { $code = [System.IO.File]::ReadAllText($file) }
        Write-Verbose "Compiling C# code:`r`n$code`r`n"
 
        # Obtains an ICodeCompiler from a CodeDomProvider class.
        $provider = New-Object Microsoft.CSharp.CSharpCodeProvider 
 
        # Get the location for System.Management.Automation DLL
        $dllName = [PsObject].Assembly.Location
 
        # Configure the compiler parameters
        $compilerParameters = New-Object System.CodeDom.Compiler.CompilerParameters 
 
        $assemblies = @("System.dll", $dllName)
        $compilerParameters.ReferencedAssemblies.AddRange($assemblies)
        $compilerParameters.ReferencedAssemblies.AddRange($reference)
        $compilerParameters.IncludeDebugInformation = $true
        $compilerParameters.GenerateInMemory = $true 
 
        # Invokes compilation.
        $compilerResults = $provider.CompileAssemblyFromSource($compilerParameters, $code)
 
        # Write any errors if generated.
        if($compilerResults.Errors.Count -gt 0)
        {
            $errorLines = ""
            foreach($error in $compilerResults.Errors)
            {
                $errorLines += "`n`t" + $error.Line + ":`t" + $error.ErrorText
            }
            Write-Error $errorLines
        }
        # There were no errors.  Store the resulting assembly in the cache.
        else
        {
            ${macaw.solutionsfactory.assemblycache}[$cacheKey] = $compilerResults.CompiledAssembly
        }
 
        $cachedAssembly = ${macaw.solutionsfactory.assemblycache}[$cacheKey]
    }
 
    # Prevent type mismatch issues caused by PowerShell wrapping of managed objects in PSObject.
    # We need to explicitly unwrap those objects because otherwise the .NET reflection classes will
    # not find the constructor or method whose signature matches the specified parameters.
    # This unwrapping eliminates the need to always wrap all your parameters in @() and to explicitly
    # cast each parameter to the correct type in each call to InvokeCSharp.
    if ($parameters -ne $null)
    {
        for($i = 0; $i -lt $parameters.Length; $i++)
        {
            $parameters[$i] = [System.Management.Automation.LanguagePrimitives]::ConvertTo( `
                $parameters[$i], `
                [System.Type]::GetType($parameters[$i].GetType().FullName) `
            )
        }
    }
 
    if ($method -eq '') # We return the assembly
    {
        $result = $cachedAssembly
    }
    elseif ($method -eq '()') # We create and return a class instance
    {
        $result = $cachedAssembly.CreateInstance($class, $false, [System.Reflection.BindingFlags]::CreateInstance, $null, $parameters, $null, @())
    }
    else # We invoke the method and return the method result
    {
        $classType = $cachedAssembly.GetType($class)
 
        $parameterTypes = @()
        if ($parameters -ne $null) { foreach($p in $parameters) { $parameterTypes += $p.GetType() } }
 
        $methodInfo = $classType.GetMethod($method, [System.Type[]]$parameterTypes)
        if ($methodInfo.IsStatic)
        {
            $instance = $null
        }
        else
        {
            $instance = $cachedAssembly.CreateInstance($class, $false, [System.Reflection.BindingFlags]::CreateInstance, $null, $null, $null, @())
        }
        $result = $methodInfo.Invoke($instance, $parameters);
    }
 
   return $result
}

With Create Component you can quickly add instances of the sample template components to your existing solution. You simply select the component type and specify the name of the new component; all other actions are automated. The component is created automatically in the correct project and folder, its namespace is set accordingly, and if the correct project is not present in the solution it is added on the fly with all correct namespaces, project settings etc.

Create Component is a feature that was added to the Macaw Solutions Factory in version 2.1.

Note: there are several series of Macaw Solutions Factory posts, on multiple blogs.
You can read all posts on the Macaw Solutions Factory site, or use this feed.

Basically, the Create Components feature allows you to think and work quickly at the logical level when creating components. You do not need to think about which project, folder, project items or settings are needed to implement the component. Instead, all that stuff is generated for you; you can start directly with implementing the component’s internals.

The Create Component feature only works on MSBuild-based projects, so this feature is not available in the Bts2006 and the Sql2008 or Sql2005 SS*S solutions.

This is how Create Component works in detail:

1. You start by creating a baseline solution with the MAST Solution Builder, by selecting an action in the Factory Guide under ‘Create Functional Area’.
   
   All you need to do now is type the name of the new Functional Area (VS solution), and select which components types (VS projects) the Functional Area should contain initially.
   Note that you only need to select the component types (projects) of which you know for sure that you will need them; you can add component types to an existing solution any time you want.
   When you click Create, the new solution is built and all namespaces, project settings etc are set to the correct values.After you close the solution builder window, you will be asked whether you want to remove the sample components and sample code from the new solution:
   

   And whether you want to open the new solution:
   

   After you make these choices, the Product.config file will be opened in VS instance where you selected the ‘Create Functional Area’ action, and the new Functional Area will be added automatically to the first non-Templates machine role:
   
   If you have more than one machine role – other than Templates – in your product, you may want to move the Functional Area to a different machine role now.

2. When the new solution is opened in VS you will see a node ‘Create Component’ in the Factory Guide, with a lot of actions in it:
   In any non-Templates solution these Create Component actions are generated on the fly, based on the Sample components in the Templates Functional Area for the current Product Dependency.
   E.g. in the DotNet3Templates.sln you can easily recognize the sample components that correspond to the actions in above screenshot:
   

   For each project item in a template solution whose name starts with ‘Sample’, an action is generated under Create Component. The name of the action specifies both the project name and the component name (without ‘Sample’).
   If a template project does not contain any sample components, an action is generated to only add the project to your solution (e.g. ‘New OperationalManagement’ in the above screenshot).

   You can easily modify the template solutions to change, add or remove sample components. You need to select the “Refresh ‘Create Component’ actions” action to regenerate the actions after you make this type of changes.

3. When you execute a Create Component action, you are prompted for the component name and everything else is done automatically:
   
   When you select OK, the component is added and it is opened in the editor:
   If necessary, the proper project for the new component will be created on the fly. You can see whether this will happen beforehand in the name prompt dialog:
   
   When you select OK, the MAST solution builder window will open and the new project will be created:
   
   As you can see in the log window the solution builder performs a lot of actions automatically, e.g. setting namespaces and project properties, and adding the project target to the FxCop project file for this solution.

   When you close the MAST solution builder window, the new component will be added and it is opened in the editor:
   

    

4. The new component that you added contains example code, to show you a typical implementation pattern.
   This sample code is marked with “region Sample” and “endregion Sample” texts – the exact syntax is specific for the file type (.cs, .xml, .sql, .aspx, .ascx etc.).
   If you do not need the sample code, you can quickly remove it from a new component by selecting the Factory Guide action “Remove Sample Content From Active Document”:
   

Even though the description of what Create Component does for you is quite long, using it is really quick and simple:

1. Click on the Create Component action for the component type that you want to add
2. Supply the name for the new component

Easy does it. Njoy!

The current version of SharePoint (Windows SharePoint Services 3 and Microsoft Office SharePoint Server 2007) delivers a powerful collaboration platform. It also offers excellent customization capabilities through configuration and custom code. This customization and configuration be done at a system level through server based installations, but also by power users through the web based site management and administration UI.

The power, ease and accessibility of customization in SharePoint encourages a quickly growing and diversifying collection of sites, through manual changes applied directly in the production environment. However this poses several big challenges for release management on SharePoint platforms, specifically:

• How to deploy and test new functionality through DTAP (Development, Test, Acceptation, Production) environments
• How to make new or changed functionality available on existing site instances
• How to handle content staging

Based on Macaw’s extensive SharePoint project experience, we have developed a strategy on how to address these challenges.

Deploying new functionality through DTAP environments

The key to DTAP-enabling SharePoint is to create automatic setup packages that leverage SharePoint’s built-in deployment technologies. Standard SharePoint solution packages (.wsp files) can carry content files, XML definitions and executable files.

In the Macaw Solutions Factory new functionality is implemented using SharePoint Features and (when needed) code. The Factory then automatically generates a WSP file, which in turn is embedded in a ZIP file, together with automatic (PowerShell) installation scripts. This ZIP file is also known as a MAST package (because it is based on the Microsoft .NET Architecture Style which is at the heart of the Factory).

In the development environment these deployment packages are generated; each package contains all information needed to deploy the relevant components of the solution on any machine in any environment. No user input is required.

The same (binary) package then travels through the DTAP environments, being deployed and tested for approval at each stage:

This approach has been used for years in many projects, and has proven to effectively DTAP-enable SharePoint solutions (as well as solutions based on other Microsoft server products).

Note: there are several series of Macaw Solutions Factory posts, on multiple blogs.
You can use this feed to track all Factory posts.

Maintaining existing site instances

The above Factory approach works well for deploying new functionality for new sites. However since at its heart the Factory leverages the standard SharePoint deployment mechanisms, this approach is limited to what type of changes can be deployed through these (XML driven) mechanisms.

E.g., a key XML structure in SharePoint is the Site Definition. Through Site Definitions you can make functionality available for new sites that will be created based on that new site definition. However, once a site has been instantiated, the SharePoint XML driven mechanisms do not allow you to modify the functionality defined by that XML. Changing a Site Definition after it has been instantiated does not modify the existing site instances, and doing this is unsupported by Microsoft – even when you only want to adjust new sites that will be instantiated after the Site Definition modification.

But there is an alternative: code that accesses the SharePoint object model can change any aspect of sites that are already instantiated. The idea is that to change functionality on (selected) existing sites, you do not write new XML features and site definitions for new sites. Instead, you write a piece of code that upgrades a site instance from version N to version N + 1. Then you automatically chain together all relevant updates to make the latest functionality available on any site.

This idea follows the same pattern as the well-known Continuous Database Integration (CDI) strategy (which incidentally is also used in the Factory for SQL Databases): you write small pieces of code that bring the functionality from version N to version N + 1; the tools do the rest.

Based on this idea, Macaw created the Site Management Tool. Like the other tools in the Factory, this approach is a combination of an implementation style (write upgrade code instead of XML definitions) with tooling to manage and automatically deploy the functionality to existing sites. The nice thing is that the same approach also works for new sites, so a specific piece of functionality only needs to be implemented once – either in standard SharePoint XML if no existing sites need to be upgraded, or in upgrade code otherwise. The SMT is itself implemented as a custom site in SharePoint:

The SMT offers considerable flexibility in controlling which updates will be applied to which sites at which moment:

• Which updates: the SMT allows you to define Update Profiles, which in essence is a set of changes grouped together. A site can be assigned multiple Update Profiles.
• Which sites to update: the SMT allows you to specify a Site Definition, a URL start or a part of the Site Title for each Update Profile, to select which site instances will be updated. In addition you can specify advanced logic in custom code to determine which sites to upgrade. So you can do this based on type of site, the area in your intranet (e.g. for a department), the site topic (a word in the Site Title) or custom logic:
  
• When to update: It is possible to have new sites, that are based on site definitions that you control, receive the relevant updates when they are created (this works with an update feature stapled to the site definition), to have updates distributed through scheduled SharePoint jobs, and on demand from the SMT site.

Schematically the SMT operates like this:

Like with CDI, this coded update approach does not allow automatic rollback or downgrade of functionality to a previous version if the data involved originates from the production environment.

However in our experience, in the rare cases that the rollback opportunities offered by standard SharePoint backups (site backups and database backups) are not sufficient, you can accomplish an downgrade by writing a specific new update to reverse the effects of the previous one.

The Site Management Tool is DTAP enabled by using the Factory to build the upgrade classes. The upgrade classes are then incorporated in the MAST package and are deployed throughout the DTAP environments, allowing the updates themselves to be tested before they are applied in production.

Content Staging

The last challenge is how to handle content staging in combination with a DTAP approach. Although not all SharePoint solutions require content staging, it is often unclear how to integrate content with the other types of changes deployed to a SharePoint solution.

Macaw’s approach to realize this is based on two choices:

• Have a separate staging environment that is treated as a second Production environment from the DTAP perspective
• Perform automatic content deployment from the staging environment to the production environment through the built-in SharePoint Content Deployment technology.

This effectively means that the production environment receives updates from two sources: content packages from the Staging Environment and functionality (MAST) packages via DTAP.

Conclusion

With these three major release management challenges addressed, Macaw feels confident that large SharePoint installations with a rich variety of quickly changing functionality can be effectively maintained.

Vincent Hoogendoorn

The Macaw Solutions Factory is quite a big thing, even though it fits both small and large solutions. In this post I will take you on a quick tour of the major systems and tools in the Factory. Later posts in the Factory Overview series will zoom in on the details of those systems and tools.

Note: there are several series of Macaw Solutions Factory posts, on multiple blogs.
You can use this feed to track all Factory posts.

In Factory Vision, Part 1: Introducing the Macaw Solutions Factory I described what the Factory does:

“Automate and standardize development and deployment on top of Microsoft Server products, targeting shared infrastructures with .NET, SQL Server, SQL BI, BizTalk and SharePoint”

and how the scope of the Factory is defined:

“All infrastructure, tooling and content used in Macaw’s primary process: Realize IT Solutions with Microsoft Technology”.

Given this it’s not really surprising that there’s too much in the Factory to describe in one post.
So without further ado, let’s start the Factory tour. I’ll present the tools in the chronological order in which they typically are first used during the application lifecycle.

Microsoft Virtual Machine Manager

One of the first things that you need to do when you start development on a new solution is to provide machines for the development and test environments. Usually you need at least as much machines as there are developers in the team, plus typically build and test servers. Since each machine logically belongs to one solution (not to a person), over time this can result in a lot of machines that must be kept available (for maintenance) but that only need to run occasionally. Virtualization offers great benefits here, allowing flexible creation and storing of machines, as well as additional benefits such as quick rollback to a checkpoint in case a machine installation gets corrupted.

In the Factory we use Microsoft’s Virtual Machine Manager, run on a powerful array of virtual machine hosts, to deliver self-servicing creation, access and management of virtual machines. In addition to fully configurable OS installations, we supply preconfigured machine templates for 32 bits and 64 bits versions of Windows Server 2003 and Windows Server 2008. New machines can be created in a matter of minutes.

The Factory Machine Installer

Virtual Machine Manager offers great flexibility through machine templates, but many Microsoft server products do not work well when they are preinstalled on such a template. This is due to the fact that these server products were not designed to handle a machine rename. Renaming machines happens when you instantiate a machine template – you need to select a unique name for the new machine for network access. At the OS level a tool named Sysprep handles this, but many Microsoft server products lack this kind of machine rename support.

In addition to that, a more flexible and less maintenance intensive mechanism is desired to deliver the desired combination of OS, server products and applications that are needed to develop or test a specific solution.

Enter Microsoft’s Deployment Toolkit. We use this toolkit to create and maintain the Factory Machine Installer, which allows users to select any desired combination of server products and applications from a single menu. With a single click all selected technologies are installed fully automatic, with all the latest updates. During the remainder of the installation (which can take up to several hours) no user interaction is needed. If a team needs to start (or is extended) tomorrow, you can create all machines simultaneously and have them ready for production work in the morning.

Automatically installing and retaining stable development and test machines saves a lot of time during the application lifecycle. In traditional situations it is not uncommon to spend several days before a new team member has a fully productive development machine for a specific solution.

For more details, see this post on the Factory Machine Installer by Maarten.

Microsoft Visual Studio

The Factory leverages the power of Microsoft Visual Studio, both its built-in functionality and its extensibility. Visual Studio Professional and Team System editions in VS 2005 and VS 2008 versions are fully supported (note that some MS Server products require using VS 2005 because they do not yet support developing in VS 2008). The Factory allows you to mix versions and editions, even within a single Factory product, as needed.

Microsoft Visual Studio Team System, including Team Foundation Server, is the application lifecycle management (ALM) platform of choice for working with the Factory. However we also support alternatives for version control (Subversion) and automated build servers (CruiseControl.NET Build Station).

The freedom of choice that the Factory provides with Visual Studio versions, editions and components, makes the Factory fit both small and large teams, solutions, and development environments.

The MAST standard

At the heart of the Factory design is the Microsoft.NET Application Style (MAST) standard. MAST standardizes both the logical design of an application built with the Factory, and how the logical design is translated to an implementation structure in a working folder. MAST is based on Microsoft Patterns & Practices guidance on application architecture, it has at its core the three-layered services application pattern:

(Note that MAST supports both the above pattern from the 1.0 guidance and the updated pattern from the 2.0 guidance, which adds a services layer and an application façade component type)

The MAST standard provides a standards-based, durable, stable yet flexible implementation structure that is consistent across Microsoft server products and technologies. MAST offers consistency, separation of concerns, alignment with underlying server products, and cost-effective automation through convention over configuration.

For solutions built with the Factory this results in very good maintainability. Experience also proves that the consistent implementation structure across solutions makes it much easier for developers to work on another solution or on multiple solutions.

The Factory IDE

The Factory Integrated Development Environment (IDE) extends Visual Studio with templates, tools and automation to create a highly productive development environment. Virtually all tools and templates leverage detailed knowledge of the standard MAST implementation structure, to achieve maximum automation. MAST significantly reduces the cost of implementing and maintaining the Factory IDE; it also increases the attainable level of automation.

Our choices on how we utilize the available Visual Studio extension mechanisms in the Factory IDE are guided primarily by pragmatism. Rather than see how far we can get with the ‘default’ technology for a specific type of extension, we consistently select the extension technology that is most effective and future proof for our purpose. This approach has resulted in some innovative and powerful Factory IDE features, such as integration with PowerShell and with the Confluence Wiki.

These are the main tools and templates in the Factory IDE:

Factory Product Wizard

In Factory terms, the thing that you build with one instance of the Factory is called a Factory Product. Since the architecture and the implementation structure of Factory products is standardized using the MAST standard, Factory products are also referred to as MAST products. Some examples of products that you can build with the Factory are:

For more details, see in this post under the heading “What can you build with the Factory?”.

Once you have a development machine installed with all required server products and applications, the next step when you start a new solution is to create its working folder. In addition to the source code, each working folder contains a copy of the Factory IDE. This allows integrated versioning of both the source and the development tools; it also makes product-specific Factory customizations easy to create and maintain. The Factory IDE is designed to run in multiple versions side-by-side, so independent versioning per Factory product is also possible in a shared development environment (see this post for more rationale on this).

The Factory Product Wizard lets you create a working folder for a new product, with (only) the tools and templates that you need for developing that product. Since the Wizard knows the MAST standardized working folder structure, which is designed in accordance to the convention over configuration principle, it requires very little user input to create a product working folder that has a consistent structure.

MAST Working Folder

Once the working folder for a Factory product has been created with the Factory Product Wizard, it will contain all the templates, tools and automation scripts that you need to start developing your product. A typical working folder created by the Wizard looks like this:

The templates in the MAST working folder are ready to be built and (automatically) deployed with a single action. This allows you to verify if your development machine is functioning end-to-end (using the automated system test suite included in the templates), even before you have written any code.

You can also automatically package and deploy the templates to other machines, e.g. a test machine. This means you get practically free deployment tests starting from day 1 of a project. No more ‘nasty surprises’ that are common when you do your first deployment much later, e.g. right before your product must be tested in a test environment, only to discover too late that something in that environment was not installed or configured correctly.

Factory MSBuild extensions

When you build a MAST Visual Studio solution, post build scripts are run via an MSBuild task that hosts PowerShell. These PowerShell post build scripts perform a rapid (incremental) deploy of the current Visual Studio solution to the local machine. Deployment is done from the working folder (where files are edited under source control) to the runtime deployment locations that are configured for the local machine:

Factory Guide Visual Studio Extension

Developers spend most of their time working in Visual Studio. To automate development tasks and offer guidance to developers while they are working in Visual Studio, we created the Factory Guide.

The Factory Guide shows actions that are relevant in the context where the developer is working in. The context consists of elements such as the part of the product they are working on (the Functional Area) and the components that are present in that part. These actions are implemented in PowerShell scripts (the Guide hosts PowerShell). These scripts have direct access to the Visual Studio object model, the working folder, and the server products running on the development machine.

This provides a very powerful and flexible automation environment. Existing scripts can be modified easily and new scripts and actions can also be added easily. PowerShell offers excellent integration with command-line tools, web services and managed code assemblies, so it is easy to leverage any type of asset in the scripts. Modifying and extending the Factory Guide is simply done in the working folder of the product being built; no additional Factory source and no Factory development environment is required, and versioning is simple.

Each action in the Guide is preceded by icons that indicate whether approved and/or community guidance on that action is available (in the central Factory Wiki):

This way, up-to-date guidance availability information is visible at a glance for all actions in the Guide – no need to click through on each action to see whether there actually is any guidance. Any Factory user can add community guidance to any action with a single click. This means that when a developer (specialist) in any project adds guidance about an action, this becomes visible to all Factory users in all projects. Since guidance is initially sparse and then is created on the fly and as needed, this leads to guidance that is both valuable (there is a proven need) and effective (directly visible to all Factory users).

We call this guidance approach: community driven guidance in context.

MAST Solution Builder

First you’ve automatically created and installed your development machines with VMM and the Factory Machine Installer, and then you have created a working folder on your new development machine for your new Factory product with the Factory Product Wizard. Finally you have automatically deployed and tested the templates that you are going to use. Due to the level of automation in these steps an experienced user will spend less than an hour on them.

At this point you are ready to define the baseline for your new Factory product – meaning that you create all Visual Studio solutions and projects that will be needed (as far as is known at this point in the project). Once you have added all Visual Studio solutions and projects to the working folder, you can add it to source control and let the team start fleshing out the various components.

We created the MAST Solution Builder to let you quickly create new solutions and projects (or add projects to existing solutions). With a single click the Solution Builder takes the selected template projects, creates a new solution (if needed) and instantiates those projects in the new solution:

Since the Solution Builder understands the standard MAST structure of solutions and projects in the Factory, it can automate many tedious and error-prone tasks when creating new projects:

• It creates the solution and project folders with the correct name
• It copies all selected template projects while removing example code
• It sets all namespaces and project properties to the correct names
• It sets all correct project and assembly references, and updates linked project items
• It creates an FxCop project file with all project assemblies added as target and with the MAST validation rules added (see MAST Validator below)
• It updates comment headers with the correct product information.

Since the MAST templates live in the same working folder as the product source, under single source control, the templates can easily be customized and maintained for a specific project. The Solution Builder will instantiate the customized templates when creating new Visual Studio projects.

The MAST Solution builder requires very little user input to do its work; you only need enter the Functional Area (~ VS Solution) name and select the project templates to add.

By applying the principle of convention over configuration, the MAST standard allows to automatically translate a logical design (consisting of machine roles, product dependencies, functional areas and component types) into an implementation structure (consisting of product- and machine- config files and Visual Studio solutions and projects).

All the above makes it possible to create a complete implementation baseline, starting from a logical (whiteboard) product design, in a matter of hours – even for complex, large solutions. At that point, the entire baseline will build and deploy automatically without errors. The team can get from logical design to fleshing out the baseline in one day, while using continuous deployment across development and test environments.

MAST Validator

MAST standardizes the logical design and many details of the implementation structure. But even though this offers a lot of value, it is not realistic to expect developers to know all details of the standard by heart, much less to apply the standard consistently. The MAST templates and the MAST Solution builder provide you with a MAST compliant baseline, but once you start fleshing out that baseline, how will you make sure that you will remain compliant to the standard?

This concern is even more pressing because the Factory IDE tools rely on MAST compliance. If your source is not MAST compliant, there is no guarantee that the Factory IDE tools (e.g. automatic deployment) will work correctly.

To address this concern we created the MAST Validator. As the name implies, the MAST Validator validates the source in the working folder against the MAST standard. We implemented the MAST Validator as an FxCop rule library:

The MAST Validator analyzes much more than typical FxCop rule libraries, which usually only analyze compiled assemblies. The MAST Validator also analyzes source files (including Visual Studio project files) and even databases.

Where applicable, MAST rules adjust themselves automatically to customized templates. E.g. in the example above, the project references in the template solution (which can be customized just as any other solution in a Factory product) are interpreted by the rule as the set of all potential valid project references. If a non-template solution contains any other project reference, the rule reports this as an error. This allows for automatically detecting unwanted dependencies, much like the Layering Diagram in VS 2010 will provide.

In case you wonder why we integrated the MAST rule library in FxCop instead of directly in Visual Studio, this is because FxCop provides us with additional UI that supports the audit process. In FxCop you can exclude rule messages with a reason, and you can see all exclusion reasons at a glance. When you want to do a quick audit of a product, you run FxCop, verify that no active MAST messages exist, and whether you agree with the rationale provided in the notes of excluded messages:

The Factory IDE will only create MAST deployment packages if there are no active MAST error messages; this effectively enforces MAST compliance throughout the application’s lifetime.

The MAST Validator helps you to keep your source MAST compliant, and to audit on MAST compliancy, with little effort. At the same time flexibility is preserved: you can ignore rule messages if you record a satisfying rationale to do so.

Factory Local Build Server

The Factory fully supports Visual Studio Team Foundation Server build servers. However as an alternative for smaller projects, a lightweight build server is also included in the Factory IDE:

The Factory Local Build Server can be run on any development machine. It offers several build types to produce a MAST Product package; with the package you can then deploy your product on all required machines in DTAP. The Factory Local Build Server is based on CruiseControl.NET Build Station; it is completely self-contained and is automatically configured for your product. It has no prerequisites apart from Visual Studio (IIS is not needed).

MAST Packages can be built from source control (so you know for sure all required files are present in SC) but also directly from the working folder. The last option allows you to create and test deployment packages even before local changes are committed to source control. This gives you the opportunity to fix deployment and integration issues before they can propagate via source control to other team members and/or environments.

The Factory Local Build Server provides the benefits of a basic build server for smaller projects without the need for additional machines, software, or configuration.

MAST Package Installer

Once you have produced a MAST Package with the TFS Build Server or with the Factory Local Build Server, you can use that single package to install your product on all machines across DTAP. Typically the debug or FileSystem packages are only deployed in development environments, while the Release SourceControl packages are deployed across test, acceptation and production environments.

The MAST package (which is a ZIP file) contains the complete Factory product as well as all information on what parts of the product should be deployed to which locations on each machine. Installing a package is simply a matter of unpacking the ZIP on each machine and then running the included Setup.cmd.

For environments with many machines deployment can be further automated to offer a central drop location for a new package to be deployed.

Since the package contains all deployment information on all machines, no user input is required. Not having to write, communicate and perform manual deployment instructions adds up to a lot of time saved during an application’s lifecycle; it also avoids many potential human errors. The deployment procedure is the same for first installs as well as updates; the logic in the package installer script automatically handles both situations correctly.

The MAST Package Installer is implemented as a set of PowerShell scripts and some integrated assemblies and command line tools. It is easy to add custom, product-specific deployment scripts by dropping in extra PowerShell scripts that follow a specific naming convention. The MAST Package Installer will then run your extension script at the appropriate moment(s) and pass it the relevant deployment information.

Factory Wiki

All written guidance in the Factory is maintained in the Macaw Solutions Factory Wiki. We set out to create an MSDN-like documentation system with both endorsed content and community content on each page. Since we want to have our guidance community-driven, we needed a Wiki with identical, rich content editing capabilities for both page content and user comments. After extensive research we selected Atlassian Confluence Enterprise.

The Wiki has good integration and extension capabilities; e.g. we used this to integrate the Factory Guide in Visual Studio to the Wiki. It also has very good import- and export capabilities (e.g. from MS Word and to PDF).

Because the Factory IDE integrates directly into one central Wiki, and any user can add rich comments on any page with very little effort, the Factory Wiki is an effective platform to quickly share knowledge amongst all Factory users.

Office Communications Server & Terminal Services Gateway

Also part of the Factory is our infrastructure to support the New World of Work. We enable Factory users to work together anytime from any location. We use Office Communications Server and Terminal Services Gateway to enable this:

A typical scenario that this infrastructure supports is:

• A Factory user needs assistance on an error that occurs in his development machine; he checks in Office Communicator which colleagues are available and then IM’s a co-worker.
• The co-worker needs to see more details so they remotely share the desktop of the user’s development machine, either using Terminal Services Remote Control on two Remote Desktop connections, or with desktop sharing in Office Communicator.
• At the same time they open a hands free voice channel via Office Communicator and/or mobile phone.
• Working on the same screen while talking, they research and resolve the issue together.

This type of ad-hoc remote pair working is incredibly more effective than documenting and communicating an issue via email (let alone traveling to the same location); it has proven a big time-saver.

Experience has learned – surprisingly enough – that working remotely together this way is actually more productive than sitting physically together at the same machine. This is because each person has his/her own display, keyboard and mouse while working together on the same virtual screen.

Factory Improvement Forum

The above Factory tools and systems help you to quickly realize IT solutions based on Microsoft server products. However, the Factory itself is also continuously developed to support new technologies and to incorporate newly learned best practices. To direct this development we created the Factory Improvement Forum, where Factory users can suggest ideas for new features and vote on them.

With this forum we can effectively prioritize new factory features. This helps to make sure that we are always working on the right new features, the ones that are most wanted and most valuable to the Factory users.

This is where the Factory both gets and delivers its value: to the people working in innovative projects, realizing real-world IT solutions.

Closing Remarks

If you are reading this, I guess you’re not put off by the substantial size of this overview and I’d like to thank you for your interest. The next posts in this series will be more focused and much shorter.

The post Factory Vision, Part 1: Introducing the Macaw Solutions Factory gives an overview of the vision that drives the Factory; this post gives an overview of the tools and systems in the Factory as it exists today.

The Factory is designed for continuous innovation, to integrate new (versions of) Microsoft technology, while gradually leaving obsolete technologies behind. Many new Factory Features are already envisioned – to support new technology, more team roles and more aspects of ALM. But also to further raise the level of abstraction and automation. The Factory is a powerful tool, whose development is guided by both pragmatism and the compelling vision as described by Jack Greenfield et al in ‘the’ Software Factories book:

The Factory approach has proven to be very valuable with today’s technology and level of effort. Even so I am convinced that in terms of attainable value, this is just the beginning.

Vincent Hoogendoorn

This is the start of a story about a three-year companywide Software Factory effort. The result is not your common code-generating or process-focused factory. In our effort to realize maximum value, we focused on solving common challenges that existing factories do not address.

We created a Factory to automate and standardize development and deployment on top of Microsoft Server products, targeting shared infrastructures with .NET, SQL Server, SQL BI, BizTalk and SharePoint:

Interested? Read on…

History

For the past three years it has been my job to move Macaw, a Microsoft Systems Integrator, to a software factory approach.

Macaw specializes in realizing IT solutions with the latest Microsoft technologies. While Macaw is organized in several Solution Centers that each specialize in specific types of IT Solutions, it is common for Macaw’s clients to run these different solution types on a shared infrastructure (deployed on the same set of machines) and a common Microsoft technology stack (deployed on the same Microsoft server product installations, e.g. SharePoint). We created a Factory to support this.

Three years ago, based on experience gained at Macaw’s (then) largest customer, I proposed a vision to create a single, durable software factory to support 80% of Macaw’s work as a Systems Integrator. The vision was recognized and adopted by Maarten Sikkema, co-founder and CTO of Macaw, and with his support things were set in motion. I became member of a dedicated Factory core team, which was tasked with designing, building and embedding the Macaw Solutions Factory company-wide.

I convinced Macaw’s key technology experts and development community leaders to participate in creating the core Factory standard – the Microsoft .NET Architecture Style (MAST). The entire Factory initiative is driven by voluntary participation (building the Factory) and adoption (using the Factory), as opposed to a mandatory approach. Although this did take a lot of extra effort and discussion to get people on-board, the resulting commitment was strong and durable because people really believed in the initiative and they felt it was also their factory. Durable community commitment is key for a durable factory approach; this is a necessary and worthwhile effort.

After the first year, Serge van den Oever joined the core Factory team. As Macaw’s resident SharePoint guru he started out adding SharePoint support to the Factory, but since then Serge continued to add many key innovative elements to the Factory (PowerShell integration, the Factory Wiki, and the Factory Guide Visual Studio extension).

As Factory usage became more and more widespread, an extended team of Factory Champions was created, with members from all solution centers. The Factory Champions were tasked with continually improving Factory usage and usability for their respective centers. Finally a strategy was put into place to integrate results of Macaw innovation projects into the Factory.

During this time, we consciously maintained an internal focus, supporting and strengthening Macaw’s capabilities as a Microsoft Systems Integrator.

Today

The Factory has been adopted throughout the company and has been sold under license for use by non-Macaw personnel. The people and processes to continually improve the Factory are in-place. We are integrating the latest Microsoft technologies as they are applied and we capture best practices as we learn them, across all solution centers. Now we feel it is time to share what we have learned and achieved.

In several factory blog post series, Maarten, Serge and I will cover the Factory vision, overview, features, usage and customer cases.
You can use this feed to easily track Factory posts by all contributors.

Vision

Ok, enough history. Since this is the first installment in the Factory Vision series, let me outline the key elements of our vision on a Software Factory for a Microsoft Systems Integrator.

Note: for this introductory post I will stick to the high-level view; it is long enough as it is. However we intend to quickly zoom in on the key details in the next posts, right down to the ‘bare metal’.

Design goals

When designing the Factory, we set these goals:

• Fit at least 80% of Macaw’s work as a systems integrator, specialized in the latest Microsoft technologies.
   
• Be effective for very small teams (1 – 2 persons) as well as large teams (15 persons).
   
• Support multiple, independently versioned solutions running on a shared infrastructure. Large projects or programs are often organized into multiple teams of 5-8 persons each, where all teams work on the same chain of DTAP (development, test, acceptation, production) environments. This means that the Factory must be versionable per solution; different factory versions must be able to live side-by-side in a shared development environment.
   
• Be effective for short, simple projects (a simple web site) as well as long, complex ones (a SOA).
   
• Deliver “Tomorrow’s IT, Today”,
  company-wide. Allow the knowledge and experience of Macaw’s top
  innovators who are working on projects to be captured with minimal overhead, and then distributed company-wide, quickly. This requires easy factory customizability in an ongoing project, frequent factory releases, and easy factory upgrades (also in ongoing projects).
   
• Be durable, meaning that the effort in maintaining the factory – integrating new technologies and practices as they are learned by Macaw – should remain constant, while the total generated value of the factory keeps increasing. This means embedding Factory maintenance and innovation in the organization, and partitioning the Factory components across the underlying Microsoft server products and across the different versions of those server technologies.

Overall vision

Overall we envisioned a durable, company-wide Factory approach that would relieve people from the same tedious, error-prone, repetitive tasks that they encounter on many projects. Instead, they are able to focus on what is specific and innovative for the current project, while the Factory provides them up-to-date, combined knowledge and experience of their best colleagues at their fingertips. This approach will bring every team closer to the level of the company’s very best combined; a dream team.

Core Factory value

Key to the value of the Factory is the value of Macaw’s best innovators, who are leaders in their respective fields, working in innovative projects. The Factory amplifies this value (making it available for other teams) and also makes it more durable (even after the expert leaves the company, the captured knowledge and experience remains available). In a market where top IT talent remains scarce, hard to come by and hard to retain, this Factory approach provides a significant strategic advantage for an innovative systems integrator.

Customers benefit from consistent high levels of productivity and expertise that – across the board – are simply unattainable for traditional systems integrators who still use a manual, one-off approach to development. When customers license the Factory for use by non-Macaw personnel, they boost those developers with Macaw’s packaged expertise.

What can you build with the Factory?

The first step in defining a Factory is describing what types of solutions you can build with it – what the targeted software product family is. We observed a large overlap across different solution types with respect to what Microsoft server products are used and what best practices are applied. In addition to that, different solution types often coexist on a single shared infrastructure.

So instead of building a separate factory for each type of IT solution, we decided to define a single Factory that can build:

“Solutions composed from a combination of custom .NET code and zero or more Microsoft server products”.

The phrase “zero or more” indicates that we target solutions ranging from the ‘traditional’ fully custom solutions to solutions based on standard server products. The latter type of solutions has become predominant for Macaw’s clients; as much as possible of the functionality is bought with standard server products. The remaining functionality is then realized by customizing and extending the standard products:

• Examples of custom components are web pages, console UI or windows UI, business components, workflows and business entities.
   
• Product extensions are specific types of components that Microsoft has defined to allow extending the functionality of their server products through code, such as web parts for SharePoint or pipeline components for BizTalk.
   
• Product configurations are product-specific settings and data that specify how to use the product’s existing functionality, such as configuration of web sites and web applications in IIS, site definitions for Moss 2007 and custom databases for SQL Server.
   

The Factory supports building small or simple solutions equally well as building large or complex solutions. The Factory is automatically configured to only include the tools and templates that you actually need for the solution at hand. If the solution is ‘light’, the factory will also be ‘light’, e.g. to build:

What does the Factory do for you?

Like most factories, this Factory increases developer productivity and quality by automating repetitive, error-prone development tasks. When building the above types of solutions, this is how the Factory addresses the most common big challenges for you:

What’s in the box?

So now we have seen what types of solutions you can build with the Factory, and which common challenges the Factory helps you address. Now it’s time to look at what exactly the Factory itself is. Here is our governing definition of the Factory:

As you can see this is quite a broad definition for a factory. We explicitly include not only the development tools and templates, but also all infrastructure that enables the Factory users to work (together) from any location. One of the main pillars of our factory is to support the “New World of Work”. This is part of our strategy to handle the challenge of distributed specialist knowledge.

How do we ensure sustained, increasing Factory value?

From the outset durability was a design goal for our entire factory approach. The main steps we took to realize this were:

• We embedded Factory usage and usability in the organization by creating the Factory Champions team and working closely together, while keeping Factory usage a choice.
   
• We proposed the Macaw Innovation Board, as a central place to provide overview and direction on all innovation projects. This allows to integrate and align innovation projects and the Factory to each other.
   
• We created a Factory Improvement Forum with voting, open to all Factory users, to allow community-driven feature proposal and prioritization. This ensures that we are always working on the right features, that are the most wanted and valuable for the project teams.
   
• We partitioned the Factory components across the underlying Microsoft server products and across the different versions of those server technologies, so we can distribute our maintenance effort proportionally to how much value is currently generated with a specific technology. Obsolete (versions of) technology can be removed in newer Factory versions very easily, while older Factory versions that still use that technology can continue to live side-by-side with newer Factory versions.
   
• We made the Factory implementation very open and easily modified and extended. We implemented a large part of the Factory functionality in open PowerShell scripts to allow quick customization without the need for separate Factory source code or a Factory development environment. Our solution templates are customized with the same ease as when coding in regular Visual Studio Solutions. This way the innovation can be driven by projects on the go, modifying their factory instance as they learn.
   
• We provide on-call consultancy from the core Factory team to help innovators extend the Factory within their project context in such a way that the result is suitable for other projects, and for integration in new Factory releases.
   
• We provide one-click functionality to any Factory user to share guidance on any Factory action, or to post remarks or requests to clarify guidance. The community contributions are then used to create or improve the approved guidance. This way we prevent a central documentation bottleneck and we prevent work on guidance that is not needed.
   
• The Factory IDE is versioned with each product that is built with it; multiple Factory versions can live side-by-side on the same machine. This way, factory upgrades and customizations can be done independently for each solution, even when the solutions are developed in shared DTAP environments. The impact of Factory upgrades is much less, because you only upgrade for projects if and when the benefits are significant in the current lifecycle stage of the solution.

Closing remarks

I hope this post gives you an impression of where we have come from and where we are heading with our Factory approach. For me it is a privilege and a joy to be realizing this vision.

I would like to take this opportunity to thank everyone who helped to make this possible. In particular I would like to thank Maarten Sikkema for his unwavering belief and support to let us realize this vision, and his personal, hands-on involvement. Also many thanks to Serge van den Oever for the great teamwork and his many innovative ideas. Thanks to the Factory Champions team who encouraged and helped their colleagues to use the Factory and who provide us with invaluable feedback and direction. Thanks to the innovators who volunteered to integrate their learnings in the Factory, and last but not least thanks to all Macawers who believe in and use the Factory.

Moving forward, realizing tomorrow’s IT, today, the road ahead beckons brightly. In this article I outlined the vision that resulted in the Factory as it exists today. More recent developments in Microsoft IT have inspired us to expand our vision. Specifically VS Team System 2010, Microsoft Azure, GoGrid, ElasticHosts, and textual and graphical modeling with M and Quadrant offer great potential for future Factory versions, to raise the abstraction level, target cloud solutions and offer factories in the cloud, and to target more team roles and application lifecycle management (ALM) aspects. But more on that at a later time…

Vincent Hoogendoorn

Steve Cook just announced that he is moving to the VSTS Architect Edition team, to work with Cameron Skinner building UML tools for Rosario on top of the DSL Tools. As you may recall, we can expect these five UML 2.1 diagram implementations in Rosario, built on top of the DSL toolkit:

• Class Diagram
• Use Case Diagram
• Component Diagram
• Sequence Diagram
• Activity Diagram

The fact that these are built on top of the DSL toolkit may provide interesting integration options with additional DSLs, e.g. if we can use the “Designer Bus” (a.k.a. “backplane”) for that (see the Software Factories in Rosario CTP post series by Edward Bakker for details: part 3, part 4 and part 5).

Stuart Kent gives some more hints in this post about interesting integration and extensibility options for these designers:

“Worth calling out are the features to allow a designer to be extended with new metadata once it has been deployed, and support for designers and models to interact. This will allow the designers delivered by Team Architect to be extended, and for new, third party DSLs to interact with them. Both these are key aspects of connecting UML with DSLs in the UML + DSL story that Cameron speaks about.”

I’m happy with this design decision by MS.

I like an approach that enables pragmatic solutions today while still moving toward the overall Factories vision. In realizing the factories vision at Macaw with the Macaw Solutions Factory, we use this type of pragmatic approach all the time, and it works great for us. I even think that such an approach is necessary if you ever want to actually arrive at the full vision (see my post on Factories 2.0).

It looks more and more like the Rosario release will really bring modeling to the masses, this will provide a lot more added value for the VSTS Architect Edition.

In the most significant post on factories to come out of Microsoft in a long time, Jack Greenfield unveils Software Factories 2.0. The software factories initiative has been repositioned within Microsoft to address a larger scope:

“Perhaps the most significant change is that the leadership of the Software Factory Initiative has moved from the Visual Studio Team Architect team to the Developer and Platform Evangelism (DPE) Platform Architecture Team. This move was designed to broaden the scope of the initiative”

Some of the things that now come into scope for the Microsoft Factories Initiative:

• “Leverage Microsoft products such as InfoPath and SharePoint“ This is a good thing! At Macaw, we designed the Macaw Solutions Factory to target all Microsoft server products right from the start. This proved to deliver much more value (for us as a systems integrator) than more common factories that focus mainly on modeling and code generation for (parts of) the .NET framework.

  Especially targeting SharePoint (WSS and MOSS) deployment and ALM aspects has provided great value for us, since this is a very challenging product in these area’s. Jack is not specific about whether “Leverage” means using these products to implement the factory itself and/or targeting solutions built on top of these products. I hope it will be both, of course.

• “integrating Software Factories with Software-plus-Services (S+S) Blueprints“
  Michael Lehman, who developed the project GlidePath infrastructure that underpins the existing blueprints, has recently moved to Jack’s new team. This will be an interesting marriage to watch!
  Jezz Santos mentions in “Factories & BluePrints – TechEd 2008” that Michael has already grabbed all Factories slots at TechEd Orlando, he seems to be moving fast.
• SOA Think Oslo…

And much, much more… which brings me to the next point:

Will “Larger” mean “Later”?
When you work on realizing the factories vision, as I have been doing with the Knowledge Development team at Macaw for some years now, one of the main challenges you have to deal with is scope.

Realizing the factories vision is most effective if you combine all aspects of the vision (think of the four pillars of software factories). If you have a company-wide initiative, your scope will eventually include all solutions technology as well as all people realizing solutions. Of these two, people require the bigger part of the effort [getting them to use and enhance the factory]. At least, this is what happened at Macaw.

Fortunately, at Macaw we were the right size of company to target our entire primary process (realizing IT solutions based on Microsoft technology) with one comprehensive factory initiative when we started. No “seperate kingdoms” of unaligned initiatives that we could not conquer had evolved – yet. We were lucky to be at the right place, at the right time, with the right idea.

Why am I telling all this? Because I recognize some of the same developments in this repositioning of the factories intiative at Microsoft. Microsoft moves to include more technology and - even more important- more people in their initiative. The move to DPE seems logical and even necessary. The “people” dimension of the scope at Microsoft is truly staggering, though. Which translates to very large efforts and very big inertia.

The real challenge in this Microsoft initiative will be to balance agility (regularly delivering tangible results that are most useful right now) with integration (aligning many Microsoft groups, products and initiatives to the right degree) and durability (earlier results not being invalidated by next steps). Because the factories vision takes such a long time to fully realize, a good balance on these three aspects is essential to both sustain the initiative and to actually arrive at the full vision.

If you misbalance, you run the risk to either fizzle out (lose momentum with people by taking too long – agility), disintegrate (collapse into a mess because directions of people and technologies are too disparate), or never reach your goal (one step forward, two steps back if next steps invalidate earlier efforts in the same initiative).

So is this factories thing a long shot?

In my opinion Microsoft is doing the right thing with this move, but you should appreciate that this initiative will take a long time to come close to realizing the full vision.

Does this mean that we should wait 10 years until all of this arrives? Definitely not, because significant benefits can be realized with what can be done today!

What I learned while working on Macaw’s factory initiative is that most important for progress and value is not substantial factory foundation technologies, but the core factory concepts and the mind shift of people that are both the factory users and it’s innovators: the people building IT solutions every day.

In my experience you can implement factory concepts with relatively light investments, using simple, existing Microsoft technology, today (more details on that in a later post). The real effort and time – but also value – is in the change in people’s mindset.

Strategy

Realizing the factories vision as a systems integrator can have a redefining effect. To realize a factories initiative and to get full benefits from it though, it needs to be a core part of the company strategy. This way, the technology, users, innovators as well as the business model all are guided to reap the very significant benefits. Building factories is not just about building a development tool…

We really are transforming the way IT solutions are being developed. When the industrial age of software development has arrived, will your company be part of it? Will your people have internalized the new paradigm? Will your business model allow you to maximize profit?

Will you be a factories 2.0 company? If you want to, my advice is - take your time: not in waiting for it, but to realize it. With the people and business model in place using current technology, you will be able to quickly leverage new factory foundation technology as Microsoft delivers on the vision, increasing your lead on competitors even further.

Edward Bakker announced the new software factory community portal which is located at sf.devrevolution.com. Edward asked me for feedback on the site early on, and was also kind enough to add my blog to the portal´s home page.

I am convinced the folks working on software factories can learn a lot from each other, and I’m all for sharing knowledge. So go check it out, be part of the community!

PS being Dutch is not a prerequisite ;-), this is intended as a global factory community portal.

Good work, Edward!

When I read Clemens’ post "Autonomous Develop Services for SOA Projects with Team Architect and Service Factory", I realized that part of his approach may integrate very well with the Macaw Solutions Factory.

Clemens describes how to enable each service in a SOA to be developed by a different team while still keeping the overall SOA design in one place. He partitions the SOA implementation across separate Visual Studio solutions (a single master solution containing the service models, and a separate solution for each service).

The Macaw Solutions Factory has had autonomous service development designed in from the start, it is a core concern addressed by the architecture style MAST (Microsoft .NET Architecture Style) that is at the heart of the factory. MAST also supports distributing a single large service across multiple solutions, and it specifies a standard pattern for dependencies within and across solutions. The factory supports independent versioning of each factory instance together with the service/application it is building while still using shared DTAP environments for the SOA. This makes service maintenance even more flexible and autonomous. So we got that part covered, it proved to work fine for a couple of years now.

Dependencies across projects and solutions in Clemens’ approach

Dependencies across solutions within a MAST service / application (dependencies across projects are standardized on the three-layered services application pattern).

However, Clemens also devised a way to keep the overall design of the SOA in a separate solution, by means of clever application of the P&P service factory and the distributed system designers in VS Team Architect. Overall SOA design is an area not yet addressed by the Macaw Solutions Factory (the focus is on building services and applications that are first-class SOA citizens, not on maintaining overall SOA design), and Clemens’ approach for that may integrate very well into our factory.

We already plan to integrate the Orcas version of the P&P Service Factory into the Orcas version of the Macaw Solutions Factory (we actually prototyped the approach during the Service Factory Customization workshop, Don and Olaf were a great help there). Thanks to the excellent extensibility of the service factory this is a real breeze (great work, guys!).

The extensibility built into the service factory looks to be a real enabler for the composability of a certain class of factories. It’s great to be able to combine approaches pioneered by different people in the factory community. Sometimes life is real easy

More goodness for factory builders: you can now start to target non-technical factory users with Visual Studio 2008 Shell, it was released on 2007/11/21. For details and download links see the post on the VSX Team Blog.

Use DSL’s and other Visual Studio extensions without scaring people with all the “developer UI” in Visual Studio, while retaining full integration options within Visual Studio.

See also my original post on using the shell for factories.

Some time ago, Jezz Santos blogged about how the software factory community, as he knew it, seemed to be contentrated in the Netherlands. That he mentioned me made me feel good of course, but I also realized that being close to so many leading people in the software factory field (outside MS that is), is a great opportunity to participate in a vibrant community.

Anyway, here I was six months later, at the Service Factory Customization workshop which is held in the Netherlands, which a good portion of the Dutch software factory community used as a pretext to exchange experiences and look at each other’s factories.

I had a great time there, some of the attendees were even allowed to present some of their own factory work (Don and Olaf were very accommodating to doing whatever gave us most value, which was really nice of them). I also presented a short flyby of the Macaw Solutions Factory (look for more posts on our factory here soon… no really). For some more details on the workshop and what we demoed to each other, see Serge’s post on it.

After the attendees presented their work, I asked Don about how the Dutch compare to what he had seen in the factories area globally, and Don confirmed what Santos said: the Dutch are at the leading edge. Sweet

So, I consider myself lucky, being at the right place at the right time. I’m really looking forward to what all these smart and passionate people are going to do to realize the factories vision, which is also my mission / job / hobby . We live in interesting times…

In fall 2007 Jack Greenfield published an article in which he includes a diagram of the factory schema that looks quite different from the one in his well-known software factories book. This is the “new” diagram from the article:

and this is the diagram as published originally in the book:

So this intrigues me… does the new diagram reflect new thinking on the factory schema? It seems to be a lot more oriented towards an implementation of a factory runtime and authoring environment. I wonder what this could mean for the new factory runtime and authoring environment in Rosario?

Until now you could only use DSL Designers in Visual Studio (Professional and higher editions). This limited using DSL’s to those development team roles that are comfortable using Visual Studio (developers and architects).

However, this has now changed with the announcement of the Visual Studio 2008 Shell. Now users can use the DSL Designers that run in Visual Studio in a stand-alone shell. So now we can also use DSL designers for supporting non-technical team roles (such as business analysts) in a software factory.

This is what a stand-alone DSL can look like:

Jezz Santos, MS Software Factories guru blogs about how the dutch seem to dominate the software factories community, mentioning me as well.

I already received happy emails from Edward Bakker and Dennis Mulder on this (15 minutes?) of fame

Good for the factory community!

Gunther Lenz and Christoph Wienands write a nice book on how to actually design and build a software factory on the .NET platform.

The book, Practical Software Factories in .NET, is based on a real, large project at Siemens, and it is a welcome companion to the original factories book by Jack Greenfield et al, which is very thorough but also quite abstract and elaborate.

Recommended reading for factory architects.

Jezz Santos, a factory builder who is leading in designing and applying software factories, predicts that Microsoft will deliver factories with their products as well. I completely recognize his reasons why that would be a Good Thing.

Jezz uses a Sharepoint Factory as an example. Which is interesting, considering that at Macaw we are building the Macaw Solutions Factory, which targets (amongst other Microsoft server products) SharePoint.

At the last TechEd in Barcelona, Jezz and I talked for about an hour on our plans for the Macaw Solutions Factory, and how Jezz would go about designing that. Apparently Jezz recognized a need…

Steve Cook mentioned my article on how we are realizing the software factories vision for Macaw. So this is why my article got so many hits lately. And as a boon, it gives us our 15 minutes of fame by displaying us on MSDN. I like it

[UPDATE this article was originally published here, I included it on this blog to get my story on factories in one place.]

Vincent Hoogendoorn, Architect, Macaw

April 2006

Summary

This article details a case on how the Software Factories vision is being pursued and realized by a Microsoft Systems Integrator specialized in .NET technology. A careful approach is detailed to gradually realize the vision, avoiding large investments and high risks while still reaping immediate and significant benefits. A detailed overview of the key first elements of the software factory is provided: a generic .NET architecture style, project templates, and a comprehensive collection of both standard and custom development tools. The experienced benefits are detailed and the potential is identified both for other systems integrators to apply this approach and for further development towards realizing the full software factory vision.

Introduction

As a Systems Integrator specialized in advanced Microsoft web technology, our larger projects reached a scale that really required standardizing and automating key aspects of our development process. Without this, scaling the development teams beyond a certain size made the development environments and -process unpredictable and inefficient. This particularly proved true for projects based on (combinations of) complex server products such as SharePoint, Commerce Server and BizTalk. To remedy this, we implemented processes, guidance, a standard architecture style, project templates and custom development tools. This approach proved very successful to stabilize and accelerate large projects. We soon realized that our approach could be generalized and made scalable, so it would benefit both small and large projects involving arbitrary combinations of Microsoft products. While studying key publications on the Software Factories vision [Greenfield04], we recognized that we in fact – in a modest way – were realizing part of that vision. We decided to embrace the wider vision and to devise a careful approach that would allow at least 80% of our development effort as a Microsoft Systems Integrator to benefit from the software factory approach.

Approach

We wanted a careful approach to realizing Software Factories, based on proven success, without taking large risks or making heavy investments. This led us to stage our approach in two phases.

The first stage is based on the proven success of our approach for large projects. The objectives we set for this stage are:

• Accelerate and scale development through standardization and automation.
• Create an optimal environment for harvesting and deploying reusable assets.
• Prove success in both small and large projects.
• Facilitate 80% of our development effort by the end of 2006.

This first stage focuses on reusing software structure, not functionality. We first want to create an environment of many products with a standardized software structure, which by itself has proved to provide a lot of value. However, such an environment will also be an optimal breeding ground for effectively reusable functionality.

The second phase of our approach takes off when sufficient functionality reuse has been recognized, proving the business case for reuse. At that point, we define one or more software product lines that – based on the common software structure –provide packaged and configurable functionality.

A key element of our approach is that we only package functionality for reuse after the potential for reuse has been demonstrated for two or three occasions within our domain. We do not want a big upfront design approach; reuse has to emerge.

Architecture Style

To realize our approach, we specified a standard architecture style for the software structure of our products. The family of systems that we target with this architecture style consists of all applications and services, built in .NET with C#, based on (combinations of, multiple versions of) Microsoft server products.

While analyzing the .NET products that we – as a Microsoft Systems Integrator – assembled since the platform became available, we recognized a single pattern for structuring software that not only proved to add significant value to our largest projects but that would also fit the vast majority of this family of systems well.

This pattern is known as the Three-Layered Services Application. Microsoft documents this pattern [MSDN05] and also provides extensive guidance on designing applications and services for .NET based on this pattern [PnP02]. Applications and Services designed according to this pattern do not only benefit from it from a stand-alone perspective, but they also integrate very well in a SOA, as detailed in the Connected Systems vision by Microsoft.

Software Structure in a Three-Layered Services Application

Since the Three-Layered Services Application pattern had proven to be valuable and durable and also appeared to apply throughout our domain as a Microsoft Systems Integrator, we decided to base our architecture style on it. To reflect that we target all of our .NET development with this style, we named it MAST (Microsoft .NET Architecture Style).

A key guiding principle for defining MAST is that functionality assumptions should be avoided. We focus on the Microsoft foundation products that we use and on the generic software structure in .NET. This is to promote a large enough scale for our (emergent) software product line(s) to be effective. Since we are a relatively small company, splitting up our economy of scope could very well reduce the advantages of the software factory approach to the point where it would become unprofitable.

The MAST standard specification was developed according to the Maturity Level for Work in Progress of the W3C Process Document [Jacobs05]. It serves both as architectural guidance and as a specification to build templates and tools for accelerating development.

We intend to publish the full MAST standard specification in the near future. However, to make this architecture style more concrete, we provide a fairly detailed overview of the specification here.

The main elements specified by the MAST standard are:

• Standard component types.
  A component taxonomy based on the Three-Layered Services Application pattern, extended with specialized component types for specific Microsoft foundation products (an example of a component type is a Business Entity, see the figure above).
• A standard namespace hierarchy.
  This hierarchy achieves a very good balance between consistency and flexibility in software structure. It promotes stability, reuse, low-cost refactoring and flexible deployment.
   
  Namespace Structure in MAST

  MAST specifies standard names for each namespace level that indicates a technical role, but not for the other (functional) namespace levels.

• A standard structure for solutions and projects in Visual Studio.
  
  MAST specifies that each component type maps to one Visual Studio project. The namespace level directly above the component type (either a functional area or a product dependency) maps to one Visual Studio solution. MAST specifies naming of solutions and solution items, projects, project folders and project items. Finally, MAST specifies Visual Studio project property values.
  Both VS 2003 (.NET 1) and VS 2005 (.NET 2) are supported.
• A standard structure for dependencies.
  
  Because each Visual Studio project contains components of a known type, MAST can specify all potentially valid project references within a Visual Studio solution.

  Combining relaxed layering (meaning a component may directly access all lower layers) with selected design pattern implementations has resulted in a matrix that specifies all potentially valid project references within a Visual Studio Solution.

  Dependencies between Visual Studio Solutions are restricted; only dependencies on the functional area named Framework are allowed. Since foundation products may have dependencies on each other (e.g. SharePoint Portal Server 2003 has a dependency on Windows SharePoint Services 2), dependencies on the framework of other product dependencies are also allowed. This results in a funnel-like structure for dependencies across visual studio solutions:
    
  Dependencies across Visual Studio Solutions in MAST
  
  A key benefit of this dependencies structure is that the impact of changes outside the framework is limited to a single visual studio solution. Since all solutions have the same structure (based on the three-layered services application pattern), refactoring functionality from solutions into the framework is very cheap; it can be done when the need for actual reuse in a second solution arises. This results in thin frameworks that only contain functionality that is actually used in two or more solutions.

• A standard folder structure for source.
  
  A key aspect of this folder structure is that a product dependency (e.g. DotNet2 or Wss2) is completely self-contained.
    
  Part of a product folder structure in MAST. Visible in the left pane are the folders for the DotNet2 product dependency with its configuration, shared assemblies and functional area’s (Framework and Templates). In the right pane the template solution with its DotNet2 specific project templates is visible.

Templates and Tools

Based on the specified standard architecture style, we created templates for visual studio solutions and projects for specific product dependencies. The templates provide software structure and only contain minimal functionality – just enough to allow actual deployment, running and testing. Separate versions of the templates were created for Visual Studio 2003 (.NET1) and Visual Studio 2005 (.NET2).

A template solution for the DotNet2 Product Dependency in MAST

As illustrated in the example above, the software structure defined by MAST is easily recognized in the templates. The product dependency (DotNet2) and the functional area (Templates) appear in the solution name. The layer and component type (e.g. Presentation.WepAppUI) appear in the project names, and the component subtypes (e.g. Pages) appear as folders within projects.

Post-build events for incremental deployment

The templates have configurable post-build events that perform quick incremental deployment to the local machine. By separating the deployment location from the editing location (e.g. for aspx pages) it is possible to do an automatic full deployment to a development machine at any time, without touching any development work (source) in progress. Developers then only need to rebuild the solution they are currently working on to integrate their changes locally. Even though we already use a continuous integration build server, we found that adding this approach significantly improves the predictability and efficiency in the development environment.

Tools
Because the .NET 2 based version of key Microsoft foundation products is still a while away (e.g. for SharePoint Portal) and we need to support 80% of our current development effort, we still need to fully support .NET 1 development with our tools (as well as .NET2). While technology currently available in Visual Studio 2005, Team System and the Guidance Automation Toolkit is very promising, we decided to start out with a toolset that supports both .NET versions. Once the volume of our .NET 2 development effort is big enough to justify the investment, we plan to create a GAT based version of the tooling. From then on, we will gradually shift our focus for further tools development towards the GAT based version.

We use the following standard tools in our automated build servers:

• CruiseControl for continuous integration build server.
• Subversion for source control.
• NAnt for automating build and deployment scripts.
• NUnit, MbUnit and IeUnit for automated testing.
• FxCop and NDepend for code analysis.
• NDoc for generating documentation.

To support and automate development using the templates and the specified architecture style, we also use the following custom tools:

• MAST Software Structure Generator.
  Based on the specified target namespace of a new component, this tool selects the appropriate template and uses that to generate the new component class in the specified namespace. If a new solution and / or project is needed to contain the new class, they are also generated on the fly. All project properties and references are then set according to the standard. We plan to add a VS 2005 front-end to the tool using the GAT.
• MAST Software Structure Validator.
  
  A tool – similar in operation to FxCop – that validates the software structure of a product by analyzing its source. All rules specified by the MAST standard are validated. As with FxCop, an online knowledgebase provides details and rationale for each rule; it also explains when each rule applies. Developers do not need to invest a lot of time in learning and remembering all details of the MAST specification – they learn from the validator’s feedback while developing the product.
• MAST Build & Deploy Script Generator.
  
  This tool generates post-build events for quick incremental deployment to the local development station (we separate the edit location from the deployment location for all artifacts). This tool also generates scripts for the build server. These scripts create builds, package them, and can be run on any machine in any environment to automatically deploy the full build. The tool uses machine-specific deployment configurations to generate and run the scripts. The configuration allows automatic deployment to different locations and deployment of different subsets of the product functionality, based on the machine environment, role and configuration.

Both structure generation and structure validation allow templates to be maintained and specialized with the product. This type of template maintenance is very quick, because the templates are just simple projects that live in the designated functional area named “Templates”, which – in a debug build – is automatically built, deployed and tested, just like any other code is.

Benefits

The key benefits we experience from this approach are:

• Because of the high consistency across projects, resource management can be done in a much more flexible manner. Dynamically varying team composition, size and locations (e.g. for virtual teams and outsourcing) becomes easier.
• Deployment (both incremental and full) is almost completely automatic; no hand-written deployment scripts are needed.
• Designing and maintaining the software structure in a consistent manner soon becomes second nature for all team members. Much less time is spent on arbitrary decisions and on translating different sets of concepts / styles between (replacement) team members.
• Communication within development teams is very efficient and unambiguous. The concepts defined in the architecture style become common language. This proves particularly valuable for communication between architects and developers.
• Because the architecture style avoids functionality assumptions, the software structure remains very stable when requirements change (over time).
• Because the same three-layered service application pattern is repeated in all functional areas (including frameworks), refactoring becomes cheaper. Refactoring now becomes a more lightweight, on-demand activity.
• Last but not least: because of the consistent structure across projects, recognizing, harvesting and generalizing functionality for reuse can be done very effectively.

The Road Ahead

As a Microsoft Systems Integrator we do not think we are unique. Our approach assumes the .NET platform and a team-based, project-based approach; however it avoids assumptions about functionality or methodology (which in our case is based on the Rational Unified Process). Our approach could therefore be valuable to other Systems Integrators as well, especially for companies that concentrate on Microsoft technology and use a team-based, project-based approach. It seems likely that good collaboration opportunities exist.

As a growing part of our development effort uses this approach, its benefits are expected to scale better than linear. Once the actual reuse of functionality reaches critical mass, we will define and implement one or more software product lines to package that generalized functionality, realizing even more benefits.

We will continuously improve and extend our factory assets (e.g. tools, templates, frameworks), moving towards the latest software factory development tooling as the market share of its supported platform grows.

The potential value of realizing the software factories vision remains compelling, although the road to get there will take some time.

References

[Greenfield04]
Software Factories – Assembling Applications with Patterns, Models, Frameworks, and Tools
Jack Greenfield, Keith Short, Steve Cook, Stuart Kent, John Crupi
Published by Wiley, ISBN: 0-471-20284-3

[Jacobs05]
W3C Process Document
Ian Jacobs, W3C
http://www.w3.org/2005/10/Process-20051014/

[MSDN05]
“Three-Layered Services Application” application architecture pattern
MSDN Library
http://msdn.microsoft.com/library/en-us/dnpatterns/html/ArcThreeLayeredSvcsApp.asp

[PnP02]
Application Architecture for .NET: Designing Applications and Services
Microsoft Patterns & Practices Group
http://msdn.microsoft.com/architecture/solvenow/solveimpdes/default.aspx?pull=/library/en-us/dnbda/html/distapp.asp

About the Author

Vincent Hoogendoorn is an architect at Macaw, a Systems Integrator specialized in advanced Microsoft web technology, based in the Netherlands.

His professional areas of interest are Solution Architecture, Service Oriented Architecture and Software Factories for the .NET platform. He recently was involved in projects for Smurfit- Kappa, Bouwfonds, Sara Lee and Akzo Nobel. Starting 2006, he is also member of Macaw’s Knowledge Development group, where he works on realizing the Software Factories Vision for Macaw.