VincentH on .NET

The current version of SharePoint (Windows SharePoint Services 3 and Microsoft Office SharePoint Server 2007) delivers a powerful collaboration platform. It also offers excellent customization capabilities through configuration and custom code. This customization and configuration be done at a system level through server based installations, but also by power users through the web based site management and administration UI.

The power, ease and accessibility of customization in SharePoint encourages a quickly growing and diversifying collection of sites, through manual changes applied directly in the production environment. However this poses several big challenges for release management on SharePoint platforms, specifically:

• How to deploy and test new functionality through DTAP (Development, Test, Acceptation, Production) environments
• How to make new or changed functionality available on existing site instances
• How to handle content staging

Based on Macaw’s extensive SharePoint project experience, we have developed a strategy on how to address these challenges.

Deploying new functionality through DTAP environments

The key to DTAP-enabling SharePoint is to create automatic setup packages that leverage SharePoint’s built-in deployment technologies. Standard SharePoint solution packages (.wsp files) can carry content files, XML definitions and executable files.

In the Macaw Solutions Factory new functionality is implemented using SharePoint Features and (when needed) code. The Factory then automatically generates a WSP file, which in turn is embedded in a ZIP file, together with automatic (PowerShell) installation scripts. This ZIP file is also known as a MAST package (because it is based on the Microsoft .NET Architecture Style which is at the heart of the Factory).

In the development environment these deployment packages are generated; each package contains all information needed to deploy the relevant components of the solution on any machine in any environment. No user input is required.

The same (binary) package then travels through the DTAP environments, being deployed and tested for approval at each stage:

This approach has been used for years in many projects, and has proven to effectively DTAP-enable SharePoint solutions (as well as solutions based on other Microsoft server products).

Note: there are several series of Macaw Solutions Factory posts, on multiple blogs.
You can use this feed to track all Factory posts.

Maintaining existing site instances

The above Factory approach works well for deploying new functionality for new sites. However since at its heart the Factory leverages the standard SharePoint deployment mechanisms, this approach is limited to what type of changes can be deployed through these (XML driven) mechanisms.

E.g., a key XML structure in SharePoint is the Site Definition. Through Site Definitions you can make functionality available for new sites that will be created based on that new site definition. However, once a site has been instantiated, the SharePoint XML driven mechanisms do not allow you to modify the functionality defined by that XML. Changing a Site Definition after it has been instantiated does not modify the existing site instances, and doing this is unsupported by Microsoft – even when you only want to adjust new sites that will be instantiated after the Site Definition modification.

But there is an alternative: code that accesses the SharePoint object model can change any aspect of sites that are already instantiated. The idea is that to change functionality on (selected) existing sites, you do not write new XML features and site definitions for new sites. Instead, you write a piece of code that upgrades a site instance from version N to version N + 1. Then you automatically chain together all relevant updates to make the latest functionality available on any site.

This idea follows the same pattern as the well-known Continuous Database Integration (CDI) strategy (which incidentally is also used in the Factory for SQL Databases): you write small pieces of code that bring the functionality from version N to version N + 1; the tools do the rest.

Based on this idea, Macaw created the Site Management Tool. Like the other tools in the Factory, this approach is a combination of an implementation style (write upgrade code instead of XML definitions) with tooling to manage and automatically deploy the functionality to existing sites. The nice thing is that the same approach also works for new sites, so a specific piece of functionality only needs to be implemented once – either in standard SharePoint XML if no existing sites need to be upgraded, or in upgrade code otherwise. The SMT is itself implemented as a custom site in SharePoint:

The SMT offers considerable flexibility in controlling which updates will be applied to which sites at which moment:

• Which updates: the SMT allows you to define Update Profiles, which in essence is a set of changes grouped together. A site can be assigned multiple Update Profiles.
• Which sites to update: the SMT allows you to specify a Site Definition, a URL start or a part of the Site Title for each Update Profile, to select which site instances will be updated. In addition you can specify advanced logic in custom code to determine which sites to upgrade. So you can do this based on type of site, the area in your intranet (e.g. for a department), the site topic (a word in the Site Title) or custom logic:
  
• When to update: It is possible to have new sites, that are based on site definitions that you control, receive the relevant updates when they are created (this works with an update feature stapled to the site definition), to have updates distributed through scheduled SharePoint jobs, and on demand from the SMT site.

Schematically the SMT operates like this:

Like with CDI, this coded update approach does not allow automatic rollback or downgrade of functionality to a previous version if the data involved originates from the production environment.

However in our experience, in the rare cases that the rollback opportunities offered by standard SharePoint backups (site backups and database backups) are not sufficient, you can accomplish an downgrade by writing a specific new update to reverse the effects of the previous one.

The Site Management Tool is DTAP enabled by using the Factory to build the upgrade classes. The upgrade classes are then incorporated in the MAST package and are deployed throughout the DTAP environments, allowing the updates themselves to be tested before they are applied in production.

Content Staging

The last challenge is how to handle content staging in combination with a DTAP approach. Although not all SharePoint solutions require content staging, it is often unclear how to integrate content with the other types of changes deployed to a SharePoint solution.

Macaw’s approach to realize this is based on two choices:

• Have a separate staging environment that is treated as a second Production environment from the DTAP perspective
• Perform automatic content deployment from the staging environment to the production environment through the built-in SharePoint Content Deployment technology.

This effectively means that the production environment receives updates from two sources: content packages from the Staging Environment and functionality (MAST) packages via DTAP.

Conclusion

With these three major release management challenges addressed, Macaw feels confident that large SharePoint installations with a rich variety of quickly changing functionality can be effectively maintained.

 

Vincent Hoogendoorn

This is the start of a story about a three-year companywide Software Factory effort. The result is not your common code-generating or process-focused factory. In our effort to realize maximum value, we focused on solving common challenges that existing factories do not address.

We created a Factory to automate and standardize development and deployment on top of Microsoft Server products, targeting shared infrastructures with .NET, SQL Server, SQL BI, BizTalk and SharePoint:

Interested? Read on…

History

For the past three years it has been my job to move Macaw, a Microsoft Systems Integrator, to a software factory approach.

Macaw specializes in realizing IT solutions with the latest Microsoft technologies. While Macaw is organized in several Solution Centers that each specialize in specific types of IT Solutions, it is common for Macaw’s clients to run these different solution types on a shared infrastructure (deployed on the same set of machines) and a common Microsoft technology stack (deployed on the same Microsoft server product installations, e.g. SharePoint). We created a Factory to support this.

Three years ago, based on experience gained at Macaw’s (then) largest customer, I proposed a vision to create a single, durable software factory to support 80% of Macaw’s work as a Systems Integrator. The vision was recognized and adopted by Maarten Sikkema, co-founder and CTO of Macaw, and with his support things were set in motion. I became member of a dedicated Factory core team, which was tasked with designing, building and embedding the Macaw Solutions Factory company-wide.

I convinced Macaw’s key technology experts and development community leaders to participate in creating the core Factory standard – the Microsoft .NET Architecture Style (MAST). The entire Factory initiative is driven by voluntary participation (building the Factory) and adoption (using the Factory), as opposed to a mandatory approach. Although this did take a lot of extra effort and discussion to get people on-board, the resulting commitment was strong and durable because people really believed in the initiative and they felt it was also their factory. Durable community commitment is key for a durable factory approach; this is a necessary and worthwhile effort.

After the first year, Serge van den Oever joined the core Factory team. As Macaw’s resident SharePoint guru he started out adding SharePoint support to the Factory, but since then Serge continued to add many key innovative elements to the Factory (PowerShell integration, the Factory Wiki, and the Factory Guide Visual Studio extension).

As Factory usage became more and more widespread, an extended team of Factory Champions was created, with members from all solution centers. The Factory Champions were tasked with continually improving Factory usage and usability for their respective centers. Finally a strategy was put into place to integrate results of Macaw innovation projects into the Factory.

During this time, we consciously maintained an internal focus, supporting and strengthening Macaw’s capabilities as a Microsoft Systems Integrator.

Today

The Factory has been adopted throughout the company and has been sold under license for use by non-Macaw personnel. The people and processes to continually improve the Factory are in-place. We are integrating the latest Microsoft technologies as they are applied and we capture best practices as we learn them, across all solution centers. Now we feel it is time to share what we have learned and achieved.

In several factory blog post series, Maarten, Serge and I will cover the Factory vision, overview, features, usage and customer cases.
You can use this feed to easily track Factory posts by all contributors.

Vision

Ok, enough history. Since this is the first installment in the Factory Vision series, let me outline the key elements of our vision on a Software Factory for a Microsoft Systems Integrator.

Note: for this introductory post I will stick to the high-level view; it is long enough as it is. However we intend to quickly zoom in on the key details in the next posts, right down to the ‘bare metal’.

Design goals

When designing the Factory, we set these goals:

• Fit at least 80% of Macaw’s work as a systems integrator, specialized in the latest Microsoft technologies.
   
• Be effective for very small teams (1 – 2 persons) as well as large teams (15 persons).
   
• Support multiple, independently versioned solutions running on a shared infrastructure. Large projects or programs are often organized into multiple teams of 5-8 persons each, where all teams work on the same chain of DTAP (development, test, acceptation, production) environments. This means that the Factory must be versionable per solution; different factory versions must be able to live side-by-side in a shared development environment.
   
• Be effective for short, simple projects (a simple web site) as well as long, complex ones (a SOA).
   
• Deliver “Tomorrow’s IT, Today”,
  company-wide. Allow the knowledge and experience of Macaw’s top
  innovators who are working on projects to be captured with minimal overhead, and then distributed company-wide, quickly. This requires easy factory customizability in an ongoing project, frequent factory releases, and easy factory upgrades (also in ongoing projects).
   
• Be durable, meaning that the effort in maintaining the factory – integrating new technologies and practices as they are learned by Macaw – should remain constant, while the total generated value of the factory keeps increasing. This means embedding Factory maintenance and innovation in the organization, and partitioning the Factory components across the underlying Microsoft server products and across the different versions of those server technologies.

Overall vision

Overall we envisioned a durable, company-wide Factory approach that would relieve people from the same tedious, error-prone, repetitive tasks that they encounter on many projects. Instead, they are able to focus on what is specific and innovative for the current project, while the Factory provides them up-to-date, combined knowledge and experience of their best colleagues at their fingertips. This approach will bring every team closer to the level of the company’s very best combined; a dream team.

Core Factory value

Key to the value of the Factory is the value of Macaw’s best innovators, who are leaders in their respective fields, working in innovative projects. The Factory amplifies this value (making it available for other teams) and also makes it more durable (even after the expert leaves the company, the captured knowledge and experience remains available). In a market where top IT talent remains scarce, hard to come by and hard to retain, this Factory approach provides a significant strategic advantage for an innovative systems integrator.

Customers benefit from consistent high levels of productivity and expertise that – across the board – are simply unattainable for traditional systems integrators who still use a manual, one-off approach to development. When customers license the Factory for use by non-Macaw personnel, they boost those developers with Macaw’s packaged expertise.

What can you build with the Factory?

The first step in defining a Factory is describing what types of solutions you can build with it – what the targeted software product family is. We observed a large overlap across different solution types with respect to what Microsoft server products are used and what best practices are applied. In addition to that, different solution types often coexist on a single shared infrastructure.

So instead of building a separate factory for each type of IT solution, we decided to define a single Factory that can build:

“Solutions composed from a combination of custom .NET code and zero or more Microsoft server products”.

The phrase “zero or more” indicates that we target solutions ranging from the ‘traditional’ fully custom solutions to solutions based on standard server products. The latter type of solutions has become predominant for Macaw’s clients; as much as possible of the functionality is bought with standard server products. The remaining functionality is then realized by customizing and extending the standard products:

• Examples of custom components are web pages, console UI or windows UI, business components, workflows and business entities.
   
• Product extensions are specific types of components that Microsoft has defined to allow extending the functionality of their server products through code, such as web parts for SharePoint or pipeline components for BizTalk.
   
• Product configurations are product-specific settings and data that specify how to use the product’s existing functionality, such as configuration of web sites and web applications in IIS, site definitions for Moss 2007 and custom databases for SQL Server.
   

The Factory supports building small or simple solutions equally well as building large or complex solutions. The Factory is automatically configured to only include the tools and templates that you actually need for the solution at hand. If the solution is ‘light’, the factory will also be ‘light’, e.g. to build:

What does the Factory do for you?

Like most factories, this Factory increases developer productivity and quality by automating repetitive, error-prone development tasks. When building the above types of solutions, this is how the Factory addresses the most common big challenges for you:

 

What’s in the box?

So now we have seen what types of solutions you can build with the Factory, and which common challenges the Factory helps you address. Now it’s time to look at what exactly the Factory itself is. Here is our governing definition of the Factory:

As you can see this is quite a broad definition for a factory. We explicitly include not only the development tools and templates, but also all infrastructure that enables the Factory users to work (together) from any location. One of the main pillars of our factory is to support the “New World of Work”. This is part of our strategy to handle the challenge of distributed specialist knowledge.

How do we ensure sustained, increasing Factory value?

From the outset durability was a design goal for our entire factory approach. The main steps we took to realize this were:

• We embedded Factory usage and usability in the organization by creating the Factory Champions team and working closely together, while keeping Factory usage a choice.
   
• We proposed the Macaw Innovation Board, as a central place to provide overview and direction on all innovation projects. This allows to integrate and align innovation projects and the Factory to each other.
   
• We created a Factory Improvement Forum with voting, open to all Factory users, to allow community-driven feature proposal and prioritization. This ensures that we are always working on the right features, that are the most wanted and valuable for the project teams.
   
• We partitioned the Factory components across the underlying Microsoft server products and across the different versions of those server technologies, so we can distribute our maintenance effort proportionally to how much value is currently generated with a specific technology. Obsolete (versions of) technology can be removed in newer Factory versions very easily, while older Factory versions that still use that technology can continue to live side-by-side with newer Factory versions.
   
• We made the Factory implementation very open and easily modified and extended. We implemented a large part of the Factory functionality in open PowerShell scripts to allow quick customization without the need for separate Factory source code or a Factory development environment. Our solution templates are customized with the same ease as when coding in regular Visual Studio Solutions. This way the innovation can be driven by projects on the go, modifying their factory instance as they learn.
   
• We provide on-call consultancy from the core Factory team to help innovators extend the Factory within their project context in such a way that the result is suitable for other projects, and for integration in new Factory releases.
   
• We provide one-click functionality to any Factory user to share guidance on any Factory action, or to post remarks or requests to clarify guidance. The community contributions are then used to create or improve the approved guidance. This way we prevent a central documentation bottleneck and we prevent work on guidance that is not needed.
   
• The Factory IDE is versioned with each product that is built with it; multiple Factory versions can live side-by-side on the same machine. This way, factory upgrades and customizations can be done independently for each solution, even when the solutions are developed in shared DTAP environments. The impact of Factory upgrades is much less, because you only upgrade for projects if and when the benefits are significant in the current lifecycle stage of the solution.

Closing remarks

I hope this post gives you an impression of where we have come from and where we are heading with our Factory approach. For me it is a privilege and a joy to be realizing this vision.

I would like to take this opportunity to thank everyone who helped to make this possible. In particular I would like to thank Maarten Sikkema for his unwavering belief and support to let us realize this vision, and his personal, hands-on involvement. Also many thanks to Serge van den Oever for the great teamwork and his many innovative ideas. Thanks to the Factory Champions team who encouraged and helped their colleagues to use the Factory and who provide us with invaluable feedback and direction. Thanks to the innovators who volunteered to integrate their learnings in the Factory, and last but not least thanks to all Macawers who believe in and use the Factory.

Moving forward, realizing tomorrow’s IT, today, the road ahead beckons brightly. In this article I outlined the vision that resulted in the Factory as it exists today. More recent developments in Microsoft IT have inspired us to expand our vision. Specifically VS Team System 2010, Microsoft Azure, GoGrid, ElasticHosts, and textual and graphical modeling with M and Quadrant offer great potential for future Factory versions, to raise the abstraction level, target cloud solutions and offer factories in the cloud, and to target more team roles and application lifecycle management (ALM) aspects. But more on that at a later time…

 

Vincent Hoogendoorn