VincentH on .NET

Steve Cook just announced that he is moving to the VSTS Architect Edition team, to work with Cameron Skinner building UML tools for Rosario on top of the DSL Tools.

As you may recall, we can expect these five UML 2.1 diagram implementations in Rosario, built on top of the DSL toolkit:

• Class Diagram
• Use Case Diagram
• Component Diagram
• Sequence Diagram
• Activity Diagram

The fact that these are built on top of the DSL toolkit may provide interesting integration options with additional DSLs, e.g. if we can use the “Designer Bus” (a.k.a. “backplane”) for that (see the Software Factories in Rosario CTP post series by Edward Bakker for details: part 3, part 4 and part 5).

Stuart Kent gives some more hints in this post about interesting integration and extensibility options for these designers:

“Worth calling out are the features to allow a designer to be extended with new metadata once it has been deployed, and support for designers and models to interact. This will allow the designers delivered by Team Architect to be extended, and for new, third party DSLs to interact with them. Both these are key aspects of connecting UML with DSLs in the UML + DSL story that Cameron speaks about.“  

I’m happy with this design decision by MS. 

I like an approach that enables pragmatic solutions today while still moving toward the overall Factories vision. In realizing the factories vision at Macaw with the Macaw Solutions Factory, we use this type of pragmatic approach all the time, and it works great for us. I even think that such an approach is necessary if you ever want to actually arrive at the full vision (see my post on Factories 2.0).

It looks more and more like the Rosario release will really bring modeling to the masses, this will provide a lot more added value for the VSTS Architect Edition.

In the most significant post on factories to come out of Microsoft in a long time, Jack Greenfield unveils Software Factories 2.0. The software factories initiative has been repositioned within Microsoft to address a larger scope:

“Perhaps the most significant change is that the leadership of the Software Factory Initiative has moved from the Visual Studio Team Architect team to the Developer and Platform Evangelism (DPE) Platform Architecture Team. This move was designed to broaden the scope of the initiative”

Some of the things that now come into scope for the Microsoft Factories Initiative:

• “Leverage Microsoft products such as InfoPath and SharePoint“
  This is a good thing! At Macaw, we designed the Macaw Solutions Factory to target all Microsoft server products right from the start. This proved to deliver much more value (for us as a systems integrator) than more common factories that focus mainly on modeling and code generation for (parts of) the .NET framework.           

  Especially targeting SharePoint (WSS and MOSS) deployment and ALM aspects has provided great value for us, since this is a very challenging product in these area’s.

  Jack is not specific about whether “Leverage” means using these products to implement the factory itself and/or targeting solutions built on top of these products. I hope it will be both, of course.

• “integrating Software Factories with Software-plus-Services (S+S) Blueprints“
  Michael Lehman, who developed the project GlidePath infrastructure that underpins the existing blueprints, has recently moved to Jack’s new team. This will be an interesting marriage to watch!           

  Jezz Santos mentions in “Factories & BluePrints - TechEd 2008” that Michael has already grabbed all Factories slots at TechEd Orlando, he seems to be moving fast.

• SOA
  Think Oslo…

And much, much more… which brings me to the next point:

Will “Larger” mean “Later”?
When you work on realising the factories vision, as I have been doing with the Knowledge Development team at Macaw for some years now, one of the main challenges you have to deal with is scope.

Realizing the factories vision is most effective if you combine all aspects of the vision (think of the four pillars of software factories). If you have a company-wide initiative, your scope will eventually include all solutions technology as well as all people realizing solutions. Of these two, people require the bigger part of the effort [getting them to use and enhance the factory]. At least, this is what happened at Macaw.

Fortunately, at Macaw we were the right size of company to target our entire primary process (realizing IT solutions based on Microsoft technology) with one comprehensive factory initiative when we started. No “seperate kingdoms” of unaligned initiatives that we could not conquer had evolved - yet. We were lucky to be at the right place, at the right time, with the right idea.

Why am I telling all this? Because I recognize some of the same developments in this repositioning of the factories intiative at Microsoft. Microsoft moves to include more technology and - even more important- more people in their initiative. The move to DPE seems logical and even necessary. The “people” dimension of the scope at Microsoft is truly staggering, though. Which translates to very large efforts and very big inertia.

The real challenge in this Microsoft initiative will be to balance agility (regularly delivering tangible results that are most useful right now) with integration (aligning many Microsoft groups, products and initiatives to the right degree) and durability (earlier results not being invalidated by next steps). Because the factories vision takes such a long time to fully realize, a good balance on these three aspects is essential to both sustain the initiative and to actually arrive at the full vision.

If you misbalance, you run the risk to either fizzle out (lose momentum with people by taking too long - agility), disintegrate (collapse into a mess because directions of people and technologies are too disparate), or never reach your goal (one step forward, two steps back if next steps invalidate earlier efforts in the same initiative).

So is this factories thing a long shot?

In my opinion Microsoft is doing the right thing with this move, but you should appreciate that this initiative will take a long time to come close to realizing the full vision.

Does this mean that we should wait 10 years until all of this arrives? Definately not, because significant benefits can be realized with what can be done today!

What I learned while working on Macaw’s factory initiative is that most important for progress and value is not substantial factory foundation technologies, but the core factory concepts and the mindshift of people that are both the factory users and it’s innovators: the people building IT solutions every day.

In my experience you can implement factory concepts with relatively light investments, using simple, existing Microsoft technology, today (more details on that in a later post). The real effort and time - but also value - is in the change in people’s mindset.

Strategy 

Realizing the factories vision as a systems integrator can have a redefining effect. To realize a factories initiative and to get full benefits from it though, it needs to be a core part of the company strategy. This way, the technology, users, innovators as well as the business model all are guided to reap the very significant benefits. Building factories is not just about building a development tool…

We really are transforming the way IT solutions are being developed. When the industrial age of software development has arrived, will your company be part of it? Will your people have internalized the new paradigm? Will your business model allow you to maximize profit?

Will you be a factories 2.0 company? If you want to, my advice is - take your time: not in waiting for it, but to realize it. With the people and business model in place using current technology, you will be able to quickly leverage new factory foundation technology as Microsoft delivers on the vision, increasing your lead on competitors even further.

Edward Bakker announced the new software factory community portal which is located at sf.devrevolution.com. Edward asked me for feedback on the site early on, and was also kind enough to add my blog to the portal´s home page.

I am convinced the folks working on software factories can learn a lot from each other, and I’m all for sharing knowledge. So go check it out, be part of the community!

PS being Dutch is not a prerequisite ;-), this is intended as a global factory community portal.

Good work, Edward!

When I read Clemens’ post "Autonomous Develop Services for SOA Projects with Team Architect and Service Factory", I realized that part of his approach may integrate very well with the Macaw Solutions Factory.

Clemens describes how to enable each service in a SOA to be developed by a different team while still keeping the overall SOA design in one place. He partitions the SOA implementation across separate Visual Studio solutions (a single master solution containing the service models, and a separate solution for each service).

The Macaw Solutions Factory has had autonomous service development designed in from the start, it is a core concern addressed by the architecture style MAST (Microsoft .NET Architecture Style) that is at the heart of the factory. MAST also supports distributing a single large service across multiple solutions, and it specifies a standard pattern for dependencies within and across solutions. The factory supports independent versioning of each factory instance together with the service/application it is building while still using shared DTAP environments for the SOA. This makes service maintenance even more flexible and autonomous. So we got that part covered, it proved to work fine for a couple of years now.

Dependencies across projects and solutions in Clemens’ approach

Dependencies across solutions within a MAST service / application (dependencies across projects are standardized on the three-layered services application pattern).

However, Clemens also devised a way to keep the overall design of the SOA in a separate solution, by means of clever application of the P&P service factory and the distributed system designers in VS Team Architect. Overall SOA design is an area not yet addressed by the Macaw Solutions Factory (the focus is on building services and applications that are first-class SOA citizens, not on maintaining overall SOA design), and Clemens’ approach for that may integrate very well into our factory.

We already plan to integrate the Orcas version of the P&P Service Factory into the Orcas version of the Macaw Solutions Factory (we actually prototyped the approach during the Service Factory Customization workshop, Don and Olaf were a great help there). Thanks to the excellent extensibility of the service factory this is a real breeze (great work, guys!).

The extensibility built into the service factory looks to be a real enabler for the composability of a certain class of factories. It’s great to be able to combine approaches pioneered by different people in the factory community. Sometimes life is real easy

More goodness for factory builders: you can now start to target non-technical factory users with Visual Studio 2008 Shell, it was released on 2007/11/21. For details and download links see the post on the VSX Team Blog.

Use DSL’s and other Visual Studio extensions without scaring people with all the “developer UI” in Visual Studio, while retaining full integration options within Visual Studio.

See also my original post on using the shell for factories.

Some time ago, Jezz Santos blogged about how the software factory community, as he knew it, seemed to be contentrated in the Netherlands. That he mentioned me made me feel good of course, but I also realized that being close to so many leading people in the software factory field (outside MS that is), is a great opportunity to participate in a vibrant community.

Anyway, here I was six months later, at the Service Factory Customization workshop which is held in the Netherlands, which a good portion of the Dutch software factory community used as a pretext to exchange experiences and look at each other’s factories.

I had a great time there, some of the attendees were even allowed to present some of their own factory work (Don and Olaf were very accomodating to doing whatever gave us most value, which was really nice of them). I also presented a short flyby of the Macaw Solutions Factory (look for more posts on our factory here soon… no really).
For some more details on the workshop and what we demoed to each other, see Serge’s post on it.

After the attendees presented their work, I asked Don about how the Dutch compare to what he had seen in the factories area globally, and Don confirmed what Santos said: the Dutch are at the leading edge. Sweet

So, I consider myself lucky, being at the right place at the right time. I’m really looking forward to what all these smart and passionate people are going to do to realize the factories vision, which is also my mission / job / hobby .

We live in interesting times…

In fall 2007 Jack Greenfield published an article in which he includes a diagram of the factory schema that looks quite different from the one in his well-known software factories book.

This is the “new” diagram from the article:

and this is the diagram as published originally in the book:

So this intrigues me… does the new diagram reflect new thinking on the factory schema? It seems to be a lot more oriented towards an implementation of a factory runtime and authoring environment.

I wonder what this could mean for the new factory runtime and authoring environment in Rosario?

Until now you could only use DSL Designers in Visual Studio (Professional and higher editions). This limited using DSL’s to those development team roles that are comfortable using Visual Studio (developers and architects).

However, this has now changed with the announcement of the Visual Studio 2008 Shell. Now users can use the DSL Designers that run in Visual Studio in a stand-alone shell. So now we can also use DSL designers for supporting non-technical team roles (such as business analysts) in a software factory.

This is what a stand-alone DSL can look like:

Jezz Santos, MS Software Factories guru blogs about how the dutch seem to dominate the software factories community, mentioning me as well.

I already received happy emails from Edward Bakker and Dennis Mulder on this (15 minutes?) of fame

Good for the factory community!

Gunther Lenz and Christoph Wienands write a nice book on how to actually design and build a software factory on the .NET platform.

The book, Practical Software Factories in .NET, is based on a real, large project at Siemens, and it is a welcome companion to the original factories book by Jack Greenfield et al, which is very thorough but also quite abstract and elaborate.

Recommended reading for factory architects.

Jezz Santos, a factory builder who is leading in designing and applying software factories, predicts that Microsoft will deliver factories with their products as well. I completely recognize his reasons why that would be a Good Thing.

Jezz uses a Sharepoint Factory as an example. Which is interesting, considering that at Macaw we are building the Macaw Solutions Factory, which targets (amongst other Microsoft server products) SharePoint.

At the last TechEd in Barcelona, Jezz and I talked for about an hour on our plans for the Macaw Solutions Factory, and how Jezz would go about designing that. Apparently Jezz recognized a need…

Steve Cook mentioned my article on how we are realizing the software factories vision for Macaw. So this is why my article got so many hits lately. And as a boon, it gives us our 15 minutes of fame by displaying us on MSDN. I like it

[UPDATE this article was originally published here, I included it on this blog to get my story on factories in one place.]

Vincent Hoogendoorn, Architect, Macaw

April 2006

Summary

This article details a case on how the Software Factories vision is being pursued and realized by a Microsoft Systems Integrator specialized in .NET technology. A careful approach is detailed to gradually realize the vision, avoiding large investments and high risks while still reaping immediate and significant benefits. A detailed overview of the key first elements of the software factory is provided: a generic .NET architecture style, project templates, and a comprehensive collection of both standard and custom development tools. The experienced benefits are detailed and the potential is identified both for other systems integrators to apply this approach and for further development towards realizing the full software factory vision.

Introduction

As a Systems Integrator specialized in advanced Microsoft web technology, our larger projects reached a scale that really required standardizing and automating key aspects of our development process. Without this, scaling the development teams beyond a certain size made the development environments and -process unpredictable and inefficient. This particularly proved true for projects based on (combinations of) complex server products such as SharePoint, Commerce Server and BizTalk. To remedy this, we implemented processes, guidance, a standard architecture style, project templates and custom development tools. This approach proved very successful to stabilize and accelerate large projects. We soon realized that our approach could be generalized and made scalable, so it would benefit both small and large projects involving arbitrary combinations of Microsoft products. While studying key publications on the Software Factories vision [Greenfield04], we recognized that we in fact – in a modest way - were realizing part of that vision. We decided to embrace the wider vision and to devise a careful approach that would allow at least 80% of our development effort as a Microsoft Systems Integrator to benefit from the software factory approach.

Approach

We wanted a careful approach to realizing Software Factories, based on proven success, without taking large risks or making heavy investments. This led us to stage our approach in two phases.

The first stage is based on the proven success of our approach for large projects. The objectives we set for this stage are:

• Accelerate and scale development through standardization and automation.
• Create an optimal environment for harvesting and deploying reusable assets.
• Prove success in both small and large projects.
• Facilitate 80% of our development effort by the end of 2006.

This first stage focuses on reusing software structure, not functionality. We first want to create an environment of many products with a standardized software structure, which by itself has proved to provide a lot of value. However, such an environment will also be an optimal breeding ground for effectively reusable functionality.

The second phase of our approach takes off when sufficient functionality reuse has been recognized, proving the business case for reuse. At that point, we define one or more software product lines that – based on the common software structure –provide packaged and configurable functionality.

A key element of our approach is that we only package functionality for reuse after the potential for reuse has been demonstrated for two or three occasions within our domain. We do not want a big upfront design approach; reuse has to emerge.

Architecture Style

To realize our approach, we specified a standard architecture style for the software structure of our products. The family of systems that we target with this architecture style consists of all applications and services, built in .NET with C#, based on (combinations of, multiple versions of) Microsoft server products.

While analyzing the .NET products that we - as a Microsoft Systems Integrator - assembled since the platform became available, we recognized a single pattern for structuring software that not only proved to add significant value to our largest projects but that would also fit the vast majority of this family of systems well.

This pattern is known as the Three-Layered Services Application. Microsoft documents this pattern [MSDN05] and also provides extensive guidance on designing applications and services for .NET based on this pattern [PnP02]. Applications and Services designed according to this pattern do not only benefit from it from a stand-alone perspective, but they also integrate very well in a SOA, as detailed in the Connected Systems vision by Microsoft.

Software Structure in a Three-Layered Services Application

Since the Three-Layered Services Application pattern had proven to be valuable and durable and also appeared to apply throughout our domain as a Microsoft Systems Integrator, we decided to base our architecture style on it. To reflect that we target all of our .NET development with this style, we named it MAST (Microsoft .NET Architecture Style).

A key guiding principle for defining MAST is that functionality assumptions should be avoided. We focus on the Microsoft foundation products that we use and on the generic software structure in .NET. This is to promote a large enough scale for our (emergent) software product line(s) to be effective. Since we are a relatively small company, splitting up our economy of scope could very well reduce the advantages of the software factory approach to the point where it would become unprofitable.

The MAST standard specification was developed according to the Maturity Level for Work in Progress of the W3C Process Document [Jacobs05]. It serves both as architectural guidance and as a specification to build templates and tools for accelerating development.

We intend to publish the full MAST standard specification in the near future. However, to make this architecture style more concrete, we provide a fairly detailed overview of the specification here.

The main elements specified by the MAST standard are:

• Standard component types.
  A component taxonomy based on the Three-Layered Services Application pattern, extended with specialized component types for specific Microsoft foundation products (an example of a component type is a Business Entity, see the figure above).
• A standard namespace hierarchy.
  This hierarchy achieves a very good balance between consistency and flexibility in software structure. It promotes stability, reuse, low-cost refactoring and flexible deployment.
   
  Namespace Structure in MAST

  MAST specifies standard names for each namespace level that indicates a technical role, but not for the other (functional) namespace levels.

• A standard structure for solutions and projects in Visual Studio.
  MAST specifies that each component type maps to one Visual Studio project. The namespace level directly above the component type (either a functional area or a product dependency) maps to one Visual Studio solution. MAST specifies naming of solutions and solution items, projects, project folders and project items. Finally, MAST specifies Visual Studio project property values.
  Both VS 2003 (.NET 1) and VS 2005 (.NET 2) are supported.
• A standard structure for dependencies.
  Because each Visual Studio project contains components of a known type, MAST can specify all potentially valid project references within a Visual Studio solution.

  Combining relaxed layering (meaning a component may directly access all lower layers) with selected design pattern implementations has resulted in a matrix that specifies all potentially valid project references within a Visual Studio Solution.

  Dependencies between Visual Studio Solutions are restricted; only dependencies on the functional area named Framework are allowed. Since foundation products may have dependencies on each other (e.g. SharePoint Portal Server 2003 has a dependency on Windows SharePoint Services 2), dependencies on the framework of other product dependencies are also allowed. This results in a funnel-like structure for dependencies across visual studio solutions:
    
  Dependencies across Visual Studio Solutions in MAST
  
  A key benefit of this dependencies structure is that the impact of changes outside the framework is limited to a single visual studio solution. Since all solutions have the same structure (based on the three-layered services application pattern), refactoring functionality from solutions into the framework is very cheap; it can be done when the need for actual reuse in a second solution arises. This results in thin frameworks that only contain functionality that is actually used in two or more solutions.

• A standard folder structure for source.
  A key aspect of this folder structure is that a product dependency (e.g. DotNet2 or Wss2) is completely self-contained.
    
  Part of a product folder structure in MAST. Visible in the left pane are the folders for the DotNet2 product dependency with its configuration, shared assemblies and functional area’s (Framework and Templates). In the right pane the template solution with its DotNet2 specific project templates is visible.
  

Templates and Tools

Based on the specified standard architecture style, we created templates for visual studio solutions and projects for specific product dependencies. The templates provide software structure and only contain minimal functionality – just enough to allow actual deployment, running and testing. Separate versions of the templates were created for Visual Studio 2003 (.NET1) and Visual Studio 2005 (.NET2).

A template solution for the DotNet2 Product Dependency in MAST

As illustrated in the example above, the software structure defined by MAST is easily recognized in the templates. The product dependency (DotNet2) and the functional area (Templates) appear in the solution name. The layer and component type (e.g. Presentation.WepAppUI) appear in the project names, and the component subtypes (e.g. Pages) appear as folders within projects.

Post-build events for incremental deployment
The templates have configurable post-build events that perform quick incremental deployment to the local machine. By separating the deployment location from the editing location (e.g. for aspx pages) it is possible to do an automatic full deployment to a development machine at any time, without touching any development work (source) in progress. Developers then only need to rebuild the solution they are currently working on to integrate their changes locally. Even though we already use a continuous integration build server, we found that adding this approach significantly improves the predictability and efficiency in the development environment.

Tools
Because the .NET 2 based version of key Microsoft foundation products is still a while away (e.g. for SharePoint Portal) and we need to support 80% of our current development effort, we still need to fully support .NET 1 development with our tools (as well as .NET2). While technology currently available in Visual Studio 2005, Team System and the Guidance Automation Toolkit is very promising, we decided to start out with a toolset that supports both .NET versions. Once the volume of our .NET 2 development effort is big enough to justify the investment, we plan to create a GAT based version of the tooling. From then on, we will gradually shift our focus for further tools development towards the GAT based version.

We use the following standard tools in our automated build servers:

• CruiseControl for continuous integration build server.
• Subversion for source control.
• NAnt for automating build and deployment scripts.
• NUnit, MbUnit and IeUnit for automated testing.
• FxCop and NDepend for code analysis.
• NDoc for generating documentation.

To support and automate development using the templates and the specified architecture style, we also use the following custom tools:

• MAST Software Structure Generator.
  Based on the specified target namespace of a new component, this tool selects the appropriate template and uses that to generate the new component class in the specified namespace. If a new solution and / or project is needed to contain the new class, they are also generated on the fly. All project properties and references are then set according to the standard. We plan to add a VS 2005 front-end to the tool using the GAT.
• MAST Software Structure Validator.
  A tool - similar in operation to FxCop - that validates the software structure of a product by analyzing its source. All rules specified by the MAST standard are validated. As with FxCop, an online knowledgebase provides details and rationale for each rule; it also explains when each rule applies. Developers do not need to invest a lot of time in learning and remembering all details of the MAST specification – they learn from the validator’s feedback while developing the product.
• MAST Build & Deploy Script Generator.
  This tool generates post-build events for quick incremental deployment to the local development station (we separate the edit location from the deployment location for all artifacts). This tool also generates scripts for the build server. These scripts create builds, package them, and can be run on any machine in any environment to automatically deploy the full build. The tool uses machine-specific deployment configurations to generate and run the scripts. The configuration allows automatic deployment to different locations and deployment of different subsets of the product functionality, based on the machine environment, role and configuration.

Both structure generation and structure validation allow templates to be maintained and specialized with the product. This type of template maintenance is very quick, because the templates are just simple projects that live in the designated functional area named “Templates”, which – in a debug build - is automatically built, deployed and tested, just like any other code is.

Benefits

The key benefits we experience from this approach are:

• Because of the high consistency across projects, resource management can be done in a much more flexible manner. Dynamically varying team composition, size and locations (e.g. for virtual teams and outsourcing) becomes easier.
• Deployment (both incremental and full) is almost completely automatic; no hand-written deployment scripts are needed.
• Designing and maintaining the software structure in a consistent manner soon becomes second nature for all team members. Much less time is spent on arbitrary decisions and on translating different sets of concepts / styles between (replacement) team members.
• Communication within development teams is very efficient and unambiguous. The concepts defined in the architecture style become common language. This proves particularly valuable for communication between architects and developers.
• Because the architecture style avoids functionality assumptions, the software structure remains very stable when requirements change (over time).
• Because the same three-layered service application pattern is repeated in all functional areas (including frameworks), refactoring becomes cheaper. Refactoring now becomes a more lightweight, on-demand activity.
• Last but not least: because of the consistent structure across projects, recognizing, harvesting and generalizing functionality for reuse can be done very effectively.

The Road Ahead

As a Microsoft Systems Integrator we do not think we are unique. Our approach assumes the .NET platform and a team-based, project-based approach; however it avoids assumptions about functionality or methodology (which in our case is based on the Rational Unified Process). Our approach could therefore be valuable to other Systems Integrators as well, especially for companies that concentrate on Microsoft technology and use a team-based, project-based approach. It seems likely that good collaboration opportunities exist.

As a growing part of our development effort uses this approach, its benefits are expected to scale better than linear. Once the actual reuse of functionality reaches critical mass, we will define and implement one or more software product lines to package that generalized functionality, realizing even more benefits.

We will continuously improve and extend our factory assets (e.g. tools, templates, frameworks), moving towards the latest software factory development tooling as the market share of its supported platform grows.

The potential value of realizing the software factories vision remains compelling, although the road to get there will take some time.

References

[Greenfield04]
Software Factories – Assembling Applications with Patterns, Models, Frameworks, and Tools
Jack Greenfield, Keith Short, Steve Cook, Stuart Kent, John Crupi
Published by Wiley, ISBN: 0-471-20284-3

[Jacobs05]
W3C Process Document
Ian Jacobs, W3C
http://www.w3.org/2005/10/Process-20051014/

[MSDN05]
“Three-Layered Services Application” application architecture pattern
MSDN Library
http://msdn.microsoft.com/library/en-us/dnpatterns/html/ArcThreeLayeredSvcsApp.asp

[PnP02]
Application Architecture for .NET: Designing Applications and Services
Microsoft Patterns & Practices Group
http://msdn.microsoft.com/architecture/solvenow/solveimpdes/default.aspx?pull=/library/en-us/dnbda/html/distapp.asp

About the Author

Vincent Hoogendoorn is an architect at Macaw, a Systems Integrator specialized in advanced Microsoft web technology, based in the Netherlands.

His professional areas of interest are Solution Architecture, Service Oriented Architecture and Software Factories for the .NET platform. He recently was involved in projects for Smurfit- Kappa, Bouwfonds, Sara Lee and Akzo Nobel. Starting 2006, he is also member of Macaw’s Knowledge Development group, where he works on realizing the Software Factories Vision for Macaw.