Another great night at the Extreme Tuesday Club where I met Chris Matts, a Business Analyst (or Agile Business Coach) who works for ThoughtWorks.  It was an interesting discussion about the role of an analyst in a XP/Agile project.  Chris views his role as helping the business customer and the development team get closer together and helping the developers learn more about the customers' domain problem.  This kind of thinking about software development practices is a key reason why I like the XP/Agile movements.

 

I'd heard about the great work Chris had done on a large project that was going off the rails.  By tracking close to the business issues he was able to influence the project being cancelled and restarted.  We spoke about how to influence projects when it's obvious they aren't going to be successful, the complexity that can be involved when the contract work depends on the project continuing and the personal integrity involved in sticking with a decision you believe in.

 

I was interested to find out more about how a Business Analyst fits into such an Agile-oriented work place.  To me one of the big gaps in the Extreme Programming methodology is the generic concept of 'The Customer'.  It's a great thing that a lightweight methodology exists that is pushing the importance of customer involvement, but to me XP is thin on the details of what's involved.  In XP Refactored the authors point out that the notion of the onsite customer is a difficult idea to achieve.  Most work places wont allow their best people to leave the workplace in order to sit alongside the development team.  Even if they did they aren't always able to effectively communicate with the team.

 

Chris' concept was that he understood the technology and the business domain knowledge.  He believed that one of his benefits was to bring the customer and the development team closer together.  He was more effective than just a customer because he was skilled in learning about the business problems and analysing the requirements for a solution. He believed he had a role to play in an Agile team by viewing his role as the Business Coach focussed on transferring his business knowledge through to the development team.

 

One of the techniques that Chris mentioned as being effective was having the customer draw the business model on a blank sheet of paper at each meeting.  They found this produced much better customer feedback than they received from the customer reviewing their diagrams.  Drawing and talking about the business model produced a greater involvement than someone simply reading and reviewing a document.

 

Chris had extended this idea further with the concept of producing documents as a way of learning more about the business himself, but not necessarily publishing or sharing those documents with the team.  A very 'Agile' perspective that says the value of the analysis is the impact it can have on the rest of the team.

 

Quite an interesting discussion developed around the concept of 'How People Learn' or how a Business Analyst could help the team learn about the business.  I take the 'constructivist' perspective which says people learn through constructing their own view of the world.  The key to learning is to engage people in thinking in order to develop more sophisticated connections between concepts.  Any activities that encourage people to actively think about topics and relationships between topics are good.   Drawing a diagram and talking about it are much more likely to engage someone in thinking than passively reading or reviewing a document.

Optimizing is a complex problem.  Deciding what to do about our coming baby's nappies (US translation: diapers) highlighted the need to understand what is being optimised and what the trade-offs are.

Currently my wife and I have been discussing what to do about nappies for our coming baby.  I'm still dealing with this at a numerical rather than the emotional level as I'm still dealing with the sock that there may be over 5000 nappies in the first few years of a child's life.  While disposable nappies have the attraction of convenience it just feels bad to throw that many things out.  There's lots of evidence that disposable nappies represent around 2% of landfill and are generally evil as this clearly biased site  demonstrates.

I spoke about this problem with a good friend of mine who studied Computer Science.  He was a father himself and didn't like the idea of not using disposables because of landfill.  He came back to me last week with the following comment:

"While it is true that nappies do contribute to a small amount of landfill, there are great contributors to landfill such as kitchen waste.  As a Computer Scientist if I was trying to optimise something I'd start with the area that I could have most impact"

I liked my friend's thinking and told it to my wife, who said "yes, but choosing a cloth nappy washing service is a simple decision that is easier to implement than having to think about all of our waste".

I'm not sure which approach is "right" (we'll be using a cloth service for the first month) but to me it highlights that optimization is a complex problem that involves understanding what to optimize and what the trade-offs are.

While it is easy to see the benefits in the idea that services expose explicit boundaries, there is still some distance between this and understanding where service boundaries should be drawn and what it should look like.  A collection or recent posts from three Microsoft Architects highlight that services work best at the business level where they can share the same logical data model and single database engine.  It's nice to see the SOA discussion break through the academic debate to this practical guidance level.

Ramkumar Kothandaraman, a Solutions Architect on the .NET Enterprise Architecture Team, talks about  implementing the service-oriented tenet that services should have a clearly defined boundary when there are issues about searching or updating the shared data.  He proposes that it can be beneficial to share a database between related services and to expose these services as a collection or business services that share the same logical data model, rather than individual services. 

I believe this is similar to Pat Helland's session at the PDC where he spoke about business services encapsulating database-resident data and that each business services' data should live in a single database engine.  If the data can't fit into a single engine then look for ways to split it into two business services but don't have a service span database engines.  If the service needs to scale then consider partitioning the data between individual database servers rather than trying to merge and replicate.

Michael Platt, a UK-based Field Architect from Microsoft, shares the view that services need to be exposed at the business level.  He provides a useful, pragmatic definition of service oriented architecture as a model or abstraction of loosely coupled business-level functionalities or services.  I also enjoyed his reflections on a recent meeting discussing SOA:

There was a lot of discussion about what SOA was: explicit boundary’s, autonomous, etc which reminded me a lot of the COM definitions but with different words. Alas there was no discussion as to what a Service should look like, how big, what level of functionality etc; this was left as an exercise for the designer.

In terms of guidance on how to architect services, he mentions a story of the good old 'component oriented architecture' days where a client had 3000 components in a system, as a warning not to go overboard again.  Michael also invokes his 'First Law' as design guidance for a services which essentially says that that services should be coarse grained because of the huge communication and network stack they have to go through.

It can be difficult to talk to groups that are hostile about Microsoft technology because their beliefs are often based on sweeping generalizations or bigoted opinions.  Although Microsoft isn't perfect and doesn't always get things right it is surprisingly good at improving its products over releases and for engaging with the community to get feedback.

John Bristowe writes about his experiences attending a Linux User Group and talking to people about Microsoft and Windows:

What's funny is when you approach these people to challenge their stereotypes about Windows, they incite some form of moral/religion conviction against Microsoft and/or refuse to discuss the issue any further. I try to tell them, "Look, I'm all for a good debate. Let's discuss this." Nope. Nothing.

I had a similar experience to this on Tuesday night at the Extreme Tuesday Club.  I walked into a discussion about C# and .NET that went roughly like this

Attendee: "Microsoft standardised C# purely as a marketing move because Java isn't administered by a standards body.  Besides, the group that administers the standard is the easiest to manipulate standards body out there."

Me: "So you're saying that even though it's administered by an international standards body you think it may not be a '100% pure' standards body and for that reason you don't see any value in it?"

Attendee: "Well, Microsoft has never been interested in open standards!"

Me: "Well what about Microsoft's work in the WS-* space.  For WSE they have shifted from a pre-OASIS version of WS-Security to the OASIS version of WS-Security to ensure industry support.  They are also holding open workshops with any vendor who wants to attend to make sure that the standards are workable.  How does that work in the context of not being interested in open standards."

Attendee: "Hrmph.  I don't know anything about that."

As John mentions, there's a kind of snigger that goes through a group when they criticise .NET.

Fortunately I managed to have deeper conversations where I found out more about pain points that were creating a negative perception of .NET amongst a corporate group that was switching from Java to .NET.  I agree that some of these things are problems with the existing .NET technologies, but found it interesting that they were seen as such major issue.  What impresses me though is that some are being fixed or improved in Whidbey and that there are many Microsoft employees comment or discussing them in public.

• No refactoring in the current version of Visual Studio.  This is difficult for developers used to a Test Driven approach to coding and the awesome support provided in Java IDEs such as IntelliJ or Eclipse.  One guy said that the lack of this feature meant that ".NET code was unbelievably brittle: it was easy to write the code initially but very difficult to change it later".  Refactoring has been added to the Whidbey version of Visual Studio.  Jay Bazuzi is a developer on the Visual C# IDE team and has an excellent blog where he talks about his work implementing refactoring support for Visual Studio.
• The inability to set different access levels on property set and get statements.  This has been fixed in the Whidbey version of C#.
• Some interesting C# enum behaviour that Chris Stevenson mentions (here and here).  These were mentioned to me as evidence that ".NET is fundamentally flawed" even though the person I was speaking to didn't actually know the issues in detail.  I admit I didn't either, but I was impressed to see that Eric Gunnerson had replied to both of Chris' posts.

I'm not trying to say that .NET is perfect or that Microsoft always gets things right, just that discussions are more interesting when they are based on detailed information rather than sweeping generalisations.  I'm glad that the Java IDEs are currently better at supporting refactoring than Visual Studio.  I love the competition and the increasing number of good ideas that it creates.  I also like that Microsoft, eventually, take notice and improves its products.

Last night I had the pleasure of catching up with fellow RD Ingo Rammer for dinner at the Cinnamon Club here in London.  Ingo is a thoroughly nice guy and I had a great time chatting with him about the usual suspects (Remoting, Enterprise Services, WSE and Indigo) as well as presentation skills. 

In this prized addition to my photo blog roll Ingo is showing me the 'Ingo Rammer Signature Photo' pose (which I suspect may be a trademark, though he was kind enough to allow me to attempt to copy it).

It was also nice to catch up with IanG, Sebastien and Ingo's fiancée Katja.

I've been blog-lite while I prepared and then gave my Indigo presentation to the London .NET User Group.  The audience seemed to enjoy it and I had a good time presenting it.  Some quick points:

• The selling point of the Indigo Service model is that it is simple and easy to use.  The benefits for developers is that they get all the benefits of enterprise-quality features such as transactions, reliable messaging and security without having to write any code.  Letting Microsoft do the plumbing allows developers more time to write code that focuses on solving complex and interesting business problems.
• I took on board Clemens' experience and started talking about the 'why' of distributed technologies before delving into how Indigo works.  Luckily this group was very switched on.  I got some good questions about the performance of Indigo across AppDomains/Processes/Machines.
• Someone asked me to provide some background on why Enterprise Services are a better choice than Remoting for communicating between components on the local machine.  Clemens' has some more information on it here and here. 

Doing a presentation where you don't have the current version of the software to demo is a challenge (the PDC Indigo bits are from an older, M4 build, which has been substantially refactored in the now-being-coded M5 build, so there's not a lot of value in showing demos with the PDC bit).  In order to get some code samples I had to transcribe screen shots from the PDC videos (difficult since Steve Swartz didn't use word-wrap in Visual Studio).  I'm filled with (even more) respect for Don Box and Steve Swartz's presentation skills after realizing they managed to do four Indigo presentations at the PDC without even compiling, let alone running any of the applications.

Given the PowerPoint dependence and Clemens' reports on the complexity of the topic I decide I'd have to resort to audience bribery.  I drew on my experience in TheatreSports and brought several bags of Minties from the Australia Shop at Covent Garden (to my horror I discovered today that they are made in New Zealand!) and threw them to the audience whenever I detected the signs of PowerPoint-induced lethargy.

Pair programming with Jamie Cansdale at ThoughtWork's London Geek Night last Wednesday highlighted how much I like this practice.  Geek Night is basically an open night for developers who want to work on projects (ranging from open source development, to personal hobby projects or just for those who want to experience pair programming).  Jamie and I were working on creating a Visual Studio Add-In to remove unused 'using statements' in C# files (essentially, to replicate the 'Imports Tidy' functionality provided by C# Refactory that Scott Hanselman says is almost worth the price of the add-in alone).

Here were the things I enjoyed about Pair Programming:

• Knowledge sharing.  The knowledge transfer ranges from ways of using Visual Studio ("what did you press to get that?") to how to structure unit tests ("would you split that into a second method?") and features of the object model ("how can I find the assembly that defines this type?").  Even in a short session there's a great amount of knowledge that can be transferred.
• Continuous development speed. When sharing the 'driving' seat it means that it's possible to keep developing at a constant rate.  When I develop on my own sometimes I find it hard to keep going once I've achieved a certain milestone.  In Pair Programming it's possible to sit back for a while and let the other person drive.
• It makes you braver. Pair programming provides courage to start having a go even at complex problems.  It's easier to get started attacking a problem using two heads at the same time.

We started out by discussing how to implement the functionality.  Jamie was arguing for a brute-force but fool-proof mechanism of using a regular expression to search through the code, comment out any using statement that it finds and dynamically compile the code to see if it generates an error.  I was itching to write my own stripped-down C# parse that could scan through the source code and understand the using statements (including the alias keyword and nested statements).  To settle the stale-mate by switching our focus from the implementation to the functionality.  We decided to write the tests first (yes, XP is a simple set of practices but it's easy to get off track).

The beauty of writing the tests first was that the tests were independent of the implementation.  After writing a couple of tests it became clear that Jamie's approach was likely to be 'the simplest thing that worked' in the two hours that we had.  The best part was that we were able to achieve a working implementation inside this time!

I'm still eyeing off the SharpDevelop open-source C# editor, which as C# parsing code which I'm tempted to use (or maybe it's time to get into Mono), but for now I'm sold on the value of pair programming and test first development.

John Udell has written an article on WS Alphabet Soup based on his discussions with John Schewuck, an architect on the Indigo team.  Here are the interesting points:

• Indigo may ship before Longhorn to enable people to provide developers with access to the functionality provided by the newer WS-* specs.  I know the message from the PDC was Whidbey < Indigo <= Longhorn, but I guess haven't really considered it will come before Longhorn (I thought this was what WSE was for since Indigo is a 'baked in' part of the .NET Framework).  Not that I'd complain if it came earlier.
• There are a lot of Web Service (WS-*) specifications, but this is a good thing because they are composable.  You only need to implement the specifications you need.  The alternative would be a single specification that tries to do everything.
• The WS-* specs are inventing a lot of what has been provided before in DCOM and CORBA.  As Don Box says, 'The entire motivation for defining SOAP was to get the industry out of the "you need my runtime to talk to me" business that DCOM and Java RMI so happily got us into in the 1990's.'
• The advantage of using SOAP as a message description format  is that it provides extensible SOAP headers ("an accordion-like construct").  Previous network communications protocols was their dependence on fixed-offset dependencies that broke if you had to extend or modify the packets.
• Microsoft is wise to spend time worrying about the usability of its APIs.  This was my initial reaction at the PDC - 'Wow, they have done a lot to make this really easy for developers'.
• The key to reducing the cognitive overload caused by the specifications is an easy programming model (this acknowledges the reality that not everyone gets as excited as John Bristowe about a new specification).  Indigo's use of attributes in the Service Model is a major plus here.  How awesome is it to be able to write code like this to add WS-Security (signing, encryption), WS-ReliableMessaging and WS-AtomicTransaction support:

[Service]
[Security(Disabled=false, 
 SecurityProfileName="MyDynamicProfile",        
 ClientAuthentication=UsageMode.Required,
 IntegritySupport=UsageMode.Required,
 ConfidentialitySupport=UsageMode.Required)]
[ContextPropagation(typeof(ITransaction)]
[Connection(DeliveryAssurances.Full)]
public class MySecureReliableTransactedService
{
    [ServiceMethod]
    [Authorization(Role="Clerk")] 
    public void CreditAccount(string acc, decimal amount)
    {
        …
    }
}

The Loosely Coupled blog also has more commentary on the interview with John.

Matthew Adams left a comment on my last post disagree with my characterisation about services being more to do with messages as opposed to remote procedure calls (RPC).   It was a good question that I thought was worthy of a lengthier response.

What I was suggesting in the original post is that overtime there will continue to be a move towards services that use messages, and in particular SOAP messages based on the XML Infoset, rather than RPC style calls over time.  SOAP because it gives a broader reach for interoperability because of it's lowest common denominator approach and associated standards.  Messaging because it gives more flexibility in terms of what is sent and the message exchange patterns that are supported. 

The Great SOA Definition Debate
I've been trying to get away for getting into a debate about what services are, mainly because I haven't found a useful definition of service that says more than 'it's a great idea to have blocks of code that do something' and when designing these blocks of code, encapsulation and loose coupling are good practices.

By Matthew's definition a service is something that can do something, at a particular addressable location on a particular protocol, with a particular message format, in a such a way as the consumer doesn't have to care about how that something is done.

Using this definition of a 'service' it is true that the service doesn't have to be implemented as a 'message' end-point.  It is possible to implement these 'services' in ASMX web services, in-process .NET classes and remotable objects.  The problem is that these choices can impact the amount that the consumer has to know about the way the service is called, if not how it is implemented.

Don's Definition of Services
I like Don Box's definition of a service as a 'program that you communicate with using messages'.  These messages are SOAP messages based on the XML Infoset (a logical view of an XML document - angle brackets and text representations may or may not be involved).

The beauty of using SOAP messaging is that it is a lowest-common denominator of solving the problems involved with providing a service:

• How do we address the service, which protocol should we use to talk to it? (WS-Addressing)
• How do we describe the format of the messages that are sent and received? (SOAP)
• How do we describe the format of the contents of these messages (WSDL, XML Schema)
• How do we describe the pattern of the messages that are exchanged? (WSDL)
• How do we describe the other characteristics we need to successfully talk (WS-Policy)

What's wrong with RPC end points?
My main issue with RPC end points is interoperability and flexibily.  RPC-style calls to RPC-end points in Matthew's terminology calls can do all of these things, but they are more likely to do it in a less-interoperable way that requires greater knowledge of the underlying platform, protocol or programming model (for example you may need to agree on the binary representation of the parameters used in the procedure call which may mean sharing DLL's or worse, an entire development platform).

Of course any remote procedure call can be thought of as just a particular kind of message.  So it's possible to create services that are based around RPC-style calls made within a message (which is how ASMX web services work today).  Within web services there are two types of SOAP message format - RPC and document.  As Yasser Shohoud points out in his article RPC/Literal and Freedom of Choice, RPC formatted SOAP messages are just a subset of the document formatted (I'd argue message-based) SOAP messages .  As he says, ASMX web services show it's possible to use a document style message format but with an RPC programming model.

[Update: See Shawn Smith's post on this topic.  In the last section of this post I wasn't very clear about the distinction between SOAP message formats and message exchange patterns.  My point was that it is a better idea to implement more flexible approaches to both the message format and the message exchange patterns because the more flexible approach can accommodate the more rigid approach.  In this sense the SOAP Document format can accommodate the SOAP RPC message format and the asynchronous message exchange patterns can accommodate RPC message exchange patterns (synchronous request/reply).

Ultimately, it's not worth getting too fussed up about this as both the ASMX and Indigo programming model will support you whichever way you'd like to work.  You can write services based around asynchronous messages or using synchronous request/response (stop-and-wait) RPC-style calls.  As I said at the start of this post, over time I believe there will be a movement away from synchronous RPC calls towards asynchronous messaging.]

As preparation for my Indigo presentation at the London .NET User Group next Monday, I thought I'd write a couple of posts about Indigo, starting with 'The Road To Indigo'.

Indigo is a new technology that represents on the unification of existing distributed application technology.  It takes Remoting, Enterprise Services, Web Services and Microsoft Message Queue and combines them into a single area of platform-level technology.  In fact, Microsoft has literally taken all of the teams responsible for these technologies and put them in the same building.  Instead of thinking of Indigo as a completely new technology, think of it as the next version of all the existing approaches where all of the best features of each are combined into a single managed-code programming model.  Here's a diagram showing the progression of these technologies:

 

Indigo takes the best features of each approach and makes them available in one location.  If you're developing ASMX and Web Services today you can't  take advantage of reliable messaging (as in MSMQ) or transactions and object pooling (as in Enterprise Services) across web services calls.  With Indigo it is possible.  If you're an Enterprise Services developer today it's difficult to take advantage of flexible versioning provided by schema or being able to describe requirements using policy files (as you can in web services with WSE).  Indigo makes it possible to take advantage of all of these features.

Indigo also lets you chose the way you want to build applications without tying you in.  Indigo supports and encourages building service-oriented applications using messages based on SOAP.  However, Indigo still supports you and provides features even if you want to create distributed applications using a distributed component approach using remote procedure calls.  Choosing one approach over the other doesn't mean giving up features such as message-level encryption or transactions.  As Ingo Rammer says, Indigo recognises that you shouldn't be locked into 'a specific programming model, a serialization model, a transport model, or even just a serialization format'.

Combining the features of these four different technology areas into a single programming model is great news for developers.  We get the best features from each previous approach while only having to learn one programming model!  There's no more complexity about which technology or programming model to use for a particular distributed application scenario. It's the same programming model whether it's a peer-to-peer application on a hand-held device, right up to a secure, reliable and transacted message queue behind a massive online retailers.  

For those times when the platform needs to be extended, Indigo also provides an excellent extensibility model.  Even in this case there's only one extensibility model to learn.

Indigo brings us closer to Microsoft's long-term aim of helping developers spend less time on the plumbing and more time on the application code.