Last week I mentioned that the Microsoft Messaging message wasn't being fully ACK'd by the community and posed some questions.  In the interest of clarifying the message and checking my own understanding, here are some answers I've come up with based on watching the PDC sessions on my daily train commute and the comments that BWill from the Indigo team left on my last post.

What's the relationship between Indigo, MSMQ, BizTalk and  SQL Server "Yukon" Service Broker?  When and where is each technology appropriate?
Basically it depends what type of application you are trying to build and what environment it needs to run in.  Here's a chart adapted from the DAT406 session:

Technology	Indigo	MSMQ	"Yukon" Service Broker
Environment	Any WS-* compliant endpoint	Windows NT	Yukon on both ends
Application	Any distributed application	Any Async NT Application	Database applications
Message Store	In memory or DB persistent store (Yukon/SQL Server 2000)	MSMQ Message Store (NT File System)	Built into Yukon
Type of Message	Persistent and non-persistent dialogs	Reliable, Express and Transactional Messaging	Transactional messaging only
Protocol	Various	Various	TCP only

Where does BizTalk come into it?
BizTalk is a product that that builds upon other technologies in the Microsoft platform.  As BWill says in comments on a previous post, choosing BizTalk or Indigo will be a question of how much of the infrastructure you want to build yourself.  BizTalk has connectors for MSMQ currently, in future it may connect to Indigo or possibly to Yukon Service Broker.  BizTalk 2004 is currently in Beta it's on a different release schedule than the other products.

Where does Yukon Service Broker fit?
Here's Roger Wolter's answer from the DAT406: Building Reliable Asychronous Database Applications with Yukon:

The key to keep in mind is that Service Broker is a database application framework, not a messaging system.  Yeah we send messages to other databases, but if you're building a messaging infrastructure there's not enough in Service Broker to satisfy your needs.  If you are building a queued, asynchronous database application and you wanted to use reliable messaging to scale out that application, Service Broker is the answer.

There were some hostile questions in the DAT406 session about why it was necessary to put a messaging layer in the database.  John Cavnar-Jonson (who definitely needs a blog) calls it an 'abomination' in the Developmentor Indigo discussion list.  Personally, I think it's part of a Dr. Evil style plan from the SQL team - if they were to add a spreadsheet to the product then a great majority of the world's applications wouldn't need an OS letting the SQL team achieve world domination.  Seriously though, there seem to be several good reasons:

• Developers that know SQL can now  develop queued, asynchronous database apps.  The ability to have asynchronous queues is a very nice architectural feature.  Being able to achieve this with SQL syntax like BEGIN DIALOG ... FROM SERVICE ... TO SERVICE is pretty cool.

• It's all in the one box.  Everything happens within the database.  Backup, restore, installation, configuration, monitoring and security are all there in the one location.  So deployment of the database is deployment of the messaging system etc. (no need to hassle with MSMQ installs).

• The  message broker is the database.  It's easy to query the status of messages, processing the queues is as simple as writing a stored procedure, the database can efficiently throttle the queue processing resources and it's possible to farm out message processing work to another machine since all that is required to process a queue is a DB connection string.

• It's fast. The Service Broker is fast because there's no need for two-phase commits for transactional messaging, there's no need to cross processes to get to the messaging platform and if the send and receive queues are in the same database then it's very fast.

Neils Berglund from Developmentor has been teaching Yukon for a while to Microsoft employees (such as Tim Sneath) and has an excellent sample chapter on Yukon Service Broker that's available for free download.

How does MSMQ fit into the longer term picture?
BWill says in my comments that the Indigo team has shared the love and embraced the MSMQ team into its building.  John Cavnar-Jonson did some research at the PDC:

I completely disagree with the idea that Indigo offers all of the functionality of MSMQ. I discussed this exact question at the PDC with Steve Swartz, Mike Vernal, and Anand (whose last name I don't recall, but he's an MSMQ program manager).  Indigo will offer reliable messaging (which is a huge improvement over current web service technology), but it will not be a full-fledged message queuing system.

As I reported from the PDC, the message was:

"We are not building the uber queuing system - we are not a replacement yet for MSMQ - we have support for routing, but we aren't replacing CISCO, we support eventing but we aren't a replacement for Tibco."

I think there's more to be said in this space.  It's likely that this is about achieving an Indigo V1 release (primarily about unifying the three different programming models and baking WS-* specification support into the platform) and then targeting more ambitious goals with future releases.

Which parts of Indigo will ship in Whidbey and which bit will ship before or with Longhorn?
Basically, System.Transaction will be in Whidbey, the rest later.  I'm still digesting Don's WSV302 Indigo Part2: Secure, Reliable, Transacted Services and Jim Johnson's Transactional Programming on the Windows Platform presentations to understand this more deeply.

Given that the last version of WSE will be wire-level compatible with Indigo and that a future version of WSE is likely to support WS-ReliableMessaging, what are the benefits of Indigo other than the simplified programming model?
Even though I love WSE I'm following the words of Hervey and accepting that WSE is V.Last++.  This question was me fishing for what features Indigo will provide me with as an architect/developer that I can't get from WSE.  BWill mentions:

Advantages of Indigo over WSE: three off the top of my head are performance, integration, and support [it's part of the platform rather than WSE's 2+1 support policy]. I'm sure there are others. Also, note that there is no guarantee that every feature in Indigo will also be WSE.

So, no bites as to what the extra functionality of Indigo might be, so I'm still fishing (e.g. digging deeper into the Longhorn SDK Indigo Samples).  Of course, Indigo has learnt from WSE, so the Indigo programming model will also be nicer (though the WSE programming model is already small and well refactored).

Indigo is committed to supporting WS-* standards and interoperability, but what extra functionality will be available if the whole environment is made up of Indigo boxes?
I'm still trying to get a feel for what features and functionality might be available in Indigo.  BWill mentions the fact that Indigo will likely run faster in an all-Indigo environment. 

Indigo does offer Peer to Peer functionality
Robert Scoble tells us:

I stopped in on Don Box yesterday and he gave me a demo. Indigo is going to radically change how we think of Internet technologies. Imagine something that looks like a website, but that doesn't require a centralized server. Now you're getting your mind around what Indigo could do. Indigo is designed to take advantage of our always on, always connected computers.

It's not difficult to spot an Evangelist with a marketing strength is it :)? Sounds like a simple programming model on top of existing network stacks opens up opportunities to use the Internet for more than just the web browsers against a central web server.  I think I'll review WSV306 Indigo and Peer to Peer apps on the train this week.

 Sam Gentile has been doing some useful questioning and thinking about Service Oriented Architecture (SOA) that lead me to think about the scope of SOA's.  Certainly SOAs are a useful architectural concept, but they are not an answer for all projects.  Currently SOAs are complex and difficult to implement and until the technology improves and it gets easier for developers to build, deploy and manage these systems, SOAs are likely to be most appropriate for a small number of enterprise projects. [Update: I've edited this post slightly to clarify some points after comments from John Cavnar-Jonson]

On the 'death of objects' and the SOA 'paradigm shift'
Same started off by saying:

SOA [is] one of those paradigm shifts - it really does mean the death of objects at least as we know them.

Before I could suggest that we fine any blogger who uses worn-out expressions like 'it's the death of [technology X]' and 'paradigm shift', Sam later clarifies that he didn't mean objects but OOA/OOD: 

I really meant OOA/OOD - how do you now design/decompose system design/requirements into architectures and I think it's less now about classic OOA/OOD and tightly coupled object design and more about a loosely coupled collection of components under a service.

I agree that proclaiming the death of objects was a misstatement. It may be more accurate to say that components, which are based around sharing types between parts of the system, are likely to become less of a primary focus with the rise of SOAs.  I agree with what Bryan Noye's says:

I don't think SOA means the death of OOP or Components at all. Just like most people build components using OOP, I think most people will built SOAs using OOP and Components. They are not competing concepts but complementary.

SOAs are important where there is a need to share messages and interoperate with unknown others
For my mind, SOAs provide the most benefits when there is a need to share data/information and  interoperate with other groups, possibly on unknown platforms, that you have no control over. There will still be plenty of applications that are built in the current n-tier component Enterprise Architectures. Talking with Jim Johnson at the PDC he made the point that the benefits of SOA with external partners may also be benefits within an organisation or service boundary. I agree, but I think the technology platform (e.g. Indigo) and management tools (e.g. whatever Microsoft are planning here) have a long way to go before these benefits outweigh those of using existing component-based technologies.

Interoperating with others often means going for a lowest common-denominator approach which is always going to perform slower than when you can go with a binary format and control both ends of the wire (as Sam mentioned, using ASMX just doesn't give the same performance as current 'binary typed' systems, which is why Indigo will do special things if it knows it is working in an all-Indigo environment).

SOAs are currently still complex and difficult to build and manage
SOAs are currently still complex and difficult to build and manage. They are complex because the standards are still being implemented (on a recent project I was on it took a major international bank nearly 3 months to convince a market-leading J2EE vendor to adopt SOAP headers and honour the 'mustUnderstand' attribute).  Newer standards like WS-Addressing are still being worked through and implemented.  It's difficult because of the layers of the technology that must be understood in order to build the systems.  Just read Clemens' description of his latest FABRIQ project to see the level of technology understanding, skill set and experience you need to build an SOA project today. As the tools develop and experience and awareness of SOA's grow they are likely to get easier and simpler to build, in the same way that client server and then n-tier were once considered complex and difficult but are now considered main stream.

Services are about outside, Objects/COM+/Enterprise Services/MSMQ are for inside
It's useful to be aware of boundaries as Don Box demonstrated at the PDC.  There are parts of SOA that are designed to be used on public organisational boundaries and some that are better deployed within an organisational boundary or even behind a service boundary.  Clemens Vasters' makes the distinction between 'near and far'.

When using services outside an organisational boundary there are benefits to using open standards and working with contracts and schema rather than type, since it's difficult to control what is at the other endpoint.  Enterprise Services and MSMQ provide useful functionality that isn't yet covered in WS-* standards, but the problems with these two approaches is that they often share binary type information or require a Microsoft box or adapter at the other end.  This doesn't mean they shouldn't be used in SOA, just that they are better used inside the organisation where it's possible to have more control over the communication and the endpoints.  Within the service boundary there's still a service to provide, and it's here that technologies like components/COM+/ES/MSMQ are likely to be just as useful as they are today.

Gregor Hohpe's Enterprise Integration Patterns provides useful SOA guidance
I think that Sam Gentile is right, there are some architectural changes that need to move towards SOA and message-oriented systems.  Luckily Gregor Hohpe has written Enterprise Integration Patterns - Designing, Building, and Deploying Messaging Solutions a great book distilling his experience of these systems.  Hervey Wilson's reading it and its on Ingo Rammer's list of recommended books as well!.

Steve Maine writes about the importance of getting the API right when designing software:

Naming the abstractions present in a software system is an important design consideration, especially when designing a consumable API. The names you choose will influence the way consumers of your API will think about your software – you can either choose to make it easy for them, or hard. Convincing ourselves that we've made the right choice is proving difficult

Rather than trying to convince yourselves about the design I'd suggest trying it out or doing some usability testing.

In terms of trying it out, I'd suggest taking the Test Driven Development idea (or the Microsoft idea of Dogfooding) and try writing the sample code first to see how the code looks when you use it yourself. Often design discussions are 'tainted' by people knowing the implementation rather than thinking from about the intention in a users mind.

For the usability testing you could start with the idea of Personas to describe the users of your software (e.g. is it for a 'mort' and 'Einstein' or an 'Elvis'). Think about what goals the users of your they are trying to achieve and then write those down. Get around 5 people who might write your code and ask them how they might achieve these tasks using the code. Give them the object model and ask them to write out the pseudo-code. If it's just the naming that you're concerned with try describing the job is does and asking potential users for what they think it might be called.

You can go all psycho-serious and see what value you can extract from this Microsoft Research presentation "Describing and evaluating API usability at Microsoft" but I'd go with Jacob Nielsen and start with the discount usability approach. Get a small number of potential users, think of open questions you can ask or tasks that they could do then follow the golden rule of sitting back and listening/watching without saying anything (the photo on the right me with Jacob Nielsen.  I'm smiling even though I paid £400 of my own money just to hear Jacob tell me this simple rule).

Recently I've been reviewing the most efficient way of retrieving values from an XML document in .NET.  As Scott Hansleman mentions, the best way to find this out is to write some code and measure the performance, so I wrote some code that can be downloaded to test the performance of three different approaches:

• an XmlDocument using XPath queries with SelectSingleNode
• an XPathDocument with an XPathNavigator
• using the XmlSerializer to deserialize into a custom class

The results show that the XmlSerializer is fastest once the initial cost of creating temporary assemblies has been overcome.  In situations where the initial performance is most important then an XPathNavigator over an XPathDocument is the fastest.

Approach
Here's more detail about each approach:

Aaron Skonnard provides some excellent background on the benefits of the XPathDocument over the XmldDocument in his article .NET XML Best Practices: Part I: Choosing an XML API.

For the timing tests I used the code available in the EggHead Cafe article "High-Precision Code Timing in .NET".  Unfortunately the Ticks property of the System.DateTime class is only accurate to around 16ms even though it displays values to 100 nanoseconds.

Results
In this app I take a small XML document with no namespace and retrieve four values from it.  Here are the figures I got when running the console application for the first run and then a repeat run:

Runs	XmlDocument	XPathDocument/Navigator	XML Serialization
1	0.00543	0.00129	0.09020
1	0.00051	0.00035	0.00028

The first time the application is run the XML Serializer has to create temporary dynamic assemblies so it performs the worst, however on subsequent runs within the application it performs the fastest since it can use a cached copy of the temporary assemblies (As Scott found out, these assemblies are not cached in .NET 1.0 if you specify a namespace).  Daniel Cazzulino has some good background on how the XmlSerializer works).  In both the first and the second runs the XPathDocument/XPathNavigator approach is faster than the XmlDocument.

In the sample code I've also presented an ASP.NET web application that can host the same tests.  ASP.NET creates a new AppDomain for each website it hosts and is maintained across requests to the server until it is shut down or recycled. Since the XmlSerializer caches its temporary assemblies based on the AppDomain, using the XmlSerializer in an ASP.NET web application or web service application is actually the fastest technique for retrieving values from an XML document.

In situations where the AppDomain is created per-request then using the XPathDocument/XPathNavigator will be more efficient than an XmlDocument and the XmlSerializier.

Discussion
The XmlSerializer is the fastest way to retrieve values from a small XML file if it is possible to overcome the cost of the creation of the temporary assemblies.  In situations where the first retrieval is the most important, or where more complex XPath queries are used the using an XPathNavigator over an XPathDocument provides better performance than the XmlDocument. 

Thankfully it will be possible with .NET 2.0/Whidbey to use a tool (sgen.exe) to pre-create and compile Serializers.  Doug Purdy covered this in his PDC talk, see Scott's notes.  I believe this will make XmlSerializer the fastest approach to retrieving values from XML, but testing will tell.

Click here to download the sample code.

A month after the Indigo 'Kimono opening' at the PDC there's still a lack of clarity about what Indigo is, how it relates to other messaging technologies and what's the best way to start developing applications today.  While a lot of this was covered at the PDC my perception is that some of the message hasn't been ack'd successfully from the audience.  [Update: See my more recent post 'More on the Microsoft Messaging message' for some answers to these questions]

The Longhorn DevelopMentor mailing list had an excellent exchange yesterday on Indigo, which has lead me to highlight some areas where I'd like a clearer message:

• What's the relationship between Indigo, MSMQ, BizTalk and SQL Server Service Broker?  When and where is each technology appropriate? (Thanks to John Cavnar-Johnson for highlighting this)
• How does MSMQ fit into the Indigo longer term picture? Is MSMQ for within the Enterprise or will it focus more on outside the Enterprise messaging?
• In terms of EAI technologies, where should we look for this functionality - Indigo or BizTalk?
• Which parts of Indigo are going to ship in Whidbey and which bits will come in Longhorn?
• Given that the last version of WSE will be wire-level compatible with Indigo and that a future version of WSE is likely to support WS-ReliableMessaging, what are the benefits of Indigo other than the simplified programming model?
• Indigo is committed to supporting WS-* standards and interoperability, but what extra functionality will be available if the whole environment is made up of Indigo boxes?

I'm  know some of these were addressed at the PDC.  I'm still digesting some of it (for example, the PDC DAT406 presentation on the "Yukon" Service Broker shows how MSMQ, Indigo and Service Broker are positioned, though not BizTalk).  Other questions I'm still researching.

Steve Maine has a post on using a Gantt Wall to manage projects: it's big, visible, lo-tech and easy to manipulate and change.  I think it's a great idea in line with the XP/Agile spirit of getting away from traditional, rational (time and motion) style project management tools in favour of simpler approaches that are more likely to work.

I think MS Project style Gantt charts are useful, but too often they form part of a projects Death March when they are never updated.  Initial estimates of features become prison sentences for developers since they become set-in-stone.  Weekly meetings are focussed on how far behind developers are from their original estimates.  The whole project becomes burdened with the fact that it's slipped, is late or behind.  This is where lo-tech idea's like the Gantt Wall are so useful.  As Darrel Norton says:

the visible wall-Gantt is more than just a Gantt chart, it builds morale, serves as a social focal point for the team, and allows the celebration of success on a daily basis.  Until we can digitize all that, the low-tech solution will continue to reign supreme.

My Favourite Project Anti-Pattern
All of this makes me think of my favourite project anti-pattern which is to simply avoid dealing with any problems on the project and force everyone to work as many hours as possible each week for the length of the project.  It doesn't make the project any more successful, but it certainly saves having to deal with anxiety that can come from admitting and dealing with the situation where the project is off the rails.  When it comes to the deadline for the project, the work may not be done, but the excuse is difficult to refute: 'could we have worked any harder?'.

WS-Addressing is one of the newer WS-* specifications. The specification brings us closer to the reality of sending SOAP messages across multiple transports in an interoperable way. Going down the plumbing level with WS-Addressing helped me highlight the gap between the specification and implementations. Along the way there's a great SOAP Mail sample application using WSE 2.0 that highlights some of the promise of being able to send SOAP messages via HTTP and SMTP ...

There's been lots of talk about offshore development recently.  Larry O'Brien is chatting with John Montgomery about whether he'll still have a job as an independent developer , Rory reports that Dell have moved their call centre back from India.  The pragmatic programmers have released a t-shirt advertising their 'how to keep your job' with the slogan 'My programming job went offshore, and all I got was this lousy T-shirt'.  The current project I'm working on has bought their offshore development team on shore.

On top of all of this I notice today that Chris Stevenson, who's been the source of many good insights into .NET/Java/XP/Agile development at the Extreme Tuesday Club and who is also from my home town of Adelaide, South Australia, is going the other way and is moving to India for 6 months.  Funny old world.  I can't wait to read about Chris' experiences.

 

Many others have noted that the PDC Sessions are available online as streaming audio with synchronized PPT and demos, so who do I think I am posting it again? Well, here's my value-add: it's possible to download the Windows Media streaming audio and the demo videos using a free tool called Net Transport.  I'm using this to max out my broadband connection as well as letting me relive the PDC sessions I was too worn out to make when I was at the conference, on my daily four hour train commute.  Today I spent time on the train with Anders, tomorrow more Don and I'm considering using Doug Purdy's presentation as my new motivational tape to help me get out of bed in the morning.

To start each PDC folder has a file with a 0Media.asx file, which is an xml file containing links to all of the media files in the presentation.  Net Transport expands the xml and starts downloading the individual files.  It's even possible to pause the downloads.  Here's the link for Don's overview of Indigo for example  (you can test the accuracy of my transcription of Don's talk at the same time!).  Once you've downloaded the files, edit the Media0.asx file to use local paths to the individual files and launch it in Windows Media Player.  You'll have to move the slides yourself, but it makes it feel more like a slide night.

If I had more time I'd adapt early & adopter's script to download the slides to download all of the streaming content.

I've just listened to Ander's PDC presentation on C# 2.0 (TLS320).  C# is incredibly cool, the new features are great and it's amazingly useful to have Anders explain them.  Languages enhancements are as exciting for a developer as a better quality hammer is to a carpenter.

 

What's interesting about language enhancements is that to take advantage of them you have to teach your brain to think more like a compiler.  The Java language exam is probably the best example of this that I've come across.  Initially I thought that having to answer questions that the compiler would check was a waste of time.  Luckily a Java development mate of mine argued that it was important, especially in becoming a better developer.  Having done the exam, I have to agree with my mate that there are real benefits from being able to look at code and know how the compiler would treat it.

 

So listening to Anders it isn't just exciting to know that they've developed a compiler that knows how to compile impressive statements, but also realising that I'll have to train my head to do the same thing.  Luckily I find learning things like this a real pleasure.   Having new C# syntax to learn and play with is like a bright blue sunny sky for me after the grey clouds of my current project (arguing with a developer who stubbornly insisted that GOTOs were the best way to code a particular method).  Sure this looks a bit scary now, but I can't wait to get familiar with the syntax and to start developing with this extra power:

 

 

Some random points from Anders speech:

• Generics  These are great in general and I can't wait to be able to use them in anger.  I have a brilliant C++ developer mate who said he wont switch to another language unless he can use collections without having to cast.  Hopefully he'll be swayed.  I also like the use of the T.default and null checking that is thrown away by the JITer if it's a value type.
• Anonymous methods Anonymous methods seem similar to Java's anonymous classes.  Dare says: "[Anonymous methods] fix the lack of anonymous inner classes in C# by adding anonymous methods for use in the exact same situations most people use anonymous inner classes for".  It was good to understand how the implementation works under the covers to allow anonymous methods to access the parameters of the outer methods (Steve Maine waxed lyrical about this feature and lambda methods on the bus back from Universal Studios, now I finally understand his passion).  I love the idea of passing code as a parameter.
• Iterators I did once try and implement these in VB6 and it was a horror, so I'm glad the story is that writing iterators that can be consumed by the foreach statement is now easy.  Also, statements like foreach (Item x in Items.Subrange(10, 20)) are just cool.
• Partial types Anders explained there were two main reasons for this feature.  I was wondering since Juval Lowry couldn't really see much point in them when he did the .NET radio show pre-PDC. Basically they help people who generate code - the generated class can be regenerated without breaking code written in the other part of the partial class, and where there are long classes (though this is a bad idea in general).  It's also a key part of the XAML story. 
• Other enhancements Finally C# lets us have different accessibility levels on property set's and get's, so the set can be internal and the get is public.  This just feels more right.

Roy's at it again, campaigning for IntelliJ and asking Visual Studio to take notice and compete with the advanced developer experience provided by the IntelliJ IDE:

 "I felt like I was not coding alone ... if you decide you should have a new method ... you just type the method name and provide its parameters if necessary.  IDEA will prompt you with an unobtrusive popup with a suggestion to create a new method ... the method will be created in the corresponding class with the proper method signature (including correct parameters), so that you can implement it later"

I saw the same feature demoed by Erich Gamma, (the G in the GoF Patterns book!) when he came to London to talk about Eclipse.  I was so impressed the hairs on my arm stood up.  Forget clapping at the PDC, you know a feature is good when your body reacts with goose bumps.  As I mentioned in a previous post about refactoring, I think this feature would solve a lot of the frustration that Randy mentioned where Visual Studio's Intellisense clobbers any new methods or properties written in tests before the object exists.  It's the difference between thinking of the IDE as a text editor and thinking of it as a developer tool.

Luckily, IntelliJ are developing an IntelliC# add-in for Visual Studio that will bring its intelligence to .NET.  I can't wait to get it.

As Roy laments, what's Visual Studio doing?  I remember the glory days, back when Jim McCarthy (of Dynamics of Software Development fame) was working with the Visual C++ team making the IDE great.  Where's the passion, the spirit, the sense of competition gone?  For all the talk at the PDC there was noticeably little about Orcas, the Visual Studio version after Whidbey.  Visual Studio is good, but it can be even greater.  Let's hope that good things are in store.

Ted Neward has some great advice about how to review a technical manuscript that I wanted to highlight.  I'm using my 4-hour train commute to review a technical book that I'm pretty excited about.  Ted's article was helpful.  He also has a great way of looking at technical books - positioning books as a deal between a book and a reader:

when I write a book, I'm offering the reader a deal: "You trade me $40 of your hard-earned cash and roughly 40-80 hours of your non-refundable time, and in return, I will give you something you didn't have before, some knowledge that you find useful." If my book doesn't keep my end of the bargain, the reader takes it back to the store, but can never get back the time spent, and so is left feeling cheated and upset. In some books, the classics like "Design Patterns", or K&R, or "Rapid Development", the deal is so obviously lopsided in the reader's favor that these books go on people's shelves right next to the monitor, for frequent perusal. But the thing to realize is that it's all about the deal. Does the book fulfil its end of the bargain?

Searching for and finding books that keep their end of the bargain is one of my key passions.