Recently I've been doing a "tour of the trenches", helping a major client with their web-based applications for the Insurance industry. It's taught me that there's a lot of development work that goes on in companies that doesn't relate to the overall aim of delivering business value through software, specifically around the building of "platforms".  Brad Abrams noted something similar when he went on his road trip, there are a lot of companies building custom platforms.

From my experience a lot of these platforms turn out badly since most companies can't afford the time or money to build and create successful platforms.  Rather than writing platforms, I'm looking for ways to use Other Peoples' Code to build solutions that deliver better business value.  At TechEd this week I'm interested in evaluating Office 2007, and particularly the server platform, as a platform to build on.

Looking at the domain of Insurance applications it's easy to see why lots of developers want to write a platform.  The basic flow is similar in structure to many other types of financial trading:

• An application form, with one or more wizard-like pages, containing controls with single- and cross-field validation.
• The application form is sent to a rating engine (usually authored by Underwriters) which determines whether the business will generate a quote and if so, what the value of that quote is.
• After a quote is generated there are various sequential and state-based workflows around that can occur (e.g. accepting the quote, revising the quote, "binding" the quote to make it a policy).

With that in mind I'm currently looking at Office Server 2007 as a possible platform to build on.  Here's a sketch of what I'm looking:

• Create the application form in InfoPath (giving me the single- and cross-field validation), then publish it as a HTML form using Office Forms Server (there was a session at PDC as well).
• Use Excel Services to host the spreadsheet that contains all of the logic for calculating the Quote.  Once we agree on a set of named cells/ranges in the spreadsheet, the underwriters can keep control of it.
• Use SharePoint's support for Workflow to implement the Workflow.

Obviously I'm just sketching at this stage (I still have questions about licensing, performance and flexibility are key) and I need to do some prototyping, but overall I'm interested in the possibility of being able to build basic Insurance trading applications on top of Office 2007 as a platform, dramatically reducing the amount of code I will need to write.

I'm currently pondering whether the new server-side capability of Excel 2007 could radically change the face of many financial application.  Excel is so widely used in finance that many companies could describe it as their platform.  I'm currently involved in writing a web application for a group who use Excel extensively and I frequently think if we could work more harminously with this spreadsheet we could build the application for much lower cost but equal or higher business value.

In the keynote last night they demonstrated the Windows Compute Cluster Server 2003 running Excel Services to perform Monte Carlo simulations using Microsoft Excel.  Initially I thought this was implausible, imagining the response if someone asked, "Can we set up a cluster server in order to run my Excel spreadsheets?", but on reflection I think this could make a great deal of sense.  Even though I'm suspicious about the efficiency of this kind of solution, viewed from a different perspective I think it could represent great business value. 

It's amazing to think that Windows and Office could lead to a scenario where a front-office trader could write software that can be executed in a cluster - an arena that previously seemed the domain of C++ and hard-core developers.  On the one hand it scares me, since there will be less custom C# development for me to do, but on the other hand I like it a lot since re-writing functionality that Excel already implements takes a long time and doesn't give as big a "bang for the buck".

Obviously I need to do a lot more research on Excel Services.  There's a good introduction here and some more technical details on how you can call Excel through web services.

TechEd Boston started Sunday night with Ray Ozzie admonishing us that there's a 'Services Disruption' coming and proposing the slightly tongue-twisting 'Client Server Service Synergy' (and the lesser-used companion phrase 'Client Server Service Symmetry') as the way forward, with Microsoft offering more services in future. 

Ray with the usual dramatic structure of a presentation (much like what is recommended in the "beyond bullets" book), setting up that we're heading for a new disruption in the way that we work.  He did the standard mainframe -> mini computers -> micro computers -> client/server -> the internet ("the mother of all disruptions") -> peer-peer set of disruptions, personalised through his own life experiences with Lotus Notes and Groove. There was the standard argument that we need to be on the lookout for new disruptions.

The drivers that are changing things for today are the promise of multiprocessors (32 to 1000 CPUs inside a single machine - up to "oodles" in Ray's words) the reduction in storage space and the growing ubiquity of bandwidth.  The interesting point that Ray highlighted was the fact that in the past it was research and corporate environments that drove datacentres, whereas today these were being driven by experience with consumer market through search, advertising and consumer shopping sites.  Ray's point was that the benefits of investments in these environments will be felt more widely.

He wasn't terribly specific about how this would happen, though he was trying to position Microsoft's approach as a mix between the client services approach and external services in a "client server service synergy".  He showed windows desktop search searching over the internet, SharePoint sites and the local machine as example of this.  He also showed Microsoft Dynamics using 'business mashups' with Windows Live virtual earth.  He also mentioned the online/offline architecture of Groove.  There was much mention of the Windows Live set of products.  There was a Windows Live Identity that I hadn't heard of before - I'm assuming it's a rename of Passport? It might have been the jetlag but it felt like another moment of marketing fatigue (apparently MOM will become System Server Operations Manager as well).

The presentation hall felt very much like an aircraft hangar and the regular sound of planes overhead reinforced this.  Ray was the first keynote speaker I've seen wearing a suit jacket on one of these talks.  Obviously the chip-implant wasn't powerful enough to convert him to the standard casual 'uniform' of a blue shirt and a pair of chinos.

Developer Developer Developer, the free day-long conference for developers, by developers is back for a third time on Saturday 3 June.  The call for speakers is open.  If you've got a topic that you'd love to share with other developers in the community, please submit a session.

Attendee registration will open later, but you can always view some of the videos from DDD II last October.

Jamie Cansdale's done it again- he's added Test With -> Coverage to the TestDriven.NET Visual Studio add-in.  A picture's worth 1,000 words:

Once the tests are run it launches Grant Drake's (aka KiwiDude) re-write of Jeff Key's NCoverExplorer.  The NCoverExplorer then displays full coverage stats as well as illustrating the coverage of code blocks.   Here's the output from a run over the NUnit code:

This is awesome work that will make it easier to understand and work with code.

In the keynote yesterday, Don Box and Chris Anderson showed how Indigo can be used in a REST style by sending Plain Old XML (POX) over HTTP.  It’s a great testament to the quality of the Indigo extensibility architecture that they could do this.  By supporting both the REST and SOAP styles it takes the heat out of the debate that one is better than the other.

One of the Hands On Labs goes through the code they used in detail.  The example uses a HTTP GET request to return a plain XML payload, in this case a RSS feed.  An extra level of detail was the fact that the RSS added a custom element indicating an 'adult content rating' system, which was signed.  There were several interesting points from the Hands On Lab.

One complexity around supporting a REST style with Indigo comes from the fact that the HTTP GET request does not have any WS-Address-style action.  To get around this you can use the mapAddresssHeadersToHttpHeaders attribute of the HttpBindingElelement to make the URI of the HTTP GET request the To value on the Indigo Message object.

Since no action is specified on the HTTP GET request the ServiceContract needs to trap any unmatched action using the OperationContract's Action named parameter:

public interface IRestStyleRequestResponse
{
   [OperationContract(Action="*")]
   Message GetRequest(Message request);
}

Since the HTTP GET request on the HTTP transport in Indigo usually returns a description of the service, this needs to be turned off using the Description property of an instance of the ServiceHost type:

// service is an instance of ServiceHost
// stop the HTTP GET returning the site description.
service.Description.Behaviors.Remove(typeof(ServiceMetadataBehavior));

To hook in this binding a custom BindingElement extension was written.  The lab walked through creating a IChannelFactory and IListenerFactory and the rest of the necessary objects as part of the WCF stack.   Just before the transport binding element in the Binding stack there's a custom encoder which does the job of writing out the Message to the wire as plain XML, rather than a SOAP message (as Don Box said yesterday, 'it lathers the SOAP off the message on the way out and SOAPs up the message on the way back in').

Office 12 was unvieled for the first time during the Bill Gates' PDC keynote.  The main thing of note was a simplified user interface devoid of menus and toolbars and replaced with task-focussed tabs and the 'ribbon' bar.  The big question: does this represent a great leap forward in goal-focussed usability or will this be the 'New Coke' moment for Office?

Bill's initial comments:

• Promise of previous PDCs has been realised.  2000 was about .NET and XML Web Service, 2003 was Avalon (Windows Presentation Foundation) and Indigo (Windows Communication Framework) and Longhorn (Vista).  A shame that Hailstorm and ObjectSpaces have been so quickly forgotten, but software is a cut throat game.
• .NET is the most popular development platform in the word
• XML has been core and has gone through three phases: surface support in terms of tools that worked with it, then platform level support in terms of .NET and XML web services, then core in terms of integration with SQL Server and Office.
• XML has become persuasive.  RSS has given us notifications, XML Schemas are being used to create industry schemas and now with WS-* the enabling of protocols to handle security and other requirements.
• Standard comments about 'an exciting time', 'still not halfway through the PC revolution'.  "Best industry in the world", "exciting times"
• PC shipments are up 15% on last year to 200 million units a year.

Funny things that came up on the closed captions on the video screen as Bill spoke:

• 'Intel a mir a cash mode' -  'IntelliMirror cache mode'
• 'Things like our asses are driving up to do this' (RSS)
• Sin yer jistic (synergistic)

Overall I was incredible impressed that the subtitles kept pace and did such an accurate job.

Windows Vista Demo

"Clarity" Demos:

• Task bar now shows preview of documents when you mouse over them.
• Alt+Tab now shows a 'flip view' that shows a strip with the same document previews.
• Windows Key and Space Bar shows a 'stacked' window view with the windows stacked three deep.
• Quick search shows in all explorer views, plus the desktop sidebar.  Interestingly it now appears at the bottom of the start bar.  It will show search results in place and search the desktop and the internet.
• The thumbnail of files shows up in explorer view.
• The search on the explorer will search document metadata (e.g. author) and text - although unfortunately there was no visual highlighting of where the search appeared.
• Virtual folders are supported.  They are defined with XML files (simple elements like cope and tags) and allow you to see all documents on the PC.  These can be organised by metadata in a couple of new ways (stack, keywords, author).
• Documents can be 'painted' with metadata by dragging and dropping.
• The side bar (think side bar, same clock as PDC 2003) hosts gadgets, written in DHTML, script or WCF).
• Gadgets are a new feature that allows information to be displayed on a laptop lid (such as flight information, or ability to control music).  Some need the laptop to be powered, some don’t.

"Confidence" Demos:

• Phishing sites have increased 500%.  IE 7 has some new features to show phishing sites.  For example, if the site uses an IP address the address bar turns yellow and there is a security icon that warns this may be a phishing site.
• Clicking on the security warning allows you to report this site as a potential phishing site.
• Microsoft is going to host a phishing site review group that will investigate the possible phishing site votes (and similar, 'this is not a phishing site' link)
• The dynamic protection service which powers this feature is opt-in.  If a site has been reported and judged to be a phishing site then the address bar goes red and comes up with a warning.
• IE7 also has tab, as well as an 'all tabs view' where you have the ability to view all open tabs in a thumbnail view.  They can be saved as a common set - so you could save your favourite 5 sites as a set and open them up each morning.
• RSS is built into the browser.  This works with the platform RSS store so the sidebar can display the RSS feeds as well.  Viewing the xml feed behind the RSS shows up with a nicely formatted page.
• Microsoft CRM is using RSS to send notifications via RSS.

Office 12 Demos
Word

• Word used to have 1500 commands and 35 toolbars.  9 out of 10 request for features in Office were features already in the product.
• No more toolbars or menus!  There's now a 'ribbon' area that is task based - so you might be 'insert'ing in word or using tables.
• The keyboard shortcuts are still available, they just don't seem visible on any of the screens.
• The team think that 60 - 80% of the features are available in the new tabs.
• The tabs do make some tasks, like adding a header or footer much easier to use.
• There are improved tool tips with more text to explain the features.
• Using the Tables tab in Excel allows you to format tables in Excel.  Selecting the table formatting example shows the formatting in place (I didn't see how to undo this command)
• Apparently they are finding people don't need much training to use the product.  They 'don't need training wheels'.  It will be interesting to see how this flies with large company's IT directors.
• Word has a nice feature to insert drop-downs, showing the formatting in place.
• There is still a File menu.  It has useful features like Finalize that let you clear up hidden text, comments and other features.

PowerPoint

• Improved shape support - automatically converts bullet points into different graphics (cycle of boxes, graph with points on them).

Outlook

• Still has the command bars!
• Is now focused around a to-do list.  Now easier to create tasks off email.  Ability to flag email with a time to follow up so that they appear in the tasks.  Should improve some of the Getting Things Done style work.
• RSS feeds are built in.
• Support for different office documents and SharePoint emails.
• Attachments can be viewed in place.

Thoughts on the new interface
It will be interesting to see the responses since it is quite a change from the current Office user interface, though I think it is a positive change - ever time I visit my dad he always has 17 toolbars in word, with one button on each of them because the toolbars terrify him so much.  The new user interface will protect him from that kind of advanced customisation.

The user interface definitely seems to work much better than it would appear when initially viewed.  It will be interesting to see how corporate customers react and whether they think the new interface will be as intuitive as it is claimed, or whether the possible re-training costs might be an inhibitor to adoption.

Jim Johnson showed how System.Transacations will allow developers to use Transactions to write more reliable software in simple way that will work across different transaction coordinators while ensuring that performance cost of distributed transactions is only paid if the code uses them.  I had a great time talking with Jim at the last PDC and it was exciting to see him demonstrating this technology that will ship with .NET 2.0, as well as a demo of the transactional file system that's coming with Longhorn Server.

Transactions: Ensuring reliability and resilience
Jim spoke about the philosophy behind the System.Transactions work.  He views transactions as a reliability aid that helps developer write resilient applications that can recover from errors and return to a consistent state, even in the face of scalability and concurrency challenges.
 
Jim showed two methods that were both written to correctly handle multithreading and concurrency, but that they may not work correctly if one depends on the other, since it would require code to trap errors and handle rollbacks.  He argued that transactions provide a simple way of ensuring that the two could work together in an atomic way, behaving correctly even in the case of errors, without a lot of complex code, which is the major use case for System.Transactions.
 
System.Transactions: an efficient, unified managed code object model
The goals behind System.Transaction are to make transactions ubiquitous.  In order to do that they had to make using transactions simpler for developers and overcome the perception that they were too slow.  They have answered the complexity problem by designing a managed code object model in System.Transactions that uses the same code to work with both local in-memory transactions and distributed transactions.  They have reduced the performance issues by ensuring that transactions only enlist the necessary resource managers.
 
Using a transaction involves some fairly simple code: 

Jeffrey Richter combines a deep technical knowledge of .NET with a great sense of humour and honesty about where things went bad, to provide a useful overview of what's new in C# 2.0 and .NET 2.0.  I find sessions like this useful in learning the story or intention behind features in the new technology, in a way that helps me understand them better.

The morning session covered an introduction to the new features in .NET 2.0 mostly around generics and anonymous methods.  Seeing some demos on the new yield statement helped me understand more about how it works.  He showed how you could use the yield statement to recursively walk all of the files and directories in the file system.  The demo was simply writing out the paths to the console window and as it was running there were obvious pauses in its performance.  Jeffrey said that these pauses were likely to be due to garbage collections, which highlighted that the ease of use of the yield statement came at the cost of performance (a class was created per directory/file which put a lot of pressure on the working set and garbage collection).  Jeffrey mentioned the C# team have a solution which reduces the performance costs of iterators but it wont ship until after C# 2.0.

He mentioned that another approach to getting the flexibility provided by the yield statement without the poor performance, was to use generics and anonymous methods instead.  For example, many of the generic collection types in .NET such as Array and List have useful generic delegate types, such as ForEach(Action<T>), FindAll(Predicate<T>) as Sort(Comparison<T>). 

As an example, I can sort a list of string types and print them to the screen in just two lines of code using this techniques:

List<string> names = new List<string>(new String[]{ "Timothy", "Benjamin", "Samuel" });

names.Sort(String.Compare);

names.ForEach(Console.WriteLine);

Whilst I've enjoyed these methods before I did enjoy hearing Jeffrey's positioning of these techniques as a pattern to use instead of the yield statement.  Jeffrey also mentioned that Microsoft had received many requests to add richer generic functionality to the generic collection classes.  His company, Wintellect, have been engaged by Microsoft to write these collections and make them available for free download, and they are currently available as the PowerCollection library.  Jeffrey believes that this functionality might eventually be made part of the Framework Class Library in future versions of .NET.

The great thing about Jeffrey's presentation is that draws on his experience as a consultant to the CLR team.  For instance, he mentioned that having different security levels on properties, a 'new' feature for C# 2.0, was originally part of .NET 1.0 Beta 1 but that it was removed since the C# team viewed properties as just a special kind of field.  From this perspective, having a different accessor made no sense.  But as Jeffrey said, after a large amount of customer features, the C# team has put this feature back. it just took them five years.

Jeffrey is also not afraid to mention the things that the CLR team have done badly.  He spoke about the justification for the static class feature of .NET 2.0.  Apparently in .NET 1.0 the Environment class that only had static methods, but late in the beta cycle a developer added a HasShutDownStarted method but forgot to make it static.  On top of this the Environment class had a private constructor so it was impossible for anyone to create a new instance of the Environment class and call this method.  The worst part of this is that it showed that the developer had not written a single test to check the method that he added!  If they had written a single test they would have seen that they could not have compiled the code that tried to write this method.

The session helped me see the benefits in understanding how a particular language feature works, particularly understanding whether it involves the language compiler or involves the CLR.  This knowledge provides the background behind some of the constraints on the features (such as why you can have a catch block inside an iterator).  The tension between the language compilers and the CLR was highlighted in the recent post-beta 2 changes to more fully support Nullable types in the CLR.  Don Box's idea last year was that many of the changes going forward are going to be driven by 'syntactic sugar' using the language compilers, while Jeffrey was more upbeat that there's still innovation happening inside .NET and the CLR.

Jeffrey is razor-sharp technically and very funny.  My favourite comments so was "Indigo loves using attributes.  They use attributes so much that you no longer have to write code"

The afternoon session started with a coverage of partial types, before spending a long time on CLR hosting and how the SQL team helped ensure that the CLR could be hosted in a secure and reliable way.  While Jeffrey is able to explain this stuff really well, it was a hard slog to keep up with the section on Constrained Execution Regions ... so I've nipped off to watch David Solomon and Mark Russinovich's Windows Internals section (or Sysinternals tools seminar as it seems to be).

I did a presentation on Indigo to the VBUG London group last night.  The turn out was pretty good, although it was obvious that many English football fans were missing (though they probably would have enjoyed the User Group more than the humiliating defeat).

Off the top of my head, here are the things that keep standing out for me about Indigo:

• The power of separating choices about the binding from the code that a developer writes.  Being able to switch from the tcp with a binary encoding, to reliable messaging over HTTP, to MSMQ just by changing the configuration file is great.
• The simplicity of the programming model.  The majority of the functionality that required Enterprise Services, such as Instancing and Sessions, are available using attributes on either an interface/class or a method such as ServiceBehavior or OperationBehavior.
• The fact that the details of the protocols are hidden under the covers.  Instead of focussing on the WS-* protocols, the focus is on the functionality required.  This has taken me a while to get my head around after having used WSE, which is powerful, but involved having to understand a lot of the details of the underlying protocols.

I'm looking forwad to some of the Level 400 sessions at the PDC, such as Kenny Woolf's presentation on Channels.

As Mehran noticed, I couldn't help but show off the Refactor! tool that's shipping for free in Visual Basic 2005.  This was in response to an attendee who believed that Visual Studio was 'a multi-megabyte bloated version of notepad'. 

Mike Taulty has a sign-up page for UK PDC Attendee Bingo.  I'm going to be at PDC and it would be great to catch up with others, especially those from the UK.  There's already a very interesting bunch of people heading over from the UK whom I look forward to catching up with.

Strong bodily reactions are a great way to measure the impact of new technologies.  While many US technical conference crowds like clapping when they see new features demoed, I think the best reaction is a more modest bodily reaction. I'll never forget the time I shivered after seeing Eric Gamma demonstrate the Refactoring support in Eclipse a couple of years ago.  One area of .NET 2.0 that is generating strong bodily feedback is ASP.NET 2.0's support for Custom Build Providers.

First there was Fritz dropping his jaw when he created a custom build provider that took his custom XML metadata file which he dropped into the app_code directory that automatically generated the strongly-typed classes for him.  Tonight I discovered that Kirk Allen Evans had a similarly-sized bodily reaction when he created a Custom Build Provider that automatically created XmlSerializer classes that allows you to drop a schema into the app_code directory and automatically generate the XmlSerializer class.

I'm with these guys: Custom Build Providers are cool (bodily reaction: light goosebumps).  I'm passionate about finding ways of driving application development through the use of metadata files and the Custom Build Providers are an innovative way that ASP.NET provides to support this kind of development.  The only downside I can see is that the creation of these classes is slightly 'automagical' and requires that developers understand what the framework is providing under the covers.  However, I think this initial learning curve is more than compensated for by the productivity that Custom Build providers enable.

Kirk Allen Evans has an interesting post about proposing a distributed architecture for a client where he finds, after doing performance testing, that the speed difference between different transports reduces as the amount of work done within each call increases.  Based primarily on this finding he's recommending ASMX web services as a better strategy than using COM+/Enterprise Services for his client.

I remember coming across this arguement at TechEd 2004 in Amsterdam where Christian Weyer spoke about some Thinktecture research which showed that the difference in speed between Web Services and other distributed technologies (remoting, enterprise services/COM+) was significant if the work done in the distributed call was minimal, but once a reasonable amount of work was being done the performance difference became less and less important.  Worrying about the speed of the transport without thinking about the amount of work done in each call is a little bit like worrying how fast a car can rev if you have it in neutral.

Of course, this is all keeping with the long-known guidance from the Indigo team, which is to start with ASMX and only go to other options if you need specific features that they provide.  Richard Turner's recent white paper provides more details.

Save the date - Saturday 22nd October 2005 is the next DeveloperDeveloperDeveloper day!.  We're planning another free community-driven day of talks on developer-related topics.  Craig Murphy has more details.  Jonathan Hodgson has updated the event website (including a countdown ... 84 days to go!).  There's a form you can fill in if you'd like to present a session.

For now, feel free to join in the discussion on the Channel 9 site.

Bill Wagner, fellow RD, and author of the excellent 'Essential C#' has an article on Code Project that debunks the conventional wisdom that String.Equal is better than Operator == for strings:

... for the string class, Operator == and String.Equal() will always give the same result. Period. You can use either interchangeably without affecting the behavior of your program. Which you pick will depend on your own personal style, and the compile-time types you're working with.

The == operator is strongly typed, and will only compile if both parameters are strings. That implies that there are no casts or conversions inside its implementation.  String.Equal, on the other hand, has multiple overloads.  There is a strongly-typed version, and another version that overrides the System.Object.Equals() method. If you write code that calls the override of the Object version, you will pay a small performance penalty. That penalty comes because there is a (slight) performance hit for the virtual function call, and another (slight) performance hit for the conversion. Of course, if either operand has the compile-time type of object, you'll pay for the conversion costs anyway, so it doesn't much matter.

The bottom line on equality and strings is that if you use the method that matches the compile time types you're working with, you will certainly get the expected behavior, and you'll likely get the best performance. If you want all the details, see Item 9 in Effective C#.

I'm posting this because I've had this discussion a few times and believe that Bill's explanation is the best I've seen.

I'm on the Ask The Experts stand here at TechEd (back tomorrow and Wednesday at 2pm) and the first person I spoke to today asked me how to solve a socket access permission when using a when using a tcp service with WSE 3.0 Tech Preview.  Working through this also answered a second common question people have, which is 'what's the default port number that WSE uses when "soap.tcp://localhost/" is specified as the service address?'.

The exception in questions was:

An attempt was made to access a socket in a way forbidden by its access permissions.

This occurred when I was trying to run the TCPStockService sample application and here are the key lines:

Uri address = new Uri("soap.tcp://localhost/tcpstockservice");
// This starts a TCP-based listener if there isn't one already started.
SoapReceivers.Add(new EndpointReference(address), typeof(StockService));

Digging through reflector it turns out the the constructor of the Microsoft.Web.Services3.Messaging.SoapTcpTransportOptions options sets the defaultPort:

So, 0x1f91 in Hex turns out to be 8081 in decimal, so by default the WSE tcp transport listens on port 8081 if no port is specified.  Now the exception message made more sense, since I had another application listening on port 8081.  Changing the port number, or stopping the process that was listening on port 8081 solved the problem.

Thanks to the MSDN Product Feedback centre I can send a suggestion to improve this error reporting straight through to the product team.

It's been a long time since I last posted.  I've been busy with a mix of presentations, work and life, such as:

• Helping Kalido ship version 8 release 2 of their two core products, the Kalido Dynamic Information Warehouse and the Kalido Master Data Manager. 
• Speaking at the Microsoft UK Architect Council and Architect Forum about Services on the Microsoft Platform today.  You can find the slides on the Connected Systems Architect forum page.
• Presenting to the London .NET User Group on Programming Indigo.
• Facilitating a 'park bench panel' on Smart Client vs Web Development at the Microsoft UK MSDN Roadshow in London.
• Doing plenty of behind-the-scenes or out-in-the-community work with local User Groups.

Some technical topics that have been taking my time:

• We set up CruiseControl.NET in my team at Kalido.  This just rocks.  It monitors source safe for checkins, checks the code out, runs a Nant script (these are great: you can compile the code using a solution file (instead of hassling with the vbc.exe or csc.exe directly) to build the code, then it runs our NUnit tests as well as providing coverage reporting with NCover and even duplicate code checking using Simian.  It all comes together with a tray icon that monitors what's happening on our build server and lets the whole team know if a build fails.  Tracking down the reason for failure is easy since there's a central web page that pulls together the log file from all of the tools (including comments from the source safe checkin box - finally, an easy way to see these comments!).  It's so exciting I'm rambling even while blogging about it.  The key point for me was that it only took one developer a couple of days to set up, even with no previous experience of any of these tools.
• Schema design.  I'm working on a new WSDL interface and have been playing around with various strategies to build the interface most efficiently and effectively.  More on that in future posts.
• Consolas.  Like Scott Hanselman, I love this font.  I've embarrassed myself my gushing about this to colleagues who are less aesthetically inclined than I am.  It's a mono spaced font, which is part of the Longhorn font set, especially designed for reading code on screen.  I've also set it up as the Windows Console Font as well.

I've also been enjoying time with my daughter who's now walking, starting to 'talk' and generally just getting into trouble (I spent last Saturday morning in the accident and emergency section of the local hospital after a walking accident.  The doctor said 'ah, we always see a lot of head injuries from kids who've just learnt to walk').

Around 160 delegates turned out on a sunny warm English Saturday to attend the DeveloperDeveloperDeveloper day.  Judging by the comments on Channel 9 (great to see so many UK people using that site) it seems to have been a great success.  It was great to catch up with so many different people, such as Santosh Benjamin, Phil Winstanley, Ian Cooper, Mike Taulty, Jonathan Hodgson and many other blogless, but nevertheless interesting people.  Simon Harriyott moblogged the event and was kind enough to give me a lift back into Reading.  He also continued his analysis of Australian accents in IT ("data down under").

Although I enjoyed giving my talk 'Web Services in .NET 2.0: Solving Today's Problems' I made a fundamental mistake of staying up too late the night before the talk.  Filled with the pleasure of finally having Visual Studio 2005 installed on my laptop (Jon Rowett caught the bug as well), I got carried away crafting a flashy demo with streaming JPEG images (using IXmlSerializable) and databinding to the results of the webservice calls.  As a result I was 'dog tired' as Ian Smith noticed and didn't present as well as I would have liked.  Dave Oliver wasn't so sure of the value of using IXmlSerializable to stream a large file over web services.  Lesson learnt.  Next time I'll get a good night's sleep and focus on small, easy to understand demonstrations.

The highlight of the sessions for me was seeing Brian Long go through .NET debugging capabilities.  He obviously had a command of the topic, managed to demo the command line debuggers for an hour without a single typo and had a great dry sense of humour.  He's done a lot of talks with the Developers Group here in the UK, which came out of the Borland Developers Group.  I love the fact that the .NET community has benefited from so many people with Delphi experience. 

To finish, Jon Rowett has a good write-up of the day as does Dave Oliver and Richard says "All in all, a very worthwhile way to spend a Saturday - the kind of training day that usually would cost the best part of £1000 per participant. Something I’d definitely do again if the opportunity arose."

Thanks again to Craig Murphy for taking the lead in organising the event and Jonathan Hodgson for doing the website.

If you like a free day of training on Microsoft technologies presented directly by developers with experience using technologies then sign up for the DeveloperDeveloperDeveloper day that's being held on Saturday 14 May at Microsoft's Thames Valley Park campus.  Don't wait to sign up - we're 75% full already and based on similar events in the US we're likely to sell out completely.  Microsoft have graciously provided the venue and are handling the registration and logistics, but all of the speakers are independent community developers! The www.developerday.co.uk site has a full overview of the event, the agenda and sessions and the speakers involved.

There are three different tracks with 6 presentations in each.  Here are a sample of some the talks from developers I know that I'm looking forward to:

• Alex Homer and Dave Sussman will be talking on SQL 2005 and another secret topic
• Ian Cooper, who runs the London .NET User Group, on Patterns of Data Access in .NET (here are my notes from a previous version of this talk I heard)
• Phil Winstanley, ASP.NET MVP, will be talking about preparing for ASP.NET 2.0
• Craig Murphy, Jedi Scrum Master and XML Web Services MVP, on Managing Iterative Development with Scrum
• I'll be covering What's new with Web Services in .NET 2.0.

I'm also looking forward to hearing about custom attributes in .NET, refactoring, test driven development, debugging tips and writing custom FxCop rules.

As well as the presentations it's also a great chance to network with other .NET developers.  For instance, I know that Jamie Cansdale is likely to be there, so if you've got any questions/comments for him on his fantastic TestDriven.NET addin there's an opportunity.

A big thanks to Mike Ormond and his team (Mike Pelton who first posted about the event) for providing the venue and logistics support.

I've just landed back in the UK after a three-week round the world holiday to Australia and back via Redmond for the Indigo Software Design Review (SDR) last week.  The SDR gave me a chance to play with a slightly updated Indigo build than the publicly available Indigo Community Technical Preview.  I was going to list the highlights of my week but then I saw that Omri Gazitts, from the Indigo Team, posted a list of his favourite Indigo features that maps pretty closely to what I was going to mention.  My key take-away was that the team have done a great job designing Indigo so that it is feature-rich and easy to use with good extensibility points throughout the system.

While I've been away Mike Taulty (Layers of Indigo, IInputChannel and IRequestChannel, Channel encoding and filtering messages) and William Tay (Message Tracing, Logging and Activity Management and the details of the default Indigo bindings) have both been doing some excellent job of kicking the Indigo tyres and exploring the CTP builds.

I'll write more when I get a fresh VPC image installed with the CTP bits.

As Rebecca Dias mentions, my article ("Why WSE") covering the high-level reasons to use WS-Security has been published on MSDN.  It covers the provides benefits WSE provides, such as end-to-end message-level security, content-based routing, and policy through leveraging the WS-Security, WS-Addressing, and WS-Policy specifications. 

Let Rebecca know what you think of it.

If you want a longer article on the same material I'd recommend fellow-RD and web services enthusiast, William Tay's piece on Solving Real World Business Problems with Web Services Enhancements in .NET

Microsoft Research in Cambridge have released the WSE Policy Advisor for Microsoft Web Services Enhancements (WSE) 2.0.  The Policy Advisor is an an unsupported tool that acts as a security diagnosis tool for WSE2 policy files (think of it as an FxCop for web service security policy files).  It analyses the policy file for common security vulnerabilities, provides a description on the risk and remedial advice.  It can be launched as a stand alone application or from the policy tab of the WSE Settings Visual Studio add in.  If you are intersted in WSE 2.0 and Policy then  download the Policy Advisor and run it against the sample files that ship with WSE 2.0 and send the research team feedback.

I've been a fan of using policy files to secure web services with WSE for a long time.  As Clemens says, authoring a policy file by hand is pushing things too far.  In combination with the WSE Settings add-in the Policy Advisor provides a great service for anyone wanting to understand and apply policy files, without having to get too focused on the XML angle brackets. The help file contains a list of all the problems the Policy Advisor can detect and is an excellent learning resource if you want to learn about the purpose of many of the policy elements.  For example:

This policy accepts messages with unauthenticated or elements. (Alarm)
Risk: The message is authenticated, but authentication does not cover and . Those elements are often used to implement replay protection, and should thus be authenticated. Otherwise, an attacker may intercept a message and generate a series of slightly different messages that will be accepted as distinct, genuine messages from the original sender. (The risk may be mitigated if the transport provides integrity protection, or if the recipient implements replay protection using other authenticated elements.)
Advice: Insert wse:Timestamp() and wsp:Header(wsa:MessageID) in the element in the assertion.

It also has warnings about the evil that using unencrypted UsernameTokens, though I'd highlight Keith Brown's excellent MSDN article on Securing the Username Tokens with WSE 2.0 as the best source of guidance in this area.

Here's how the tool integrates with the WSE Settings Visual Studio addin:

Below is a screenshot of the report that the Policy Advisor produces, in this case it is reporting against the secure conversation sample that ships with WSE 2.0.  The top part of the window describes the report, the bottom tree view highlights all of the issues found and the relevant policy for each problem.

Seeing Google maps today helped me realise the power of the browser as a cross-platform development environment.  I believe that the combination of client side callbacks with DHTML and JavaScript dramatically reduce the need for Java Applets or ActiveX controls in web-based applications.

The problems usually levelled at browser-based applications are that they lack the responsiveness and rich interaction experience provided by traditional forms-based applications.   I think the Google apps (Google Suggest, GMail and now Google Maps) prove this point wrong.

I was in a discussion at work today where a colleague was arguing that any browser-based application requiring rich drag-and-drop and data entry grids would have to use ActiveX controls or Java applets.  However I think that developments over recent years in cross-browser support are showing that this kind of functionality can be achieved in the browser.

Here are some links that convinced me further:

• This amazing cross-browser JavaScript DHTML API, Drag & Drop for Images and Layers.  It's a 38KB file that supports drag and drop (including making the image semi-transparent while being dragged) and a whole range of other commands.
• The ability to retrieve new data into a web page without having to refresh described in this overview of the XML HTTP Request Object and its cross-browser support (all wrapped up nicely in ASP.NET 2.0's support for client side callbacks).  What makes me laugh is how this functionality was originally provided through the Microsoft XMLHTTP object from MSXML but Mozilla and Safari have copied it with an XMLHttpRequest object - who said that IE wasn't innovative?
• A range of commercial DHTML grids such as ActiveWidget's Grid 1.0, including this sample showing editing.

I can't wait to see more development around these technologies (I'd really love FreeTextBox to use client side callbacks to autosave my blog posts to save me from losing so many posts!).

As Christian Nagel notes the INETA Europe - UK and Ireland web page launched today, including the new INETA UK and Ireland Regional Speaker Bureau.  The Regional Speaker Bureau is a collection of technology experts and highly-rated speakers who are now available to present at regional User Group events.   If you’re in the UK or Ireland and would like to hear any of these people speak at your User Group then tell your User Group co-ordinator to contact me or let me know directly (benjaminm at benjaminm.net).

INETA already has a European Speakers Bureau, but to highlight the local talent and encourage more events we’ve established the INETA UK and Ireland Regional Speakers Bureau.  This group includes:

• Peter Blackburn – SQL Server MVP co-author of Hitchhiker's Guide to SQL Server 2000 Reporting Services, currently receiving a 5-star review on amazon.com
• Jonathan Greensted – ASP.NET MVP
• Ian Griffiths – C# MVP, Technical Author and an expert on Avalon
• Rob McDonald – Technical Author, internal trainer to Microsoft and founder of co-founder of DevTrain and highly-rated conference speaker.
• Benjamin Mitchell – Regional Director
• Graham Parker – VB.NET MVP and retiring chairman of VBUG

These speakers are on top of the three existing UK members of the INETA European Speaker Bureau:

• Alex Homer – ASP.NET MVP, Technical Author, Conference Presenter
• Richard Grimes – Visual C++ MVP, Technical Author, Conference Presenter
• David Sussman – ASP.NET MVP, Technical Author, Conference Presenter

This work is part of my role as the INETA User Group Liaison for the UK and Ireland which I’ve been doing since late last year.  My aim is to further improve the .NET Community in this corner of the world by ensuring the regional User Groups get access to great speakers for their meetings.  If you know of a great speaker who’s not on the list (we’re currently looking for MVPs, MCTs, Technical Authors or anyone with a proven track-record of great presentations), or you are interested in speaking at User Group events yourself, let me know.

Mark Fussell, the WSE Program Manager posts that since the MTOM specification has moved to a full recommendation it means that it can be implemented in WSE 3.0.  This is great as it means that we'll have a way of solving the problem of how to send binary data in an efficient, interoperable way that plays nicely with the other WS-* specs .

For anyone who wants to take the pulse of the UK .NET bloggers, James Crowley who runs the Developer Fusion site, has put together a page of aggregated UK Developer blogs, with an RSS feed as well.

Ian Cooper gave a presentation last night's London .NET User Group on Data Mapping Patterns in .NET.  He explained many of the patterns from Martin Fowler's book Patterns of Enterprise Application Architecture.  He started with the basic Transaction Script pattern through to the Table Model and finally the Domain Model.  Along the way he demoed the Data Access Application Block (which to my surprise, only half the audience admitted knowing about).

I enjoyed seeing many of these patterns shown in action using nHibernate.  I haven't looked at the ORM frameworks for a while and was pleased to see how far things have developed.  Ian recommended the book 'Hibernate in Aciton' by Christian Bauer and Gavin King as a good introduction.  You can read a sample chapter and a book review on theserverside.com.

Ian's main point was that you should look to use nHibernate or another existing ORM tool rather than writing your own (avoid the ORM Vietnam issue that Ted Neward mentions), but to be careful not to see ORM tools as a hammer that makes all problems look nails.

Graham Parker, the retiring VBUG Chairman, was on before Ian talking about Java and .NET Interoperability.  I missed the start of the session but there was lot of good discussion from the attendees.  A large number of people  were aware of the Mono project and it's recent developments such as support for ASP.NET, Windows.Forms and ADO.NET.  There was also discussion about how Source Forge Source Gear are using Mono for their Vault commercial product.

Max Kington chipped in from the floor with a number of good insights based on his experience with Java.  I had a good chat with him afterwards on a range of topics from grid computing, web services to his claim that '2005 is the year of the domain specific language'.

All up another good LDNUG event.  Ingo Rammer is going to speak at the next event on Wednesday 23 February!

John Lam, together with Dominic Baier, shows how to implement a hex encoded SHA1 hash of a password plus a scope URI, as suggested by Keith Brown an improvement over WSE's default password handling for UserNameTokens.  Nice work.

In a recent Opinari email (I can't find it on his website), David Chappell defends his belief reuse is one the fundamental benefits of Service-Oriented Architecture and that SOA will be more successful than previous approaches such as object-oriented or component-oriented development.  His rationale is that services operate at a higher level of reuse than binary classes, and they solve many of the human problems that prevent reuse, such as having to have one single view of business entities and forcing reuse through 'draconian' human policies.

Here's my summary of his three major arguments:

1. Service orientation will lead to better reuse of business-oriented software because it is about sharing application functionality at a higher level than binary classes:
   • Evidence for this comes from the fact that the most successful examples of business reuse comes from packaged applications such as SAP and PeopleSoft
   • Attempts to share business classes, such as San Francisco, haven't been successful.
2. Previous approaches to re-use have failed due to human reasons rather than technology:
   • It is hard to agree on a definition of a business entity, such as customer. 
   • Even if there is agreement about the 'one right view' then it is likely to change over time.
   • It is difficult to force developers to reuse software - given a choice (or allowed to get away with it) most developers will reinvent rather than re-use.  Where reuse has been achieved with previous technologies, it was mostly through 'coercive management structures'
3. Service-orientation overcomes some of the human issues involved with reuse because:
   • Services don't require a single definition of business entity, just conformance with a particular schema. A service-consuming application may have a different view of a business entity as long as it can map to the service's view of that business entity.
   • Services can better handle changes over time.  Service-boundaries are a better way of insulating consuming applications from changes to the internal implementation than base classes and binary inheritance as there is a cleaner separation of external interface from internal implementation.
   • Since services are more explicit with their boundaries, and often these boundaries are physical, it is harder for developers to find ways around them.  This is similar to Clemens' point that well designed services do not share databases.  David's argument is that these design decisions make it easier to enforce reuse.

I like his focus on the human problems that plague reuse.  I also agree that sharing schema is a better approach than sharing class (part of the four tenets of service-orientation), and that using interfaces rather than inheritance leads to more flexibility to cope with change over time (a well-known principle).   I find the argument that service-orientation will make re-use easier to enforce less persuasive, since this seems more to do with deployment than the design philosophy (a component-oriented application could also reuse components that encapsulated a private database).

Scott Guthrie has written an excellent post about the ASP.NET 2.0 development process and the steps to Beta 2 and a 'go live' license.  This is an interesting post on two levels.  First, I love learning about how Microsoft builds software (I love the game of shipping of software and finding out ways to improve it).  Second, I think ASP.NET is an incredible product and am excited about being able to 'use it anger' (as they say in the UK) soon.

Here are my notes on the key concepts from Scott's post.  It's interesting to compare the ASP.NET 2.0 process to Hervey's description of the WSE 2.0 process.

Zero Bug Bounce (ZBB).  A point where there are no open bugs for a particular milestone that are older than 48 hours.  The point is "to push teams to sustain a high rate of bug fixing over a period of time". The present goals is to avoid pushing any bugs past Beta 2, meaning that there will be a true zero bugs bounce occurring.  There is a lot more information on the ZBB concept: Eric Gunnerson provides a definition,  Ron from the Visual C++ team talks about the three types of bounce, Larry Osterman a long-term Microosftie talks about it in the context of Zero Defects and it's also part of the Microsoft Solutions Framework (MSF).

Regression Rates.  For the managed code parts of ASP.NET the 'regression rate is around 4% -- meaning approximately 1 in 20 fixes introduces a new bug in the product'

Bug Rates.  The test team opened 514 bugs last week, and the dev team closed 648 bugs.

Automated Testing.  The team has 102,000 test cases(!).  Automated FxCop runs are done over the product.  There are two levels of check in test suites.  There are base-level unit test coverage of the product prior to checkin that have a block level code coverage of around 64% and take several hours to run.  More extensive tests are run each night and take around 12 hours to run.

Checkin process.  All code is peer-reviewed (even for senior devs).

Security push.  After the ZBB returns to zero there is:

'an in-depth multi-week security analysis of the code in the product … The goal at the end is to feel confident that the product is solid and ready to be attacked by thousands of hackers out there while running on the Internet.'

Tell and Ask Mode
In Tell mode any team is allowed to fix whichever bug they wish, but they have to be able to stand up and justify their decision to the 'central division ship room'.  This helps slow the rate of checkins, which reduces the chance of regression issues, helping the quality of the code base increase.

Ask mode means that the central division ship room must give permission to make a change.  During this mode, the stress testing can continue.  The low rate of checkins makes it easier to identify the source of stress-test failures.

Go Live License. Scott mentions that Beta 2 will provide the first 'go live' license.  I can't wait for this to happen.

Another thing I think is admirable is that the ASP.NET 2.0 release seems pretty close to what Scott mentioned back in January as Chris Garty clarified off my notes from Scott's presentation.  Although they have had to cut and descope some features (See this article from Jonathan Goodyear, fellow RD) the team look like they've done pretty well sticking to schedule.

In an interview with Mary Joe Foley, Joel Spolsky says that Indigo will make building applications easier, but it won't lead to a whole new class of applications.

If you look at the different aspects [of Longhorn], like Indigo, I'm not really even interested. It's just a big communications architecture that makes it easier for programmers to build communications things. But there's no application you can't build right now because you don't have a good communications architecture. It might be harder (without it), but it's not going to enable you to build a whole new class of applications.

I think there is some truth in this statement.  As Steve Swartz has said, Indigo is a replacement of the infrastructure and an evolution of the programming model, through simplification and a unification.   As Clemens has demonstrated that you can use existing Microsoft technologies (ASMX, MSMQ, Enterprise Services, and Remoting) to do many interesting things today.  If you need to build a service oriented application it’s worth starting today and not holding off until Indigo arrives.  However, I think Joel underplays the benefits of making the programming model for service oriented applications simpler. 

The problem with building service oriented applications today is that it takes talented developers, like Clemens to do them.  As Don has mentioned, one of Indigo's goals is to make things so easy that he will have to join MS to survive. 

The applications that are developed on Indigo may not be ‘a whole new class of applications’ (though I think it’s highly likely that the killer apps of the future will be based on top of communication technologies), but the fact they are easier to develop them should lead to more of these types of applications being developed.  This is in the same way that VB enabled an increased number of COM applications that could have been built with C++.  It’s an interesting argument as to whether VB lead to ‘a whole new class of applications’ being developed.

There are several questions that I still have about the impact of Indigo:

• Where will the business imperative to build applications that communicate more come from?  Is it that there is a demand today but businesses aren’t developing the applications because of the technical cost? 
• Will the increased ease of development encourage businesses to build applications that they haven’t considered building today?
• Will the pool of distributed application developers increase, or will Indigo just be adopted by the developers who are already building distributed applications?
• Will the increased simplicity of Indigo mean that the architectural concepts behind service orientation will be more widely used?  For example, I’m puzzled as to why MSMQ doesn’t get more love – message queuing is an important architectural design and MSMQ works very well with a very simple programming API – but I don’t see it used as widely as I think it could be.

Thoughts? Comments?

Here are the code samples I used my webcast on WSE Messaging last Monday.  The webcast will be available for download from this link soon [update: it's available by clicking on the register button and getting through the login page].

Here’s the code from the demonstrations.  The major sample – a suggestion service – is based around Keith Ballinger’s talk at TechEd San Diego and my version of this talk at TechEd The sample shows:

• A Windows Form client that sends a message to an ASMX web service.
• The webservice logs the message to a file, before sending a one-way message onto another Manager service using WSE’s support for sending SOAP messages over TCP
• The Manager Service is hosted in a Windows Form client that receives the message using the WSE ISoapInputChannel and its in-memory queue.  This means the messages aren’t displayed until the manager explicitly requests them.  The Manager Service also acts as a publisher of these messages to any service that has subscribed.  The implementation shows a simple Publish/Subscribe model using SoapServer/SoapClient.  When the messages are retrieved from the memory queue they are published to all of the subscribers.
• The Boss Eventing Service is the final Windows Form client.  It sends a Subscription message to the Manager Service and receives notifications when the Manager Service retrieves a new message.

Other resources I mentioned/recommend:

• I thoroughly recommend Aaron Skonnard's Hands On Lab on WSE Messaging (available on this page).  It provides some great walk-throughs showing all of the key features of WSE messaging (including how to build your own lightweight chat app using SOAP over TCP).
• Simon Horrell's MSDN article on WSE and messaging.
• For a full-blown WS-Eventing implementation, check out John Bristowe and Co's Plumbwork Orange project on GotDotNet.

Simon Guest lists his top 10 tips for web services interoperability between .NET and IBM WebSphere and BEA WebLogic [via Christian Weyer who comments that things will points will go away with .NET 2.0, see his series 'The web service empire strikes back' for more details]

The command pattern allows you to wrap a command inside an object so that you can pass it to another object to execute that object.  We were using a simplified version of the pattern where we had a command interface that has an Execute method and a Results property to retrieve the results:

public interface ICommand

{

      void Execute();

      string Result{ get; }

}

Objects of this type are then passed in as an IList to a CommandExecutor that loops through the commands and calls Execute on each one:

public class CommandExecutor

{

      public void Execute(IList commands)

      {

            // loop through the commands and

            // call the command.Execute method

      }

}

So we wanted to write a test that ensured that the Execute method was called on each command that we provided to the CommandExecutor.  Initially we thought about sending through a concrete Command (say a ConcatenationCommand) and determining that the result was what we expected.  However, this would have coupled our design to that particular Command - if we made a change to the results that returned we would have to update our tests.  In this case we aren't concerned about the actual result, just that Execute is called on each command.  The question was how can we test this behaviour without having to worry about a concrete Command or its result?

This is where Mock objects come in (see the original Mock paper or the Pragmatic Programmer's overview).  They allow you to test the behaviour of an object rather than just its outcome.  The idea is to create a 'fake' or stub object that validates the object was used in the way we expected.  In our case we could create a MockCommand that derives from ICommand.  We could provide a boolean property - ExecuteWasCalled - on the class that could be set inside the execute command.

public class MockCommand : ICommand

{

      public bool ExecuteWasCalled;

      public void Execute()

      { ExecuteWasCalled = true; }

      public string Result { get { return "result"; }}

}

While this would work fine, since our needs are pretty simple in this case we could use NMock which provides a way of creating a DynamicMock object based on an Interface.  The DynamicMock exposes a property that can set expectations about the behaviour of an object, such as a particular method being called.  The DyanmicMock object also has a MockInstance method that returns an instance of the Interface. Using this technique saves us having to create a concrete Mock command.  Here's the resulting code:

While I still have reservations about the extent to which MockObjects are useful, I think that NMock and its DynamicMocks are a useful technique to quickly produce loosely coupled test code.

If you'd like to understand more about how to do messaging in WSE and would like to see the  three levels of messaging within WSE in action then why not register for my first MSDN webcast here?.  I'm going to be presenting it on Mon 9 Aug at 1PM EST (GMT -8). 

I'll be covering some of the demos from the Keith Ballinger presentation from TechEd San Diego that I presented in Amsterdam, along with some new material.  Here's the abstract:

Using messaging systems to support application functionality allows technical solutions to better match business problems.  WSE 2.0 provides messaging implementations that range from low-level explicit messaging through to high-level, more transparent models.  This webcast demonstrates various messaging levels within WSE including a custom WS-Eventing sample and how message, network addresses, intermediaries and queues need to become first-class citizens of your application. Learn how messaging can enable you to create powerful web services applications that cross machine and network boundaries.

I've put these details on my updated presentations page as well.

Dave Bettin writes about his experiences doing a user group presentation on Indigo.  His main disappointment was that few attendees seemed to know about Indigo or the existing ASMX web services in .NET.  This got me thinking about how to get the message out about distributed apps, service-orientation and Indigo.  Is it the case that most developers need to be educated about the benefits of scalable, distributed, service-oriented apps or is it that they don't need these approaches to solve their business problems?

Dave's feedback is similar to Clemens' experience earlier in the year, where he spoke of the difficulties selling the relevance of Indigo today before he settled on explaining the 'why' over the how.  Both Dave and Clemens' experienced the same difficulty making the connection between Indigo and developers' jobs today.  Clemens' says that one of the barriers is that people think Indigo is only going to be relevant for 'Big Apps', when in fact it will be important to anyone who builds apps that expose functionality to other apps. 

One argument is that in order to get more interest in Indigo we need to educating developers about the benefits of services and distributed apps and get them to stop writing monolithic apps and start separating the logic across tiers and services.  We need to teach them more about the benefits of the services and distributed application approach, such as scalability.  Sam Gentile touched on this sentiment when he wrote that most .NET developers don't grok distributed computing. 

The other position regarding evangelising Indigo is mentioned by Alex Lowe in a comment on Clemens' post, is that:

Of course, we know that in the real world there are lots of applications that don't expose functionality to other applications. There are plenty of line of business applications for small and medium businesses that don't require ASMX, Remoting, and in the future Indigo.

Alex also has a summary post on his position where he says that the number of applications that need (and by extension, developers) that actually need to know about how to architect scalable applications is low.  He argues that bloggers are a distorted sample because they represent the group that need to build scalable distributed applications and they are generally well up on things.

I'm interested what other people think.  Is the problem that most developers don't understand the benefits of service-oriented distributed applications or is it that most developers are writing apps that wouldn't benefit from service-oriented distributed applications?  What business application scenarios are most likely to benefit from adopting service-orientation and Indigo?

private ISoapInputChannel channel = null;
private void Form1_Load(object sender, System.EventArgs e)
{
   // Get our channel that's listening for incoming messages
   channel = SoapTransport.StaticGetInputChannel(
      new Uri("soap.tcp://localhost:8088/admin"),
      SoapChannelCapabilities.ActivelyListening);
}

private void button1_Click(object sender, System.EventArgs e)
{
   // When we're ready, process the messages off the in-memory queue
   SoapEnvelope message = channel.Receive();
   String messageBody =
      message.GetBodyObject(typeof(String)) as String;
   this.listBox1.Items.Add( messageBody );
}

In the Form_Load event the channel is retrieved from the SoapTransport.StaticGetInputChannel method, based on the channel listening on a particular network address (in this case using tcp) with particular capabilities (the options are ActivitelyListening, meaning the channel should create a listener at that address, or None, meaning the channel should connect to an existing listener).  As messages are received they are placed in an in-memory queue managed by this channel.  When the Button_Click event is fired the next message can be retrieved from the channel by calling the Receive() method which returns a SoapEnvelope from which the body can be unpacked.

I hosted a lively session on what Service Orientated Architecture really means at TechEd Amsterdam.  While it was a Birds of a Feather session, I decided to run it as a park bench format in order to take advantage of having David Chappell, Michele Leroux Bustamante and John Hooper (blogless MCS UK Architect) come to the session.  Here were some of the interesting discussion points that came up:

• There was some agreement that SOA is a pragmatic marketing term that unites many existing architectural principles around SOAP. 
• The closest agreement about a definition for SOA was that it was based around common architectural principles of encapsulation, loose coupling and messaging.
• There was some discussion about whether asynchronous messaging was a necessary part of service orientation. My feeling is that since you can achieve synchronous patterns over asynchronous communications that having asynchronous messaging capabilities is extremely useful.
• Although it's possible that these principles could be applied without SOAP, it's the fact that Microsoft, IBM, BEA and others have agreed that SOAP will be the lowest common denominator that is the pragmatic reason behind the current push for SOA.
• David made a point that service orientation will be whatever Indigo supports when it ships.  Shipping software always wins.  I think there's a lot of merit in this argument, but to the extent that SOA is based on generic architectural principles it is worth considering using these principles in systems that are design today (as Clemens demonstrated all conference).  If SOA principles help solve your business problems today then it's definitely worth starting today rather than waiting for Indigo.
• Some delegates were suspicious that using Indigo would enable them to interoperate with other systems that didn't use Indigo.  There was some confusion about the idea that Indigo will provide an object model that can be used to develop a system that has the capability to send messages between the systems that are based on interoperable WS-* specifications.
• The four tenets of service orientation are necessary but not sufficient for a system to be considered a service oriented architecture.  Some people thought they were too technologically focussed because they were tied too closely to XML technologies.
• David mentioned that the best SOA installation he'd seen was using CORBA several years ago - it had support for finding services, common schemas etc.  Michele backed up the need for shared industry schemas based on some of her experienced.
• John Hooper was interested in Pat Helland's assertion that services should not share transactions, which led into the difference between WS-Transactions (classic two-phase commit) and WS-BusinessActivity (compensating transactions).
• A delegate who was learning about the topic came up to the chairs and spoke about what he'd learnt at TechEd.  This was great to hear and generated a lot of discussion.  Clemens' presentations clearly had some impact with many in the audience.

Here's me with David Chappell after the session.  I've been a fan of his since reading Understanding ActiveX and OLE when I started Microsoft programming.

Clemens' session on his ProseWare application at TechEd Amsterdam last week was one of the best conference sessions I've seen.  Proseware is "an industrial-strength, robust, service-oriented example application that newtelligence has designed and implemented for Microsoft".  The application clearly demonstrates how to go about building services today with currently shipping technology, reinforcing that there's no need to wait for Indigo to start building service oriented apps!  I'm hoping that we see a public release of these bits soon on MSDN.

Points that grabbed me:

• Guidance on where to use messaging patterns: Use the OneWay pattern where there is no intelligent immediate reply or no reply is needed, use Request Response pattern where a message asks a question that can be answered immediately (in under a second) and use the Duplex pattern when a message asks a question that can be answered later as the service has capacity (e.g. anything that takes over a second).
• Clemens showed how to achieving 'near enough' reliable web services with HTTP and services that use MSMQ transactional queues behind the service interface.  If there were any problems placing the message onto the queue then an exception would be returned inside a SOAP fault response.  He optimised this further by using a void response type on the web service method, even though it was not marked as OneWay, so that if there were no problems placing the message on the queue then the web service response message would be small.
• ProseWare is based around a repository of XML schema files which he uses to dynamically generated the 'message' classes in the application using pre-build steps.
• All of the service projects shared these schemas rather than having any project references linking to the binaries (services are autonomous).
• The pre-build step inserts ISerializable attributes onto the message classes so that they can work with Remoting.
• These message classes are used as the only input parameter to all of the public web service methods.  These classes leverage XML Schema's support for allowing any element or any attributes to come through, which is a powerful way of allowing for future extension to the message.  Dare goes through this in one of his previous posts.  The XML Serialization attributes look similar to this:

[System.Xml.Serialization.XmlAnyElementAttribute()]
public System.Xml.XmlElement[] Any;

[System.Xml.Serialization.XmlAnyAttributeAttribute()]
public System.Xml.XmlAttribute[] AnyAttr;

• Clemens showed how to use the properties in COM+ 1.5 in XP/2003 to set the home directory for an application, meaning it is possible to use a .NET config file to store the config. He's blogged about this previously here and here.
• He created his own object pool to create something similar to the ADO connection pooling but for COM+ objects.  He simply pops a component out of the pool and pushes it back when he's done, avoiding the excessive overhead when calling new in a COM+ environment (which has to create the pipeline connections).  He has also blogged about this JIT activation pooling here and here.
• He mentioned how LRPC, which is used under the covers in ES, is the fastest way to go cross-process on a single machine.  In order to get this benefit you need to use the JIT activation pooling in order to avoid having the performance gains wiped out by the cost of creating ServicedComponents (again, something Clemens' previously blogged).
• The nice part was that Clemens had done the hard yards and built an application installer that handled created the SQL Server and Windows user and group accounts.  He even showed where he'd found bugs in the OS and how he'd had to work around them.  Information like this is priceless (well, worth a lot of contracting dollars) if you're ever working in a situation where you need to achieve these outcomes.

Should we write test code against interfaces or something more abstract than the implementation?
James mentioned that MBUnit is a tool that allows you to test against an interface.  The question was whether you should create interfaces that enable tests to be written against them in case further implementations were created in future.  James' attitude was that this might result in wasted work ('you aint gonna need it') since you may not need it, or may not need it now.  Instead, abstract things out when you need them - don't create an interface just to test it.

James also said that an interface is not a good example of the contract of what is being done - it is the name of the method with input and outputs, but does not reflect how the method reacts to the input. James writes tests that show the real interaction between someone that calls the code and what it produces.

How much of the class should we test? Public methods only or protected and private methods as well?
Around a third of the group thought you should test protected and private methods, about another third thought that we should only test public methods.

The arguments for testing internal and protected methods:

• To ensure that the internals work. One delegate mentioned writing a test before each internal method. Someone else said the start with the public method then refactor and hide the method by turning it private.
• Another argument was An argument is that if you write a granular class with lots of private methods, then the tests should be just as granular as the thing being tested.
• The internals are the most important as they contain the biggest areas of logic, so they should be tested directly.

The argument for testing public methods:

• Private method tests inhibit refactoring - you have to refactor your tests with a change, increasing the burden and making it less likely that the tests are updaed.
• You may need to test all of the scenarios from the real world against the public method. If your class is written well then the private methods should be tested.
• If you separate the tests from the production code then you must test the public methods.
• Using public methods gives you a clear limit to the number of unit tests that you need. Public tests will get most of the issues, there's no payback from more investment by testing private and internals.

James says that there is no winning this argument. He favors testing the public interface because it decouples the test from the implementation which will discourage refactoring. However, there are cases where it doesn't make sense to expose something just to test it. He thinks it is 80/20 or 90/10 favouring testing through the public interface.

Whidbey will have friend assemblies that allow developer's to split two assemblies, this other assembly is akin to being inside the same assembly. You can do this today with a multi-module assembly built through the command line, not through Visual Studio.

How do we introduce Unit Tests to a 'legacy system' without tests?
No one in the session had seen legacy code that is easy to test if the testing hasn't been thought of upfront.  One person mentioned that they created tests for each issue logged through the help desk and then used these as a regression suite.  This also demonstrated the value of unit testing to their organisation.

James suggested drawing a line around a piece of the code that you need to change. Test it's external behaviour, make changes and ensure the tests still run. They aren't unit tests, but who cares? Create a boundary, create tests and then change. The key point was to conserve effort and only write tests on things that you are going to change.

James also mentioned a book to be published in September by Michael Feathers called 'Working Effectively With Legacy Code' that describes how to handle this difficult situation.

How to convince people to do test first? Argue against the concept that it will take too long to write the tests first?
One delegate mentioned that this is hard because the best way to convince people is to show results, which requires a practical example, which means you have to know how to do it (what classes, how to unit testing). It takes time and experience but it will work eventually.

Someone else mentioned the green bar of success (my favourite) on the screen is a big part of demonstrating it.

You have to make sure the person has the right mindset. Not everyone has a zero-defect mindset. They want to write tests more than write code.

James says the story is not about the individual developer who wrote the code and knows it works. It is about the team and the ongoing evolution and maintenance is where the tests matter. How can you be confident of your result without them? With unit tests I know they works, before I relied on something else, but now I know.

Why should we write the tests first?
James says that no one will take my word for it. What convinces people is having to test something that hasn't been created with testing as a priority. If a setup method has 20 lines of code and many objects being created and only contains a single assertion - it wasn't written with testing in mind. Someone will say it is really hard to test it.  So the next question is 'What would you do differently?'  The answer to this question is what people should do in Test First.

James said that he believes that testing has to be seen as a primary part of development. We have to incorporate test inside development. There may be QA, but they start at a different level. You have to incorporate testing as part of development.

In the PAG group they have 2,000 unit tests that get run against the library every time someone modifies the code. When James talked to the testing and QA group who do integration testing -they didn't know what to do - the unit tests did the easy part of the testing. They have to start at a higher, more complex level to start thinking about critiquing what is going on rather than I just pass 32,000 characters in a web app and it breaks. These are necessary tests, but it doesn't take a lot of skill to do these types of tests.

Someone made the point that software development lags behind electronic engineering where hardware must be tested first.

James mentioned that studies have show that Test First is 16% longer, but the quality was much higher.

How do you test a function that is dependant on other objects or libraries?
One delegate mentioned that using Dynamic Mock objects relied on finding a sweet spot.  They are useful with objects that have relationships, but the problem is that if you refactor your code then it is highly likely that a whole slew of tests that will break.  The problem is compounded with dynamic mocks since you only see this at run time rather than compile. It works well at mocking the database, but not the more dynamic

James described mock objects as the situation where object A interacts with objects B and you need some way of 'switching out' Object B to create a 'dummy response' from the calls of Object A.

James believed that if you are interacting with something complex, building the simulation of something that is complex is a worthless activity - you spend more time writing the simulator than the test and it doesn't tell you anything? Just because my mocks work, does my real system function?  James uses a rule that if mock objects have an if statement inside them then that's too far - the behavior is too complicated and there's no value (it says there are multiple situations). They do have value, but we need to separate a simulator and a mock object. For example, writing a mock JDBC implementation - that is way too complicated.  Be skeptical of spending a lot of time spending time building MockObjects.

Someone asked - 'Don't you think that it depends on the situation?' James: "I could answer everything that way"

James mentioned the Inversion of Control pattern - this is the a situation where object A depends on object B - the developer wants to be able to switch object B out as a mock or a stub - how can we do that? Inversion of control says you create the dependent object B, or a mock or stub external to object A and then pass it into object A as a interface parameter in a constructor. That's a lot of complexity to add to the initialization,  James was interested in opinions about whether the complexity is worth it?  He mentioned that some people have pushed back on the idea because of the complexity of the construction - if all you are doing this for testability (it also decouples the design) then it is too much work?  He believes that what has to happen is that we need different language structures and we will never get there if we dismiss these ideas now. In the Java world there are lots of work being done on using containers in another way.

How do you we test the database? Where do you get your data for testing? How do you unit test things that involve the database?
James thought that the problem with the database is similar to the MockObjects. It is a lot of effort to Mock out the database. Sometime it is easier to use the database than mock, you need a database to run the unit tests and you have to fill the database, read in a dump, which takes longer and is more effort.

James' experience is that at some point you write tests against the database because he wants to test that the Data Access Layer works. When he write DAL tests he uses the database. But he don't use the DB for anything that uses the DAL - he mocks them out. A technique he use to ensure he's written good unit testing is to deliberately break something and see the results.  If he finds a cascade of errors, it means he hasn't isolated the tests correctly.  He also mentioned that if he comes back to an a feature afterwards and spend a lot of time in QA it means he didn't do well enough with the tests.

With databases,  it is a good idea to  use a transaction and rollback the transaction at the end of it to ensure that.  There was some discussion about how to construct database tests.  The problem with building the database in scripts the unit tests take too long and they wont be run. How long is too long?

How long should the tests take?
There were a range of experiences including:

• Four or five machines run it each product, it takes 6 hours across machines.
• Someone else has developer tests and then smoke and build tests, it can take a long time in the nightly build.
• Another project had two CruiseControl systems that built at different intervals, running a short and long set of tests.

James thinks an hour is too long - the reason is that ideally you would be able to get very quick work around. This is how it works successfully. Waiting an hour for the feedback means that a developer could only make 8 or 10 changes a day. In that hour a number of things are accumulated. James mentioned his experience on an early project where it took 12 hours to recompile that application.  In this project if someone made a change to a header file in C++  they had to recompile, so what they'd do is create global static variables instead of the header to ensure it works before 'doing it correctly' (which was never done). Sorting out problems was very complex because it was effectively an integration nightmare..

Should all developers run all tests? It depends on the architecture - if there are subsystems you could do that. In XP it says all of the tests all of the time. James' group run 2000 tests in 340 seconds against 3 databases.

How do you manage dependence tests with the database? How to restore state with persistent storage.
It is a good idea to stub out the system so that you can test against something that you depend upon. Then when the implementation is delivered you can run the tests and it becomes clear where there is an integration problem.

One delegate spoke of how he stubs out the system, then write tests and implementation, then the stubs are removed and the real implementation of the rest of the tests are done. The platform should support this - James mentioned a project where he used 'linker polymorphism' - relied on the linker to substitute.  Another delegate said they had written something that as part of the the daily build checks out the project file, it changes the XML structure of the C# project to switch references and changes them to the actual assemblies.

How do you test concurrency? Multithreading?
These are just really hard. Testing timing problems with unit testing are hard. One delegate spoke about how he had written unit tests for a multithreaded apps and said it was one of the best examples to convince others of the value of unit tests. It took 2 months to create, but then it only took 1 day to shift it from Java to C#.

The xUnit tools should support this but don't make it any easier.

Another delegate used a timer library and extended the Nunit assertion.

Roadmap for MS
VS Team System will include unit testing support some time next year. It will have a number of integrated testing tools. The keynote yesterday showed a tool 'very much like' and 'subtly' different tool to Nunit. James will be talking about the difference.

There are load testing tools, web testing tools in there. It is a team system that is extensible by many different partners. Many partners, such as CompuWare (did a functional UI testing), doing stuff inside this 'platform' that can be extended.

Someone will write something that will allow Nunit to execute in this environment.

How do you generate tests?
Team system will create a stub test from the code. These are just boiler plate - it doesn't do analysis of the running code and propose a 'good test'

One delegate had a poor experience with a tool that created tests automatically because the tests didn't take into account the intention of the class which resulted in the tests failing. Writing the test manually is more useful as these can act as a specification for the class.

James thinks looking at the implementation to drive the tests is looking at it the wrong way. If you wrote the implementation wrong and then create tests off these, what's the value? The tests should say 'this is what the implementation should do'

People often propose this if they haven't done unit testing before.

How do you test distributed applications?
It relies on subsystems - just write tests that test the local machine, when you have to integrate it all together you might need to run tests, but these are not unit tests in this situation.

How do you unit test GUIs?
One delegate used Rational Robot - costs a fortune. If you are careful and script it then you can get away for a few builds without it breaking. SendKeys was also used.  Another person just avoided the problem by trying to get all of the functionality out, recognising that it is hard to test the UI.  Someone said that it is hard to test drag-and-drop operations.

James menioned that Robot is not good to drive development because they require the UI to be done, but it is hard to do.  NUnit Forms and NUnit ASP were also mentioned.

What frameworks can you recommend?
Nunit, csUnit, mbUnit, CLRUnit
HarnessIt (commercial)
Xunity (Commercial)

How do you do Web unit testing?
HttpUnit works, but it is Java, these can be used successfully to drive HTTP requests and do some testing on the HTML that is returned.
They suffer from many of the brittleness problems that the robot testing does - you have to use id's on the elements - but it requires a lot of thinking about how you output.

David Chappell presented a barn-storming presentation based on the idea that the future is services, that services will be called by business processes and that we need to look for a platform that will manage business processes.  He argues, convincingly that we can't expect App Servers to perform this role.  The answer comes with Business Process Platforms.  Here he positions BizTalk 2004 as the answer and goes so far as to claim that it will be the major product at future TechEd's and that getting close to business processes (through BizTalk) could be a key part of keeping your job as a developer since business processes are much harder to outsource than simple services.  You can also read David's whitepaper on BizTalk 2004.

Application Evolution
There are four waves of applications: Mainframe, client/server, multi-teir and now service oriented.  They share the idea that there is a database at the back end, but the key with services, it that they are designed with the idea they can be consumed by other applications, not just humans.  The reason why we can have services today is that we have web services  - an agreement on soap and other specs - amongst all of the vendors.  It's a huge change that we are at the start of - 4th generation.

Soap is like TCP for applications.  It took years between the start of TCP and its ubiquity: the same will happen with SOAP and web services.  It may be five years away.

To think of SOA as just about soap is folly - the reality is that going forward we will see some apps exposing their services via SOAP, but we will also see other diverse ways.  Not all apps will be SOAP.

Who calls services?
SOA talks a lot about how to build about how to define and build services and miss the point of 'who is going to call these services?'

David proposes three groups: UI (portal, asp, jsp, win forms etc), other services (we will have composite services) and business process (some central business process platform that will manage relationships between services).

The need for a Business Process Platform
Where should we build these business processes?  Is it in an app server such as J2EE container?  No, an app server all by itself is not the right place to build service-oriented service.

We are seeing a new kind of platform designed to support business processes.  In other architecture shifts we've seen new platforms - mainframe to client services produced RAD tools, the shift to tiered apps produced App Servers, and now with services we need to support business processes that drives services.

Requirements of a Business Process Platform
What do we need from a business process platform? Something that manages communications with other applications, business process implementation, scalability, modifiable business rules, process monitoring, tools for working with trading partners, cross-app authenticaiton, human interaction with business processes.

Rules change much faster than processes change - so separate out the rules from the processes.

Business Processes: Your job may depend on them
Business processes are more immune from outsourcing than the services.  So as developers we have to start caring more about business processes.  In five years time we'll need to be closer to the business or move to Bangalore.

BizTalk: A Business Process Platform
David mentioned that he had previously avoided BizTalk because he thought integration was messy, boring and 'on the side'.  However, he thinks it will move to the centre of application development.  If you believe in the move to service orientation then you have to believe that business processes that drive those processes are fundamental, therefore BizTalk will be the centre of it all.  It is about to go mainstream as the service-oriented world becomes a reality.

It's not about B2B and EAI.  These are just subsets of the larger space of business processes. 

BizTalk Engine
This is built fundamentally built around the concept of a message.  It doesn't mean only asynch, you can use RPC, but what is processed here is messages.  The incoming message comes in and is processed by a receive adapter - software that knows how to talk to a service or application (there are lots).  Here is the difference between AppServers.  AppServers just support SOAP, but not the diversity of communication technologies.

The message is processed by a receive pipeline.  It does many things, including converting it to XML. The message then comes into a MessageBox (a SQL Server database) that other engine parts subscribed to (e.g. show me all messages from this organisation).  An orchestration (the BizTalk term for a business process) retrieves the message.  It may publish a response to the MessageBox, then back through a send pipeline and a send adaptor.

BizTalk Adapters
Microsoft provides adaptors for MQSeries, SOAP and SAP.  You can make your own or buy them.

Tools
Platforms need tools.  You can build your own adaptors in the Microsoft.BizTalk.Adapter.Framework namespace (notice, a namespace, demonstrating it is a .NET application).  There's also a pipeline designer, biztalk editor (used to create XSD schemas) and a mapper (mappings and xslt transformations between schemas).

Some customers are simply happy with the mapper.

Process implementation with BizTalk Orchestration
Orchestrations compile into .NET assemblies.  It has simple shapes like if-then-else statements, loop, send, receive and parallel actions.  The process logic is simple and doesn't require a high-powered language (e.g. you don't need C# or VB.NET).  Using a graphical language is a decision that Microsoft and many others have done.

Another advantage of a graphical language is that you can use it to communicate with people that understand the business domain.

This visual language can also be authored in Viso for use by Business Analysts.  This is something that will grow over time.

Orchestrations are another reason for preferring Business Process Platforms over AppServers (which implement processes through lower level code rather than graphical representations).

State
Business processes involve people, so state may need to be maintained for a very long period of time.  So we can't use in-memory.  The Business Process platform needs to manage it.  BizTalk does this by storing state automatically and reloading it if needed.

Scope
Business processes involve transactions over long periods of time.  So BizTalk avoids locks (services shouldn't let others take locks on their data), it uses long running transactions that use compensation.  Biztalk uses scopes to manage transactions that can be atomic or long-running.

Correlation
If I send two purchase orders to a service (ERP app), how can I get the correct invoice response from the service?  You can't use request response synchronous calls because the real world doesn't work that way.  So you put in a GUID in the message, but how would this work if you can't alter the response from the service?  You could match on particular fields.  This is what BizTalk does - you define fields that should be used to match responses. 

Scalability Support
BizTalk host instances enable request to be automatically load balanced across orchestrations and MessageBoxes.

Modifiable Business Rules with the Business Rules Engine
Rules change more rapidly than processes - it makes sense to separate them.  If you are building a business process in BizTalk then you can bake process and rules together in an orchestration.  However, if your process has volatile rules, you can build the process as an orchestration and put the rules in a set of rules defined by the rules engine.  This is worth doing so that you can change and redeploy the rules easier than deploying the orchestration.

BizTalk provides a business rule composer that allows business rules to be expressed in a more natural way.  You define terms (sort of like creating an object model) and then the business process rules (like script glues object models into an app).

Process Monitoring with Health and Activity Tracking
There are two levels of monitoring: technical  and business level.  The Health and Activity monitoring tool (HAT) shows the technical side.  The business level if Business Activity Monitoring (BAM) that shows real-time information about running processes (as separate from Business Intelligence data, which is not real time). BAM is based on views on a tracking application.  Business people can use Excel to do this, developers have interfaces they can use.

The Goal: Business Process Management
There's no agreement yet on what a Business Process platform should have, but we are getting a picture.  It has to communicate with other apps, scale out, support business process implementations, workflow with human beings, modifiable business rules and process monitoring.  BizTalk 2004 is Microsoft's implementation of this.

Why is SOA more well known that BPM (Business Process Management)?  Names are confusing in this case, but there's a change coming to the way we build software.  It implies more than SOAP, it requires service for building business processes.  Biztalk is a founation for building, managing and monitoring business process, in the world today and the service oriented world to come.

Pat Helland's just finished a great presentation on services (a highly polished of the version he gave at the PDC and in other locations available online).  The highlight was him singing 'Mr CIO Guy' - a 'speculative retrospective' based on what the future could look like if the Harvard Business Review's article stating that there was no more competitive advantage in having IT were true - all to the tune of American Pie by Don McLean.  Don Box was on bass and David Chappell was on piano.   It received a standing ovation from the audience (a first for TechEd?).

The video will be available on www.pathelland.com in a couple of days.

They keynote started with 6,000 attendees finding african drums on their seats.  To paraphrase Apocalypse Now, "I love the smell of animal hide in the morning".  A group of drummers from South Africa warmed the audience up with some massage (not successful) and drumming (much more successful).  It was more exercise than some delegates in a long time.  The sound of 6000 drums around the room was an excellent start.  They were used instead of clapping to respond to cool announcements.

Accessibility

An interesting demo followed by a blind computer user, showing how special software drives an external Braille device.  Combined with screen reading software it allows him to use the computer.  Interesting demo that showed how frustrating it is to navigate web pages (even those sites designed with accessibility in mind) for blind users.

What's coming in future

The main keynote showed what was coming in the Yukon timeframe of 2006 including Windows 2003 R2, Visual Studio 2005 and SQL Server 2005.  The Longhorn timeframe was shown as 2008 (or in somewhat difficult to understand maths, "in the next 2 to 3 years").  These include Longhorn Client and Server, Office 12, and Visual Studio Orcas.

The other messages was that the focus is now on platforms, frameworks, interop and EAI tools.  So we're no longer about being a specific language developer, or just a system admin.  Now it's a more holistic view based around the 'lifecycle' (bingo!)

Visual Studio Team System Demo

A quick demo of the code coverage, unit testing and static analysis tools along with the 'whitehorse' designers.  The presenter mentioned that the Team System enables you to develop SOA (bingo!) applications.  This confused me as SOA is mostly about the design of external interfaces rather than the deployment of the application within a trust boundary.

Visual Studio Express Edition

See my previous post.

Deployment

Dynamic Systems initiative.  This allows the system to be modelled, check it against the virtual description of the environment and validate the application.  This will be part of MOM 2005, System Centre 2005 (the new SMS that integrates them all).  In Longhorn there will be one integrated management tools.

Virtual Server 2005.  Host multiple servers on the one box.  This will allow parallel test environment alongside production.  Over time there will be true virtual Windows support from Microsoft.

Voice over IP

The CommNet here has Dell machines with phone handsets that can be used to make free phone calls anywhere in the world.  It was also shown how this can intergrate with outlook to record calls and make notes that were associated with the call.  I love this kind of integration.

Mobile Development

Two guys came dressed as full-size Windows Smartphone and Pocket PC devices (what is it with these guys and costumes?) and they created a smartphone application in Visual Studio that could post a photo to their Windows Moble blog.  They showed how to use the Windows Mobile Platform and Visual Studio to build an app that posts a blog entry from a mobile phone by calling a web service.  They then uploaded the app to a central website so that users can purchase and download it via a mobile.  It downloaded and installed on the handset.  They took a photo, the handset added the location information automatically.  You can see the post here.

Jim Gray

Jim Gray spoke about Skyserver.sdss.org The goal is to get the data from one telescope online.  There is 5 years of data from scanning the sky, with the full map to be completed in 2007 (It's 10 billion records and over 2 TB in size).  It can be used by teachers and students to learn about astronomy and computational data mining.  It uses a web service (displayed in a web page).  Jim showed how these images can be inverted and can call out significant features of the image.  You can see this in action here.   It also allows for custom SQL Select statements to query the data.

They are trying to federate these archives and put it into a query.  It's skyquery that publishes a schema web service and a data query web service.  They all talk to the portal, the portal determines which of the 15 centres should answer it.  Does a query plan across repositories.  Demoed a 'transcontinental query' across Baltimore, Cambridge.  Determines the plan in the first call, executes it in the second. http://www.skyquery.net/

Jim also talked about how the answers needed to be stored in partial form.  They allow people to create databases on the portal server that can store answers queries that take a couple of days.

A second project was about CERN.  It is building an enormous accelarator in 2007.  It produces a Gig of data a second (this is the result of screening out the Terrabyte per second of original data!).    They are looking at using 64-bit processors and 10GB internet.  http://ultralight.caltech.edu.edu/lsr-winhec/  They are doing a CD per second (7.1 Gbps) at this stage with Windows 2003 64 bit version.  Disk to disk is up to 450MBps.

The big announcement from TechEd Europe is that there will be 'nominally' priced Express Editions of Visual Studio for hobbyists, students and enthusiasts, including the replacement for SQL MSDE, available in beta from the end of this week. 

Microsoft have realised that they need to make Visual Studio more available to those who want to learn to write application (perhaps not professionals).  All of the major languages will have Express Editions with Visual Studio (C#, VB.NET, C++, J#) as well as SQL Server 2005.

They will be available at a nominal low price that will not be a barrier to getting into programming windows.  As Joel said last week - Microsoft will basically give the development tools for free.

SQL 2005 Express is the replacement for MSDE.  This will be free for download from the end of this week (some more audience drumming drumming).

The focus is learning how to program, evaluate .NET, interact with students and build cool apps.

They all come with a starter kit that contains samples.  The VB one allows you to catalogue the DVD collection.  It uses Amazon's web service to pull down the artwork and details of a DVD (nice!).  The edition shows all of the standard controls.  It also ships with code snippets.  Edit and continue is also enabled (big drumming!)

The MSDN content is now going to search MSDN, MSDN online and the codewise community sites.

Strangely the demo showed the presenter copying and pasting (cargo cultist anyone?) including all sorts of declare statements to enable an image to work as the desktop background.

The intellisense will now offer to fix problems (e.g. missing trailing ')') and offer to add them for you.  Finally.  The demo also showed VB's My feature to enable quicker access to functionality.

Darrell Norton writes a review of Debugging the Development Process.  This is my all time favourite book on what the Program Manager role at Microsoft involves.  I gave it to a friend in Sydney a few years ago who was so impressed that he took a day off work to read it cover-to-cover.

It was one of three "essential reading " early MS Press books that also included:

• Code Complete by Steve McConnell (Edition 2.0 is out now).  Sam Gentile raves about this book.  I love it too.  In fact, I bought this book before I started developing professionally because I was so interested in the material.
• Dynamics of Software Development by Jim McCarthy.  The most interesting psychological take on development I've read (with great concepts like "Don't go dark" and "Don't flip the bozo bit").  David Gristwood from MS UK is presenting on concepts from this book at TechEd Amsterdam and has a reprint of Jim's article '21 Rules of Thumb - How Microsoft Develops its software'.

Before blogs these books were the only way to get close and find out more about how Microsoft develops software.

After writing about how WSE 2.0 can use policy and config files to secure web services with no lines of code, I was thinking about how 'magic' it seemed and had an aha! moment when I realised that this demonstrated the power of the Pipes and Filters pattern or aspect-oriented style approaches.  I believe that these approaches will play an important role in service-oriented applications in future.  Here are some recent quotes that back this up.

In my post about using Policy with WSE to create secure web services with no lines of code, I mentioned that this seemed like a good practical demonstration of an aspect-like approach, similar to the Pipes and Filters patterns from Gregor's book.  The policy filters are hooked into the incoming and outgoing messages pipelines are able to ensure that messages to and from the service conform to a particular policy, including retrieving tokens and signing and encrypting.  This means that the security of the service can be configured outside the service code, making for cleaner implementations.

Harry Pierson mentions that WS-Policy is aspect-like in his interview on TheServerSide.NET:

We do a lot of thinking around aspects in Web services, we just don’t call them aspects Web services, we call the Policy. All of the work around Policy is very aspect-like. If you’re spinning up a Web services with Web services enhancements, the new 2.0 stuff, there’s a Policy that defines, okay, if you want to talk to me, you have to be encrypted and digitally signed ... now I can communicate that to whomever is sending me a message. ... the WS-I engine can actually now say, okay, I’m expecting messages that [are signed and ecnrypted]  so I can actually enforce to make sure that those things have actually occurred before it ever reaches the business logic. So, that’s very aspect-like, and that’s going to be very critical going forward around service oriented architecture

Ted Neward writes about a recent presentation on Shadowfax and mentions it uses the concept of a pipeline of interceptors. He mentions that Shadowfax uses this approach to deliver functionality such as tracing, authorization, duplicate detection, instrumentation, authentication, authorizations and transactions.  Chris Garty started an interesting thread about this on the Shadowfax message board, where it was revealed that the Shadowfax team spent some time Gregor Kiczales (one of the creators of AspectJ).

This same pipeline/interception approach has been used in WSE, ASP.NET, Remoting and COM+.  Indigo will implement the same kind of approach using Channels and ChannelProviders.  I'm going to keep reading around this area to understand more about this approach and where it how it can be used successfully.

Celebrating the launch of the Web Service Enhancements (WSE) 2.0 at Tech·Ed 2004, Benjamin Mitchell and John Bristowe talk about the advanced Web services specifications that it supports, focusing on WS-Security.

I'll be on .NET Rocks! tonight talking about WSE 2.0 and building web services with Microsoft technology today.  I'll be joined by fellow Regional Director and Commonwealth Citizen, John Bristowe.  If you can't listen live, it should be available streaming on Monday.

Update: As Carl has mentioned, the streaming/download version of the show has been posted.

my previous post, Mark Naughton asked an excellent question about how he'd apply WSE 2.0 security to a particular scenario.  The answer highlights how to determine which SecurityToken to use in your environment, how to encrypt a UsernameToken with an X509 certificates with code and policy as well as handling authorization with X509 certificates and determining how to distinguish tokens received by a service.

Martin's scenario

An End User uses a Web-based UI Application (ASP.NET 1.1).
The Web Application talks to a Web Service (ASMX) for data storage and other processing.
The Web Service needs to identify the End User and the "direct" calling application (The Web UI App), since there may be more than one "direct" calling application.
We also want to sign and encrypt the Soap messages in both directions.

Since we're talking about adding security to a web service, we'll need WSE 2.0 installed on all of the calling applications. 

What's the best SecurityToken to use?
The first thing to consider is what SecurityTokens are applicable to the scenario.  Aside from custom xml or binary tokens, the three options that WSE supports out of the box are as follows.

Username Name and Password

• For - easy to construct, no distribution problems, WSE handles automatic Windows authentication and the construction of the SecurityToken.Principal for authorization.
• Against - on their own they are only as secure as the passwords.

X509 Certificate

• For - cryptographically strong, can carry extra information (e.g. certificate name). 
• Against - need to manage the distribution to clients.

Kerberos Ticket

• For - powerful cryptographically, inbuilt WSE support for authentication and creating the SecurityToken.Principal for authorization.
• Against - your application and the service you access must be running on computers joined to a Kerberos realm (e.g. not good over the Internet) unless you implement a custom security token service to issue service tickets.

So any of these tokens can identify the end user or the application - it's a matter of working out which one works best for your situation.  If you can handle the distribution and installation of X509 certificates to all of the calling applications, I'd suggest using them to sign and encrypt the message.  In your scenario, the ASP.NET web server could create a UsernameToken to represent the End User of your web application.  For best security, I'd suggest encrypting the UsernameToken with the X509 certificate (hiding the username and password/password digest).

The code would look something like this:

// ... Assume we have an X509SecurityToken and a UsernameToken, and a reference to the web service called proxy.

// Add both tokens to the SOAP envelope

proxy.RequestSoapContext.Security.Tokens.Add( x509token );

proxy.RequestSoapContext.Security.Tokens.Add( usernameToken );

/* Encrypt the username token with the X509 token.

 When encrypting, WSE looks for XML elements with an Id attribute from the http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd namespace, which the username token uses.

The "#" indicates the Id with this value is local to this message */

proxy.RequestSoapContext.Security.Elements.Add(new EncryptedData( x509token, "#" + usernameToken.Id ));

// Encrypt the message body with the X509 token to ensure no one can read it.

proxy.RequestSoapContext.Security.Elements.Add(new EncryptedData( x509token ));

// Sign the message with the X509 token to ensure its integrity

proxy.RequestSoapContext.Security.Elements.Add(new MessageSignature( x509token ));

You'll also need to decide on what token type to use when sending the signed and encrypted response.  Again, I'd recommend using an X509 certificate for the most cryptographically strong security.  The downside is that you'll need to install the certificate on each of the clients.  If you can't handle this install requirement then you are stuck with UsernameTokens.

Distinguishing the tokens on the service
Using the combination of a UsernameToken and an X509SecurityToken to represent the identity of the end user of the calling application and the identity of the calling application itself makes it easy for the web service to work out which token is which.  The web service has to search through the tokens in the RequestSoapContext.Security.Tokens collection to locate each token.  If you decided to use two username tokens for example, you would need to distinguish them somehow.  For username tokens you could achieve this through the username values, or perhaps by giving them well-know identifiers in their securityToken.Id property.  For X509 certificates you could use the certificate name.

Performing authorization
If you use a UsernameToken encrypted with the X509SecurityToken, and you don't want to send the password as plain text, then you'll need to create your own UsernameTokenManager.  This is responsible for authenticating the user and creating the SecurityToken.Principal object which can be used for authorization.  For the X509SecurityToken you can create a custom X509SecurityTokenManager and in the AuthenticateToken method, after calling the base class's implementation, then create your own generic principal and attach this to the SecurityToken.Principal (Ingo Rammer wrote an excellent article on this last September for MSDN, but it's disappeared.  You can find my notes on it here).  The benefit of doing this is, rather than just testing for the certificate name inside the service code, is that WSE Policy validation input filters inspect the SecurityToken.Principal when verifying the policy assertion.

Wrapping it all up with policy
As I've indicated in previous posts, using policy and configuration to avoid writing security code within your service is an excellent idea.   You could do that in this situation as well, except that signing the username token is a little tricky to indicate in policy (you'd have to hand-craft the policy XML file as this is a little way-off the WSE Security Settings Tool territory).   In a confidentiality assertion you'd need to modify the MessageParts elements in the policy file to indicate the UsernameToken's Id attribute.  I'll leave this as an exercise to the reader (as my daughter is about to wake up again), but Aaron Skonnard shows how you can use XPath 1.0 to achieve this in his excellent article, WS-Policy and WSE 2.0 Assertion Handlers.

Mehran Nikoo mentions the new Thames Valley User Group - looks interesting:

If you live/work near Thames Valley Park then this is for you.

The kick-off meeting is at Microsoft Campus on June 21st. Scott Guthrie (co-founder of ASP.net) and Mike Ormond (Microsoft DPE team) are among the speakers.

My TechEd conference-buddy John Bristowe has a blow-by-blow account of my CTS302 Securing Web Services with WSE 2.0 session at Teched.  Michael Earls has some notes and a couple of photos from the repeat session (which was a little fast because it turned out to be 45 minutes rather than an hour).  Aaron Skonnard mentions my first session in his TechEd trip report on his new PluralSight blog:

Benjamin Mitchell's session on Web services security using WSE was excellent. His was the clearest presentation I've seen on general security concepts along with concrete code examples.

That's going straight to the pool room. 

After covering so many other peoples' talks it feels strange to read coverage of my own talk.

In the last post I showed how it takes only 1 line of code to ensure that a web service client signs all messages with a UsernameToken by creating a send-side policy with the WSE 2.0 Security Settings Tool.  In this post I show the same feat can be achieved with an X509Token without writing a single line of code.  I also show how this functionality powers WSE's support for automatic secure conversation without having to write any code, something that blew me away the first time I saw it.

X509Tokens can be located through Policy and Config
In the last post I covered how the PolicyEnforcementOutputFilter checks the send-side policy when processing output messages through the Pipeline and attempts to find a matching token to fulfil the policy.  In the case of UsernameTokens, this means searching the SoapContext.Security.Tokens collection or looking in the PolicyEnforcementTokenCache (hence the one line of code).  However, with X509Tokens it is possible for WSE to locate the certificate without a single line of code.  The Security Settings Tool allows you to configure which X509 certificate you would like to use and stores an identifier for this key in the policy file.  This information is combined with the the <x509> element in the Microsoft.Web.Services2 config section handler that specifies which certificate store to find the token in.  So the combination of the policy file and the config file gives WSE enough information to find the correct X509 certificate without writing any security-related code within the service.

Policy saves code on the receive-side as well
Policy files can be used to save writing code on the receive-side as well.  On the receive-side the PolicyValidationInputFilter is used to validate that the incoming message meets the assertions defined in the policy file.  The policy file can perform checks such as whether the message is signed and/or encrypted with a specific token type or token as well as whether particular message parts have been signed.  If an incoming message does not satisfy these assertions then a security fault exception is raised before your service code is even executed.  As with send-side policy, the WSE 2.0 Security Settings Tool can help you author this policy, saving you from paying the XML angle bracket tax. 

The samples provided with WSE 2.0 have examples of solutions that rely on code and the same solutions using policy.  Comparing these solutions side-by-side highlights the many benefits of using policy instead of code to perform receive-side validation.  The first is that it keeps your service code much cleaner.  Second, it saves you having to remember to make the same calls at the start of each service.  Third, you can change your security configuration without having to recompile the code.

Putting it all together: automatic secure conversation
The best example I've seen of the power of no-code security through policy and configuration files is the support in WSE 2.0 for automatic secure conversation.  WSE supports the WS-SecureConversation specification that defines a SecurityContextToken that is a fast, light-weight security token that can provide message-level secure communication across multiple calls between a client and a service.  It's fast because it is based on a shared symmetric key, rather than an asymmetric key (which is over 1,000 times slower to process).  WS-SecureConversation builds upon WS-Trust which defines the notion of a Security Token Service that receives RequestSecurityToken messages and returns the issued SecurityContextToken as part of a RequestSecurityTokenResponse message.  WS-SecureConversation uses these mechanisms to request and retrieve the SecurityContextToken.  While all of this may sound a little complicated, it is possible to achieve all of this in WSE using the Security Settings Tool.  Using the ideas presented above, if you use X509Tokens then all of this can be achieved without writing any code.  This is the first demo I showed in my TechEd presentation.

Here's my take on how it performs this magic under the covers (feel free to chime in any time Hervey).  On the send-side, the PolicyEnforcementOutputFilter loads the policy file which specifies that all sent messages must be signed and encrypted with a SecurityContextToken.  I think that WSE makes an assumption that the web service can act as a SecurityTokenService and issue SecurityContextTokens (This is enabled on the service by adding the automaticSecureConversation element to the config file).  So when a SecurityContextToken assertion is found in the policy file WSE loads the SecurityContextTokenManager class and calls the LoadTokenFromSecurityTokenAssertion() method.  This method retrieves the tokens that will be used to sign the request before calling the RequestTokenFromIssuer() method that sends the RequestSecurityToken message and unpacks the SecurityContextToken from the RequestSecurityTokenResponse message sent back from the token issuer (which is often the same location as the service).  The PolicyEnforcmentOutputFilter then uses this SecurityContextToken to sign and encrypt the outgoing messages.

Phew, that certainly was a lot of digging with Reflector.  But it illustrates how powerful policy can be: you can request tokens from a token issuer and use them to sign and encrypt messages without writing a single line of code.  This blew me away the first time I saw it working (I didn't believe it until I saw the wire-level traces).  I pinged John Bristowe and Christian Weyer asking 'how does this work?  It seems like Magic but I know it can't be'.  When I thought about it more I realised that this was a demonstration of the power of the concepts such as aspect oriented programming or the Pipes and Filters pattern from Gregor Hohpe's Enterprise Integration Patterns.  More on this in a future post.

Making more of a good thing: custom policy assertions
As well as using the built-in WS-SecurityPolicy features that WSE enables with its Security Settings Tool, it is also possible to create your own custom policy assertions as John Bristowe has demonstrated.  Aaron Skonnard also has more about custom policy assertions.  WSE has great extensibility hooks that let you write code that uses your own policy assertions, allowing you to write validation code in one location that can be hooked into your service through the config file without having to reference it in your code.

While playing with the WSE Security Settings Wizard I discovered that the generated policy requires a DerivedKeyToken to be used to sign the messages rather than the original security tokens.  This is a good thing, but isn't obvious from the wizard screens.  I thought I'd provide some background on what derived keys are, why they are useful and how to ensure your WSE services use them through code or policy. 

Derived Keys: what are they and why are they useful?
Using a derived key is a good thing as it means a different key is used to sign and/or encrypt each message.  Changing the key each time makes it more difficult to perform a ciphertext-only attack.  The DerivedKey can use many different algorithms to generate the key.  WSE 2.0 uses the algorithm defined in the WS-SecureConversation specification, which creates the derived key by performing a SHA1 hash over the original key (in the case below, a reference to the key), along with the combination of a label and a nonce value (a unique value that's seen only once).  Here's what a DerivedKeyToken based upon a UsernameToken looks like on the wire:

<wssc:DerivedKeyToken wsu:Id="SecurityToken-d06a92f7-990c-43a4-a996-f8f3a359e450" wssc:Algorithm="http://schemas.xmlsoap.org/ws/2004/04/security/sc/dk/p_sha1" xmlns:wssc="http://schemas.xmlsoap.org/ws/2004/04/sc">
   <wsse:SecurityTokenReference>
   <wsse:Reference URI="#SecurityToken-6c059c7a-c748-44b1-b957-4b927dd2a8f3" ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#UsernameToken" />
   </wsse:SecurityTokenReference>
   <wssc:Generation>0</wssc:Generation>
   <wssc:Length>16</wssc:Length>
   <wssc:Label>WS-SecureConversation</wssc:Label>
   <wsse:Nonce>mDRVPaA7343zsrMjD+CatA==</wsse:Nonce>
</wssc:DerivedKeyToken>

See Martin Gudgin's MSDN article on Using WS-Trust and WS-SecureConversation for further information.

Ensuring compliance with the WSE generated policy
To comply with the server-side policy that the WSE Security Settings Wizard generates you can either create a derived token in your code or use a send-side policy.  The advantage of using the send-side policy is that it requires fewer lines of code and provides the potential to change the security configuration in future without having to recompile the code.

Signing a message using a DerivedKeyToken with code
Creating a DerivedKeyToken requires two more lines of code than would be required to sign a message with a standard user name token.  The first extra line creates the DerivedKeyToken and the second line adds it to the RequestSoapContext.Security.Tokens collection so that it will appear on the wire when WSE sends the message.  Here's the code:

// Create the username token and add it to the message headers
UsernameToken usernameToken = new UsernameToken("TechedClient", "TechEd2004!", PasswordOption.SendPlainText );
proxy.RequestSoapContext.Security.Tokens.Add( usernameToken );

// Create a derived token, based on our username token
DerivedKeyToken derivedToken = new DerivedKeyToken( usernameToken );

// Add the derived key token to the message headers
proxy.RequestSoapContext.Security.Tokens.Add( derivedToken );

// Sign the message with the derived key token.
proxy.RequestSoapContext.Security.Elements.Add( new MessageSignature( derivedToken ) );

Signing a message using a DerivedKeyToken with send-side Policy
The simplest way to ensure that a message is sent using a DerivedKeyToken based on a UsernameToken is to use the WSE Security Setting Tool to create a policy file on the client specifying that output messages should be signed with a UsernameToken.  When WSE sends the message through its pipeline the PolicyEnforcementOutputFilter will create a DerivedToken based on the UsernameToken and sign the message with it.  For this to work, WSE needs to know where to find the UsernameToken to base the DerivedKeyToken on.  WSE first searches for a matching token in the SoapContext.Security.Tokens collection and if it doesn't find any there it looks in the PolicyEnforcementSecurityTokenCache. 

So you can either use the first two lines of the sample code above, or the following:

// Create the username token and add it to the message headers
UsernameToken usernameToken = new UsernameToken("TechedClient", "TechEd2004!", PasswordOption.SendPlainText );
PolicyEnforcementSecurityTokenCache.GlobalCache.Add( usernameToken );

The advantage of using the PolicyEnforcementSecurityTokenCache is that the username token can more easily be attached to any message leaving the client, regardless of which service is being used (e.g. it is independent of the webservice the message is being sent to, unlike the SoapContext which is related to a specific webservice address).  The second advantage is that if you later decide to sign with X509SecurityTokens rather than UsernameTokens then you wouldn't need to change any code or recompile (the UsernameToken in the GlobalCache would simply not be used).  I'll write more about this topic in a future post.

Jim Newkirk got his 'day in the sun' to speak about Test Driven Development and the tool out in public promoting Test Driven Development and the tools support he's been involved with using Microsoft Visual Studio Team System.

He started out with a quick audience poll of how many people had heard of Test Driven Development (around 80%) and how many were actually using it (about 30%). So a clear victory for marketing over behaviour change there!

The Two Tenets of Test Driven Development:

• Never write a single line of code unless you have a failing unit test.  The goal is to take requirements and express them as test
• Eliminate duplication

How to do TDD
Jim starts by blocking out 4 - 8 hour sessions of development. He spends 15 - 20 minutes at the start of each session thinking about what he is going to do and brainstorming a list of unit tests.

A key part is not to get hung up on completeness, you can always add more later. The purpose of the tests is to describe completion requirements.

The flow of a TDD session: Red, Green, Refactor
The process is:

• Start by writing a test for a new capability
• Compile
• Fix any compile errors
• Run the test and see it fail
• Write the code to make the test pass
• Refactor as needed (clean up any duplication)

The purpose is about how to use the functionality, not how to implement it! The process allows you to build confidence through having a set of tests that pass.

The most successful way to do test is to do it before the development. If you start it first then you need to think about how to test.

Features in Visual Studio Team Systems
Jim used a stack example to demonstrate the process of TDD as well as the support in Visual Stuido Team systems. The first test looked as follows:

[TestClass]
Public class StackFixture
{
   [TestMethod]
   Public void IsEmpty()
   {
      Stack stack = new Stack();
      Assert.IsTrue(stack.IsEmpty);
   }
}

So, the same approach as NUnit, just with new names!

One cool feature was writing a class name followed by a method name that didn't exist yet. After compiling, Jim used a 'smart tag' to choose to create the method stub inside the target class. It wrote this stub and had a 'NotImplementedException' inside it. This is functionality similar to Eclipse and is good to see.

When developing with WSE it is often useful to be able to see what is going out on the wire.  Changes in the WSE 2.0 release mean that is no longer possible to use tracing tools such as tcpTrace and MSSoapT.  Christop Schittko has a good post on the background to this problem and shows how to use the inbuilt-WSE trace capabilities to get around it.  There's also another solution, which is to use Mindreef SOAPScope's WebProxy, and as I was writing this I noticed that Mike Taulty has also posted his WSE 2.0 trace tool, which has become my new default favourite.

Background
The previous approach to tracing in WSE was to listen on a one port with a tracing tool, then forward the request onto another port.  WSE 2.0 now checks to make sure that the WS-Addressing To header in the SOAP message matches the address in the HTTP Header.  This means the existing tracing tools will return a fault.

The SOAPScope solution
SOAPScope has implemented it's own version of System.Net.WebProxy that allows proxying to localhost.  It also means that the address in the SOAP envelope matches the address in the HTTP header.  To use it you just need to add the following to the client's config file:

<system.net>
   <defaultProxy>
      <proxy proxyaddress="http://benjaminm:5049" bypassonlocal="false"/>
      <module type="Mindreef.Net.WebProxyEx, MrTools, Version=3.0.0.0, Culture=neutral, PublicKeyToken=90f6595dbbe888f3" />
   </defaultProxy>
</system.net>

Mike Taulty's solution
Mike has written a new WSE 2.0 Trace Tool that uses custom input and output filters into the WSE Pipeline. These filters copy the message into a new SOAP envelop and post it a tracing 'web' service listening on the a tcp port, using the SOAP Messaging (soap.tcp:\\) support within WSE 2.0.  Using it requires adding the following to the config file:

<microsoft.web.services2>
  <filters>
    <output>
      <add type="WSETracingFilter.WSEOutputFilter,WSETracingFilter, Version=1.0.0.0, Culture=Neutral, PublicKeyToken=1c1f2f7177e1ff79" />
    </output>
    <input>
      <add type="WSETracingFilter.WSEInputFilter,WSETracingFilter, Version=1.0.0.0, Culture=Neutral, PublicKeyToken=1c1f2f7177e1ff79" />
    </input>
  </filters>
</microsoft.web.services2>

It's very nice! I'm going to be using it tomorrow in my talk at TechEd.

Clemens' talk was about managing state across multiple layers within a .NET application. His message was that there are many types or state and many approaches to dealing with it. It's not just about the ASP.NET session object! He covered a definitions of services, state and its types as well as how to manage state including transactions.

Statelessness doesn't really exist
Stateless doesn't really exist. Everything is stateful when it runs. Just because a component doesn't remember anything across calls doesn't mean there isn't a state penalty. Keeping information on the stack is a way of maintaining state.

Definition of services
A service is autonomous -lives and can be deployed by itself.

A service has its own store. It might be as system with 20 modules having 20 databases.

A service is not XML and SOAP. This is just one way of talking to services.

Services shouldn't share databases
One of the gems I picked up from the talk was that we shouldn't necessarily tightly couple everything at the database layer by putting it all in one place. Sometimes this is done for speed, but the benefit may disappear if you put it in a cluster.

Martin Fowler posted about this today:

The recent rise of Service Oriented Architecture seems to mean very different things to different people, but one plausible thread is a rise of autonomous applications with their own ApplicationDatabase that communicate through service interfaces - effectively replacing shared database integration with rpc or messaging based integration. I'm very sympathetic to this view, particularly favoring integration through messaging - which is why I encouraged the development of EIP. In this view of the world the integration database is no longer the default assumption.

What is state?
All the data an application needs to remember. It can be:

• volatile (the stack manages volatile state)
• transient (a stock ticker)
• permanent.

Transient state may contain useful data
Clemens mentioned that transient data may contain useful data that is worth storing. An example is the contents of a shopping cart at an online store. Keeping this data can provide useful information about the behaviour of people on a site (how many don't complete an order?).

Don Box and Doug Purdy did a 'keynote' for the Connected Systems Track.  They started out by asking what questions the audience wanted to see.  A great set of questions were proposed and the answers contained some of the most valuable content in the session.  Here are my notes on their answers, and some they didn't get time to do.

How does WSE 2.0 fit in with the Indigo direction?
It lets you use the protocols we have today.  WSE takes your ASMX investment and keeps you in the game as we do this protocol work.  If you don't track the protocols it may not be so important. 

Indigo will be the primary technology for using the WS-* specifications in future.  WSE takes your ASMX investment and lets you add support for those specifications today.

What's the future of DIME?
MTOM.  DIME was an experiment - we were on the wrong track that didn't support security.  Microsoft got together and did PASWA that became MTOM.  It will be in Indigo and other MS technologies.

WS-Security vs. SAML?
There are many different kinds of tokens that may be used, such as Username, X509 certificates and Kerberos tokens.  Don said it was unlikely that a token type, like SAML will become the 'single token format to rule them all'.  No definite answer on where the SAML support was.  As I learnt on Saturday, trying to implement SAML support is a non-trivial exercise - it would be nice if there was a clear statement from Microsoft on when it will be supported in the platform (so that you don't have to share my dll in order for us to use it when we talk).  I think it will be part of the identity management work in future.

How successful is WSE at interop?
Microsoft do bake-offs with WSE where they get all the vendors in a room and try and make the specifications work.  There wasn't a definite answer other than this.

How do you talk SOAP from a Windows Service?
Don's answer was that you do the hard work to host ASMX inside a service then put an ASMX façade and call into the service with ES or Remoting.

I thought this missed the point that the recently released WSE 2.0 supports Soap Messaging, which allows you to implement SOAP messaging over TCP.  I think this would be a far easier way of hosting SOAP within a windows services.

What is the technology to replace COM+ in the long term?
ES investment will keep working.  Deployment, interception and synchronization are being brought forward into Indigo.  Many of the ES semantics are a direct correlation with the Indigo model.  Doug mentioned that ES programmers will be the most prepared to work with Indigo when it ships.

Is there an issue with the verbosity of web services payload?
Don's answer was that Indigo will 'negotiate up' and switch to a faster way of communicating if the other endpoint uses Indigo.  How they do this is to be seen (there were comments at the PDC that the first Indigo call will be a policy request to see if the other end is an Indigo endpoint).  Don mentioned that the industry is having a hard time defining binary protocols that allow user definition are difficult.  Binary protocols that support user defined structures are hard.

How do we discover services and determine policy at run time?
Don mentioned that UDDI was a solution you could use today.  In a show of hands only 4 of 200 attendees were using UDDI (2% adoption?).  According to Don it makes some hapy, but some customers want more.  They want a more flexible model for describing things without having to use the tModel (which is hard to grok).  There also other groups want to discover services on devices, so WS-Discovery is where Microsoft are headed.  It is a small spec that is easy to understand that can be easily implemented.

Will we need to continue to be plumbers to do web services security?
I thought this was a great answer:

For a while.  WSE makes it easier, but if things go wrong you'll need a plumber.  We have not done our job in Indigo if people have to understand the protocols.  Your common developer needs to solve business problems, not the protocol problems.  Indigo is adding value without focussing on the protocols. 

No matter how good WSE does, since we are ironing out the interop you'll still need to read WS-Sec.  Now at least we only need plumbers when things go wrong.

What's the migration path to SO?
This was really the content of Richard Turner's talk in the track, he's also written a great post on detailing prescriptive guidance on preparing to upgrade to Indigo.  The basic message is don't do tricky things.  If you are doing something that was hard to figure out, maybe that was for a reason.  So things like SoapExtensions or custom message sinks in Remoting are not going to upgrade well.  There's lots of material out there that shows that Microsoft have a 'good story' on upgrading from various technologies.

Unanswered questions:

• What is the MSMQ equivalent for COM+?  Is MSMQ going away?
• How does EIF fit into SOA?
• Strategies for native to managed interop?
• Will Indigo support mailslots?
• Are there any application blocks for SOA?
• How do we handle events across appdomains?
• Security - you have authentication and authorization - what about any attacks through the channel - filtering content? Content-awareness in firewalls?

I'm with all of the 'Blue Shirts', speakers and the Microsoft staff, in the keynote overflow room, sharing the experience of watching Steve Ballmer on a video screen.  Here are some key points:

• He's looking trimmer. A gasp of 'Atkins!' went around the room.
• Key messages - do more with less.
• The next 10 years are going to be even greater than the last.
• Only Pfizer spends more than Microsoft on R&D.
• Remember 10 years ago TCP/IP was a separate business to the OS.
• Integration is the key. How many data access layers does Microsoft need.
• How can we narrow down the skillset required to know how to use the products. Integrate to reduce the overhead required to use the platform.
• Windows XP SP2 has taken priority over Longhorn recently.
• It used to be 'features, features, features' now it's 'listen, listen, listen'.
• Watson is one of the biggest advances in computing. Being able to send crash reports to Microsoft means there is a statistical way of rating the issues that users are having.
• Integrated innovation and customer responsiveness to do more with less.
• Security is key focus.
• Spam is too cheap to send - we need to add cost and burden. Using techniques like making the sender prove who they are.
• Interoperability has been a key focus. Microsoft has done more than most people have ever given them credit for in integration. Microsoft is absolutely behind the XML stack as an open standard. The 'best and most important thing to happen to our industry'. It's an 'architected' way of doing interoperability. The old way was writing XML to connect each system.
• Microsoft Office beta web services - allow Office to be a smart end client to web services.
• Becky Dias gets on stage.
• WSE 2.0 is released! Also the Microsoft Office Information Bridge are entering Beta.  Basically web services integrated with Microsoft Office task pane.
• She's clicked on someone's name in Outlook. A task pane has come up with a form that lets her do a stock trade, calling a webservice and gets an ID back again, all without leaving Outlook.
• Demoing policy. Not sure if the audience are getting this. But it's very cool. We don't have to right code anymore. Definitely should have spent more time polishing the WSE Settings Tool wizard screens.
• .NET has more than 50% of the US market. Customers think it is 66% more reliable, 70% think it is faster, 2.7x people think it is more secure.
• The VSIP program has been increased. Oracle and SAP and TibCo will use Visual Studio for their platforms.
• Visual Studio 'Team System' - now trying to do more as part of the software development life cycle. Group development, modelling, testing and deployment.
• It looks like we now get bug tracking within Visual Studio.
• Showing a Whitehorse style screen that diagrams the deployment of the application and can check to ensure that it will work in that environment.  It produces 'build errors' when you compile it.
• Can specify that the system passes build rules, static analysis and unit tests. 
• Finally we have unit tests that are part of the build system (this got a clap!).  Now we know what James Newkirk has been doing at Microsoft!  Rewriting NUnit!
• Also includes code coverage tools as well (another clap)
• There's also a security version of FxCop that is built into 'Visual Studio Team System' based on Microsoft Research's work on Secure Computing Initiative.
• But wait, there's more .... load testing as well (more claps!)
• Back to the Information worker.  Steve has the feeling that SharePoint Team Server, Portal Server, Office and Live Meeting haven't been as well adopted as they should have.
• There will be advances in searching as a result of 'strong competition' (see Google)
• Why choose Microsoft over Linux or Java? More integreated innovation, better responsiveness and trustworthiness, partnerships, choice (more applications, better interoperability).

Overall I was a little disappointed that 'Crazy Steve' didn't make an appearance.  There was no sweat, no ranting, no cheering with the crowd.

Scott's a very funny man and hosted a very entertaining session on code generation.  My jet lag really kicked in at the start of this session, so you might like to see Peter Provost's blog for more coverage as well as Jon Galloway:

• Jon Lam talked about the 'usability tax' from using XML. XSLT is a programming language that is hard to maintain. He prefers using PERL for writing code. XML is hard for humans to maintain.
• Scott started by talking about the CodeDOM as being 'the opposite of terse'.
• Discussion about the line between creating a generic engine versus just solving the problem with code. How do we deal with the trade-off between producing a solution to the current problem versus creating a generic non-specific approach.
• Computer languages are for people to work with, so we write in C#. It is a code generator that produces IL that produces assembly code to run. We want a higher level language to build software.
• 3 kinds of code-generation: Wizard skeleton, Compiler - template and generics, Modelling - result is not a model but an assembly
• Some discussion about whether generators were useful for producing quick and dirty one-off situations.
• Discussion about what the output of code generation should be. Is it the code files, or is it the compiled DLL?
• How do you manage changes - should you do it at compile time or run time? What about it you need to modify things after they have been generated? Should you make them plugins, use interfaces or work with the config files.
• Sometimes code-generation make it hard for others to maintain.
• Don't do the rules engine that solve the 'verbs' problem. Think about the nouns.

How Corillian do code generation:

• Scott talked about how Corillian do it. They model the nouns in a visual tool using an XML schema underneath that can be extended and allows you to create your own vocabulary. The elements on the schema come from another namespace that includes domain specific attributes.
• They then we use a free-ware tool called code-smith which lets you write code-generation syntax in an ASP.NET syntax (<% for each … %> to output the data). They use that to create the code (rather than HTML in the ASP example).
• They created an XML schema adapter that looks at an XML schema and gives you a collection of top-level types and subtypes. Then all the different places where they have domain-specific knowledge they use 'aspects' by placing that logic inside the setters and getters.
• The schema describes the contract between asp.net and the host and the asp.net and the front end like the device that it displays on.
• The adapter reads the schema and presents it in a friendlier way. The CodeSmith studio is an IDE for doing this. It has adapters that takes anything that presents a collection (e.g. a database), then for each table in tables - generate the code.
• The easiest way to jump into code generation it is to use the strongly-typed collection classes that comes from CodeSmith - accounts in ArrayLists should become an accounts object - it allows you to simulate the behaviour of generics now.
• Any time in the schema with max-occurs unbounded - we know it is an array and autogenerate a strongly-typed object it.
• The business people edit the XML in XML spy. Corillian separate domain objects from the message. They have a WSDL explorer. They use WSDL and a custom binding to generate the whole banking services. The proxy is generated from the WSDL - binds domain objects, messages and verbs.
• In future they are looking at using schematron that describe restrictions (e.g. saying something is an integer is not the same as saying it is a social security number). The intention is that the attribute on an element in the schema maps through an attribute in .NET.
• Scott's belief was that anything in the schema should be carried forward because the metadata should not be lost.

If you're at TechEd and would like to catch up with me I'm on MSN Instant Messenger at benjamin AT benjamin DOT net.   I'm especially interested in anyone who's doing work or got thoughts on web services, Indigo or extreme programming. 

I'm also on Scott Hansleman's Microsoft Regional Director Bingo card (available from booths 49-50 in the Pavillion) so come and say hello and I'll help you win a prize. 

Roy was concerned that this represented the 'cult of the worshipping masses' and that we 'should not be handing out autographs, but software, tips, tricks and code'.  Well, happilly I can report that the goal of the Bingo game is to encourage attendees to talk to the Regional Directors.  It's sending the signal that we're here to connect with attendees and share experiences and transfer knowledge.

Meeting the RDS at TechEd is just like a .NET rocks episode but live and in person.

Here are some of my tips on how to make the most of the week at TechEd.

Make a session plan.  Know your entry and exit points.  TechEd is sold out.  Not only that, it is overbooked.  Expect to be sitting in the aisle if you aren't clear about which sessions you are going to and how to get there.  Spend some time learning the floor plan on the first day so you can get between rooms without getting lost. 

Don't stare too long watching the PowerPoint slides.  Every attendee is going to get a DVD with the slides and audio after the show (it will likely be on the web as well), so don't cram your schedule too full with sessions.  Pick the key sessions to attend, you can watch the others later.

Connect with people about the technology.  Instead of going to the breakout sessions, make sure you spend time in the Cabana areas and the Community Lounge.  The Cabana areas are small presentation areas where you can 'heckle' (well, ask questions and interact) the presenter.  The Microsoft stand in the Exhibit Hall has   many key people from the product groups at the event.  They are here to meet you, answer your questions and help you understand the technology.  Take advantage of this chance to have one on one conversations.

Speak to the Presenters, Authors or Regional Directors you see.  Don't be afraid to approach these people if you see them.  They are at the event to answer your questions and find out about your experience.  Use them. 

Focus on questions you'd like to have answered and areas of knowledge you'd like to improve.  Aside from the above, I think there's also a RIO networking area where you can go and find experts who can answer your question.  There are lots of key people from most of the Microsoft teams at the events and they are here to talk with you.   Spend some one on one time with experts rather than just sitting in the audience. 

There was a panel discussion with John Bristowe, Scott Hanselman, Joe Homnick, Joe Lindsay, Terry Mohn, Ted Neward.  the highlight for me was that there still isn't good tool support, or a good story from Microsoft, on how to manage services once they have been deployed.  Other points:

• Scott mentioned that his company provide services to 30% of all online US Banks and 50% of them rely on web services. So they are out there and being used.
• An excellent point was made there isn't good tool support (or a good story from Microsoft) on how to manage services once they are deployed. The tools to ensure that services are up and running and providing guaranteed levels of services are not here yet.
• There was some discussion about whether we should care about angle brackets or not. Scott made the point that we should focus on Infoset. John Bristowe made the point that it can be useful to understand the specs and know what is happening on the wire.
• Ted mentioned that editing WSDL is too hard and no-one at the event other than Scott had actually done it. Scott made the point that most of us had edited HTML files because we did not like the way FrontPage formatted them, so why hadn't anyone done the same with their WSDL?
• Some good points from the more business-focussed members of the panel that they are more interested in tools making developers productive than in having developers that understand the plumbing in web services.
• Discussion about what the definition of SOA is. Answers included 'components with a longer wire', 'objects with explicit boundaries', 'hooking shit together'.  I had a useful discussion aftewards with Joe Homnick that helped me see that SOA is most useful as a concept when talking with businesses about how to architect projects.  As I've mentioned before I'm still not convinced that SOA is definable, but it is based on top of good architectural concepts such as encapsulation, data hiding, loose-coupling, service discovery, messaging patterns and asynchronous processing.  So if we need to call this SOA in order to get all excited about these topics then it may be a necessary evil.
• I followed with the question 'what problems is SOA trying to solve?' answers included high-level re-use and interoperability.
• Discussion about location transparency not being a good idea. Ted suggested the original idea was really about being able to write code without having to know the location of the components, rather than hiding the fact that this was an expensive cross-network call to a component.
• Discussion about the fact that Amazon uses Web Services well but that XML over HTTP is more popular than their SOAP implementation. There was some discussion about alternatives to SOAP such as REST, but they were seen as more limited compared to SOAP. The key for me is that tool vendors need to implement great tools for SOAP so that it is easy to use as these competing approaches.
• The more business focussed members of the panel made the point that the technology is not the main thing, it's supporting business functions. For them the point of the Interoperability work is that it les them go out and buy products that they know can talk together.

Ted Neward gave the keynote at our WS-Interoperability Day even today.  Here are some of the points he made:

• It's important to learn from the past, in terms of previous distributed computing approaches, when looking at Service Oriented architecture and web services.
• One of the key things that made HTTP a ubiquitous approach to solving interoperability problems was that it was simple.  For me this highlighted the fact that it is essential that good developer programming models be developed over the top of the WS-* standards so that it's simple for developers to take advantage of the functionality without necessarily having to be a plumber. 
• It is important that vendors continue to work together and not start developing their own non-standard features.  Ted mentioned that vendors are currently being driven to ensure compatibility in order to make money from the new approaches, but once a market it established they may start fracturing away by adding their own non-standard implementations.  Dare covered this in his post detailing the SQL standards.
• Ted made the point that Objects work better when they are deployed together because they need to talk together.  Good objects are well factored and make a lot of calls.  This means that the same techniques does not extend well to situations where objects are distributed.
• The fastest call that can be made to a network is around 1000 times slower than a local in-process call.  This can raise to 10,000 times slower when the network is involved.  To illustrate what this means, Ted used the analogy that this was similar to his 20 minute commute to work taking 70 days (his math).
• SOA is just a new way to get developers to listen to the worn-out message that  'Distributed Objects Don't Work'
• Interoperable systems work best when the architecture is built from the centre out.  If you start at the .NET and work towards Java for example it's likely that you'll find you've implemented something that is incompatible.
• The problem with starting from the inside out is that WSDL, which provides the definitions, is hard to work with and you really need a tool to author it.

I flew into San Diego last night where Michele Leroux Bustamante had agreed to meet me at the airport.  The plane was an hour and a half late and I was wondering if she'd still be there or even recognise me.  Turns out I needn't have worried she'd be in the group with the three laptops and the wireless router:

That's Anant Kadiyala, Michele and Heinrich Gantenbein getting in all the preparation time they could before our interop demo.

I'm presenting a TechEd session in the Connected Systems track on how to use Web Services Enhancements 2.0 to secure web services.  TechEd looks like being a mini-WSE festival, with Aaron Skonnard doing the pre-conference on Sunday, Keith "My blog has gone" Ballinger talking up a storm on messaging over multiple machines and networks, plus loads of other applied web services talks.

In my session I'm going to cover all of the security features in WSE using as many code demonstrations as I can fit it.  I'll cover the basics such as security tokens, signing and encrypting before moving on to more advanced topics such as token issuing and establishing secure conversations.  I'll show how WSE 2.0 allows all of this to be done with code as well as policy and configuration files.  The best part is that straight afterwards you'll be able to go and do some Hands On Labs authored by Aaron Skonnard covering these topics.  How's that for in-depth educational experience?

Here's the official description

CTS302 Connected Systems: Using Web Services Enhancements v2.0 (WSE) to Secure Web Services
Wednesday, May 26 2:00 PM- 3:15 PM, Room 10
Speaker(s): Benjamin Mitchell
Track(s): Connected Systems, Developer Tools and Technologies, Security
Web Services are being used to cross application boundaries, especially between enterprises. Such interactions need to be secure. See how to use WSE v2.0 and the security protocols that it implements to secure Web service interactions within and beyond the Trust domain

For those interested in doing some background reading before the event, I strongly recommend WS-Security Drilldown in Web Services Enhancements 2.0 by Don Smith.  Rebecca Dias thinks its the best article she's read on WS-Security and I'd agree.  Also Hervey Wilson's blog mentions some truly magical features that will whet your appetite for what you could see.

Martin Gudgin points to the Working Group Draft of the WS-I Basic Security Profile.  I believe that the web services security will play a major role in increasing the adoption of web services in future.  It's great to see the WS-I behind a push for greater interoperability in web services security.

Web services security enables message level security, through signing and encrypting parts of a message, so that secure communication can be established even across public networks such as the Internet.  Increasing support and compliance for this security infrastructure across platforms and development languages will lead to even greater adoption of web services.

I'm hoping that they develop test cases that can be used to prove compliance to the standard, similar to those already developed for the WS-I Basic Profile.  In preparations for this Saturday's Web Services Interoperability Day Michele Leroux Bustamante has had to assume the role of 'human interoperability tester'.

Joshua Block a senior staff engineer at Sun and author of Effective Java gave a great session on API design.  Joshua highlighted that good API can be great company assets because good APIs capture and retain customers.  The talk was particularly enjoyable because Joshua illustrated his points with reference to good and bad Java APIs.  You can read the full session write up by James Robertson.

I love the challenge of API design.  As Joshua mentioned, it's incredibly hard, and when working on large public projects like .NET or Java you only really get one shot at it.  Once it's out there it's extremely difficult to make changes to it.

This is where I think the Indigo team is on a great thing.  Steve Swartz describes his role as making sure that the Indigo programming model is easy to use.  Don Box mentions today that he sees his role as removing the need to have to absorb complex detail in order to create working programs.  This team knows how important it is to get the API correct.  As Don said in January "the APIs are the real lock in".

I was surprised that Joshua didn't say more about the need to test the APIs with a developer audience.  He did mention that when designing and interface for others to provide implementations for, it's worth using the 'rule of threes' and writing three implementations yourself before shipping the API to ensure that it's possible.  But he didn't really say anything about API usability testing, the use of personas or the need to get APIs out into the community for feedback.  These are all things that I believe Microsoft is doing incredibly well with Longhorn (I'm not really in a position to say regarding Java, but would be interested to know what goes on there).

This session was run by Alan Cameron Willis who works with the Visual Studio team out of Cambridge in the UK. Domain specific languages are meant to express requirements and solutions of a particular business domain. Alan is working on the tools like Whitehorse that are visual designers included in Visual Studio that help development teams design and build applications.

We worked in groups to come up with our own domain specific language that we could use. We were asked to imagine that we had created a Point of Sale system for a restaurant and wanted to sell that solution to different types of restaurants (burger bars, bistros, drive-thru's).  Could we come up with a language that could be used to help model, sell or create the software for these different settings?

Alan suggested a range of different communication from text, to different types of diagrams (e.g. UML), to animations. The key issue from my point of view was who the language needed to communicate with and for what purpose. Is the language for communicating requirements or is it to capture requirements and generate the code?

It's nice to see that Microsoft have highly academic involved in creating their software, and that guys like Alan are also turning this into useful tools such as Whitehorse that we can use as developers.

Alan has posted a summary of the domain specific languages discussion on the OT2004 wiki.

I attended the UK launch of BizTalk 2004 today.  Nearly everyone else there was in suit and tie, highlighting that BizTalk is a business focused product.  I like BizTalk and the 2004 product is a great improvement on previous versions (Darrel Norton has a great list of BizTalk resources).  Messaging and integration are two areas where I think IT can add a lot of business value.  BizTalk is making technology that had previously been very expensive available to greater number of customers and developers.  As Cameron Reilly notes, one CIO he knows says he could buy BizTalk 'out of the stationery budget'.

There were some interesting customer presentations including how Virgin Megastore were using BizTalk to spot employee theft by analyzing real time point of sale information (case study here).  Loss Prevention Agents (I'm thinking Agent Smith from The Matrix) are notified on their mobile devices as business rules are triggered. 

My favourite customer moment was the CTO Scottish and Southern Energy who revealed their unofficial mission statement was "boring but successful".  Brilliant.

Mehran was there with his assortment of Microsoft OS gadgets and even gave me a ride back to London in his very nice new BMW with Windows Automotive.   This is serious geek gadgetry.  There's 'Toy Boys' like Carl Franklin sings about and there's guys like Mehran who go out and buy the car.

While it is easy to see the benefits in the idea that services expose explicit boundaries, there is still some distance between this and understanding where service boundaries should be drawn and what it should look like.  A collection or recent posts from three Microsoft Architects highlight that services work best at the business level where they can share the same logical data model and single database engine.  It's nice to see the SOA discussion break through the academic debate to this practical guidance level.

Ramkumar Kothandaraman, a Solutions Architect on the .NET Enterprise Architecture Team, talks about  implementing the service-oriented tenet that services should have a clearly defined boundary when there are issues about searching or updating the shared data.  He proposes that it can be beneficial to share a database between related services and to expose these services as a collection or business services that share the same logical data model, rather than individual services. 

I believe this is similar to Pat Helland's session at the PDC where he spoke about business services encapsulating database-resident data and that each business services' data should live in a single database engine.  If the data can't fit into a single engine then look for ways to split it into two business services but don't have a service span database engines.  If the service needs to scale then consider partitioning the data between individual database servers rather than trying to merge and replicate.

Michael Platt, a UK-based Field Architect from Microsoft, shares the view that services need to be exposed at the business level.  He provides a useful, pragmatic definition of service oriented architecture as a model or abstraction of loosely coupled business-level functionalities or services.  I also enjoyed his reflections on a recent meeting discussing SOA:

There was a lot of discussion about what SOA was: explicit boundary’s, autonomous, etc which reminded me a lot of the COM definitions but with different words. Alas there was no discussion as to what a Service should look like, how big, what level of functionality etc; this was left as an exercise for the designer.

In terms of guidance on how to architect services, he mentions a story of the good old 'component oriented architecture' days where a client had 3000 components in a system, as a warning not to go overboard again.  Michael also invokes his 'First Law' as design guidance for a services which essentially says that that services should be coarse grained because of the huge communication and network stack they have to go through.

It can be difficult to talk to groups that are hostile about Microsoft technology because their beliefs are often based on sweeping generalizations or bigoted opinions.  Although Microsoft isn't perfect and doesn't always get things right it is surprisingly good at improving its products over releases and for engaging with the community to get feedback.

John Bristowe writes about his experiences attending a Linux User Group and talking to people about Microsoft and Windows:

What's funny is when you approach these people to challenge their stereotypes about Windows, they incite some form of moral/religion conviction against Microsoft and/or refuse to discuss the issue any further. I try to tell them, "Look, I'm all for a good debate. Let's discuss this." Nope. Nothing.

I had a similar experience to this on Tuesday night at the Extreme Tuesday Club.  I walked into a discussion about C# and .NET that went roughly like this

Attendee: "Microsoft standardised C# purely as a marketing move because Java isn't administered by a standards body.  Besides, the group that administers the standard is the easiest to manipulate standards body out there."

Me: "So you're saying that even though it's administered by an international standards body you think it may not be a '100% pure' standards body and for that reason you don't see any value in it?"

Attendee: "Well, Microsoft has never been interested in open standards!"

Me: "Well what about Microsoft's work in the WS-* space.  For WSE they have shifted from a pre-OASIS version of WS-Security to the OASIS version of WS-Security to ensure industry support.  They are also holding open workshops with any vendor who wants to attend to make sure that the standards are workable.  How does that work in the context of not being interested in open standards."

Attendee: "Hrmph.  I don't know anything about that."

As John mentions, there's a kind of snigger that goes through a group when they criticise .NET.

Fortunately I managed to have deeper conversations where I found out more about pain points that were creating a negative perception of .NET amongst a corporate group that was switching from Java to .NET.  I agree that some of these things are problems with the existing .NET technologies, but found it interesting that they were seen as such major issue.  What impresses me though is that some are being fixed or improved in Whidbey and that there are many Microsoft employees comment or discussing them in public.

• No refactoring in the current version of Visual Studio.  This is difficult for developers used to a Test Driven approach to coding and the awesome support provided in Java IDEs such as IntelliJ or Eclipse.  One guy said that the lack of this feature meant that ".NET code was unbelievably brittle: it was easy to write the code initially but very difficult to change it later".  Refactoring has been added to the Whidbey version of Visual Studio.  Jay Bazuzi is a developer on the Visual C# IDE team and has an excellent blog where he talks about his work implementing refactoring support for Visual Studio.
• The inability to set different access levels on property set and get statements.  This has been fixed in the Whidbey version of C#.
• Some interesting C# enum behaviour that Chris Stevenson mentions (here and here).  These were mentioned to me as evidence that ".NET is fundamentally flawed" even though the person I was speaking to didn't actually know the issues in detail.  I admit I didn't either, but I was impressed to see that Eric Gunnerson had replied to both of Chris' posts.

I'm not trying to say that .NET is perfect or that Microsoft always gets things right, just that discussions are more interesting when they are based on detailed information rather than sweeping generalisations.  I'm glad that the Java IDEs are currently better at supporting refactoring than Visual Studio.  I love the competition and the increasing number of good ideas that it creates.  I also like that Microsoft, eventually, take notice and improves its products.

I've been blog-lite while I prepared and then gave my Indigo presentation to the London .NET User Group.  The audience seemed to enjoy it and I had a good time presenting it.  Some quick points:

• The selling point of the Indigo Service model is that it is simple and easy to use.  The benefits for developers is that they get all the benefits of enterprise-quality features such as transactions, reliable messaging and security without having to write any code.  Letting Microsoft do the plumbing allows developers more time to write code that focuses on solving complex and interesting business problems.
• I took on board Clemens' experience and started talking about the 'why' of distributed technologies before delving into how Indigo works.  Luckily this group was very switched on.  I got some good questions about the performance of Indigo across AppDomains/Processes/Machines.
• Someone asked me to provide some background on why Enterprise Services are a better choice than Remoting for communicating between components on the local machine.  Clemens' has some more information on it here and here. 

Doing a presentation where you don't have the current version of the software to demo is a challenge (the PDC Indigo bits are from an older, M4 build, which has been substantially refactored in the now-being-coded M5 build, so there's not a lot of value in showing demos with the PDC bit).  In order to get some code samples I had to transcribe screen shots from the PDC videos (difficult since Steve Swartz didn't use word-wrap in Visual Studio).  I'm filled with (even more) respect for Don Box and Steve Swartz's presentation skills after realizing they managed to do four Indigo presentations at the PDC without even compiling, let alone running any of the applications.

Given the PowerPoint dependence and Clemens' reports on the complexity of the topic I decide I'd have to resort to audience bribery.  I drew on my experience in TheatreSports and brought several bags of Minties from the Australia Shop at Covent Garden (to my horror I discovered today that they are made in New Zealand!) and threw them to the audience whenever I detected the signs of PowerPoint-induced lethargy.

Pair programming with Jamie Cansdale at ThoughtWork's London Geek Night last Wednesday highlighted how much I like this practice.  Geek Night is basically an open night for developers who want to work on projects (ranging from open source development, to personal hobby projects or just for those who want to experience pair programming).  Jamie and I were working on creating a Visual Studio Add-In to remove unused 'using statements' in C# files (essentially, to replicate the 'Imports Tidy' functionality provided by C# Refactory that Scott Hanselman says is almost worth the price of the add-in alone).

Here were the things I enjoyed about Pair Programming:

• Knowledge sharing.  The knowledge transfer ranges from ways of using Visual Studio ("what did you press to get that?") to how to structure unit tests ("would you split that into a second method?") and features of the object model ("how can I find the assembly that defines this type?").  Even in a short session there's a great amount of knowledge that can be transferred.
• Continuous development speed. When sharing the 'driving' seat it means that it's possible to keep developing at a constant rate.  When I develop on my own sometimes I find it hard to keep going once I've achieved a certain milestone.  In Pair Programming it's possible to sit back for a while and let the other person drive.
• It makes you braver. Pair programming provides courage to start having a go even at complex problems.  It's easier to get started attacking a problem using two heads at the same time.

We started out by discussing how to implement the functionality.  Jamie was arguing for a brute-force but fool-proof mechanism of using a regular expression to search through the code, comment out any using statement that it finds and dynamically compile the code to see if it generates an error.  I was itching to write my own stripped-down C# parse that could scan through the source code and understand the using statements (including the alias keyword and nested statements).  To settle the stale-mate by switching our focus from the implementation to the functionality.  We decided to write the tests first (yes, XP is a simple set of practices but it's easy to get off track).

The beauty of writing the tests first was that the tests were independent of the implementation.  After writing a couple of tests it became clear that Jamie's approach was likely to be 'the simplest thing that worked' in the two hours that we had.  The best part was that we were able to achieve a working implementation inside this time!

I'm still eyeing off the SharpDevelop open-source C# editor, which as C# parsing code which I'm tempted to use (or maybe it's time to get into Mono), but for now I'm sold on the value of pair programming and test first development.

John Udell has written an article on WS Alphabet Soup based on his discussions with John Schewuck, an architect on the Indigo team.  Here are the interesting points:

• Indigo may ship before Longhorn to enable people to provide developers with access to the functionality provided by the newer WS-* specs.  I know the message from the PDC was Whidbey < Indigo <= Longhorn, but I guess haven't really considered it will come before Longhorn (I thought this was what WSE was for since Indigo is a 'baked in' part of the .NET Framework).  Not that I'd complain if it came earlier.
• There are a lot of Web Service (WS-*) specifications, but this is a good thing because they are composable.  You only need to implement the specifications you need.  The alternative would be a single specification that tries to do everything.
• The WS-* specs are inventing a lot of what has been provided before in DCOM and CORBA.  As Don Box says, 'The entire motivation for defining SOAP was to get the industry out of the "you need my runtime to talk to me" business that DCOM and Java RMI so happily got us into in the 1990's.'
• The advantage of using SOAP as a message description format  is that it provides extensible SOAP headers ("an accordion-like construct").  Previous network communications protocols was their dependence on fixed-offset dependencies that broke if you had to extend or modify the packets.
• Microsoft is wise to spend time worrying about the usability of its APIs.  This was my initial reaction at the PDC - 'Wow, they have done a lot to make this really easy for developers'.
• The key to reducing the cognitive overload caused by the specifications is an easy programming model (this acknowledges the reality that not everyone gets as excited as John Bristowe about a new specification).  Indigo's use of attributes in the Service Model is a major plus here.  How awesome is it to be able to write code like this to add WS-Security (signing, encryption), WS-ReliableMessaging and WS-AtomicTransaction support:

[Service]
[Security(Disabled=false, 
 SecurityProfileName="MyDynamicProfile",        
 ClientAuthentication=UsageMode.Required,
 IntegritySupport=UsageMode.Required,
 ConfidentialitySupport=UsageMode.Required)]
[ContextPropagation(typeof(ITransaction)]
[Connection(DeliveryAssurances.Full)]
public class MySecureReliableTransactedService
{
    [ServiceMethod]
    [Authorization(Role="Clerk")] 
    public void CreditAccount(string acc, decimal amount)
    {
        …
    }
}

The Loosely Coupled blog also has more commentary on the interview with John.

Matthew Adams left a comment on my last post disagree with my characterisation about services being more to do with messages as opposed to remote procedure calls (RPC).   It was a good question that I thought was worthy of a lengthier response.

What I was suggesting in the original post is that overtime there will continue to be a move towards services that use messages, and in particular SOAP messages based on the XML Infoset, rather than RPC style calls over time.  SOAP because it gives a broader reach for interoperability because of it's lowest common denominator approach and associated standards.  Messaging because it gives more flexibility in terms of what is sent and the message exchange patterns that are supported. 

The Great SOA Definition Debate
I've been trying to get away for getting into a debate about what services are, mainly because I haven't found a useful definition of service that says more than 'it's a great idea to have blocks of code that do something' and when designing these blocks of code, encapsulation and loose coupling are good practices.

By Matthew's definition a service is something that can do something, at a particular addressable location on a particular protocol, with a particular message format, in a such a way as the consumer doesn't have to care about how that something is done.

Using this definition of a 'service' it is true that the service doesn't have to be implemented as a 'message' end-point.  It is possible to implement these 'services' in ASMX web services, in-process .NET classes and remotable objects.  The problem is that these choices can impact the amount that the consumer has to know about the way the service is called, if not how it is implemented.

Don's Definition of Services
I like Don Box's definition of a service as a 'program that you communicate with using messages'.  These messages are SOAP messages based on the XML Infoset (a logical view of an XML document - angle brackets and text representations may or may not be involved).

The beauty of using SOAP messaging is that it is a lowest-common denominator of solving the problems involved with providing a service:

• How do we address the service, which protocol should we use to talk to it? (WS-Addressing)
• How do we describe the format of the messages that are sent and received? (SOAP)
• How do we describe the format of the contents of these messages (WSDL, XML Schema)
• How do we describe the pattern of the messages that are exchanged? (WSDL)
• How do we describe the other characteristics we need to successfully talk (WS-Policy)

What's wrong with RPC end points?
My main issue with RPC end points is interoperability and flexibily.  RPC-style calls to RPC-end points in Matthew's terminology calls can do all of these things, but they are more likely to do it in a less-interoperable way that requires greater knowledge of the underlying platform, protocol or programming model (for example you may need to agree on the binary representation of the parameters used in the procedure call which may mean sharing DLL's or worse, an entire development platform).

Of course any remote procedure call can be thought of as just a particular kind of message.  So it's possible to create services that are based around RPC-style calls made within a message (which is how ASMX web services work today).  Within web services there are two types of SOAP message format - RPC and document.  As Yasser Shohoud points out in his article RPC/Literal and Freedom of Choice, RPC formatted SOAP messages are just a subset of the document formatted (I'd argue message-based) SOAP messages .  As he says, ASMX web services show it's possible to use a document style message format but with an RPC programming model.

[Update: See Shawn Smith's post on this topic.  In the last section of this post I wasn't very clear about the distinction between SOAP message formats and message exchange patterns.  My point was that it is a better idea to implement more flexible approaches to both the message format and the message exchange patterns because the more flexible approach can accommodate the more rigid approach.  In this sense the SOAP Document format can accommodate the SOAP RPC message format and the asynchronous message exchange patterns can accommodate RPC message exchange patterns (synchronous request/reply).

Ultimately, it's not worth getting too fussed up about this as both the ASMX and Indigo programming model will support you whichever way you'd like to work.  You can write services based around asynchronous messages or using synchronous request/response (stop-and-wait) RPC-style calls.  As I said at the start of this post, over time I believe there will be a movement away from synchronous RPC calls towards asynchronous messaging.]

As preparation for my Indigo presentation at the London .NET User Group next Monday, I thought I'd write a couple of posts about Indigo, starting with 'The Road To Indigo'.

Indigo is a new technology that represents on the unification of existing distributed application technology.  It takes Remoting, Enterprise Services, Web Services and Microsoft Message Queue and combines them into a single area of platform-level technology.  In fact, Microsoft has literally taken all of the teams responsible for these technologies and put them in the same building.  Instead of thinking of Indigo as a completely new technology, think of it as the next version of all the existing approaches where all of the best features of each are combined into a single managed-code programming model.  Here's a diagram showing the progression of these technologies:

Indigo also lets you chose the way you want to build applications without tying you in.  Indigo supports and encourages building service-oriented applications using messages based on SOAP.  However, Indigo still supports you and provides features even if you want to create distributed applications using a distributed component approach using remote procedure calls.  Choosing one approach over the other doesn't mean giving up features such as message-level encryption or transactions.  As Ingo Rammer says, Indigo recognises that you shouldn't be locked into 'a specific programming model, a serialization model, a transport model, or even just a serialization format'.

Two pieces of news have made this Valentine's Day a geek day of love.

• JetBrains, the creators of IntelliJ have released an ealy build of their C# tool, ReSharper which I have been waiting for.   Luke Hutteman has a good review of the features (and info on how to access the download site, which currently isn't handling new registrations).
• Erik Sink announces the release of Vault 2.0 which includes a free Single User Edition.

Bruce Williams from the Indigo team has a nice walk through of the the XML elements in WS-Eventing.  When you strip away all of the namespaces and XML noise, it's a nice, understandable piece of work.

I like WS-Eventing because it's another tool in the architecture toolbox.  To me it suggests the end of polling in web service applications.  In some of the applications I've worked on today the client has had to poll the webservice to check for updates.  With eventing the client will be able to wait to be told if something has changed, rather than pestering the server like a kid in the back of a car on a long road trip asking 'are we there yet?' all of the time.   It's much more efficient to wait and be notified than constantly having to poll to check if something has changed.

I'm not 100% sure about how large corporates will handle security will work across the Internet and corporate firewalls.  For the client to receive the notification they have to have an open port that the notification service can send to.  Perhaps a solution is to use an approach like the MSDN SoapMail sample, where messages were returned to some central location, like a mailbox, which is polled by another service that picks the message up and forwards it to the client.  Perhaps the answer lies in the Security Considerations area of the spec which suggests the notification should be signed with security tokens.

When's ASP.NET Whidbey going to be available?
The aim is to ship a Beta in June and for RTM in Q1 2005.   There will be a 'go live license' that will let people put Beta 2 into production, later this year.  The Beta in June will be a public beta and it will be possible to download ASP.NET and Visual Studio Whidbey.  The build that Scott demonstrated with was from last Monday when he said they achieved 'Code Complete.  So they are now in that long stabilisation phase. 

Cooler applications in less time (even for enterprise applications!)
The key message is that ASP.NET allows you to create cooler applications in a fraction of the time it takes today. This sounds very 'marketese', but based on what I saw demoed it's very true. The hardest thing is that there have been plenty of demos like this before - from FrontPage and Visual InterDev. It used to be the case that you'd walk away saying 'good demo, but it will never work on a serious enterprise site'. But this time they've got the architecture right. Databinding to plain objects makes it possible to do proper 3-tier development. The Provider model means that there are interfaces at nearly every level that you can implement or override (apparently they'll ship the source code to the existing providers to make this easier).

It will be interesting to see how ASP.NET 2.0 works in real world situations.  Many of the features demonstrated like the login and authorisation and the paging and sorting of data are things that I've developed in the past so I know that they can literally take weeks to develop.  Now they are simple drop-and-drag and configure scenarios.  Also, since many of these ideas, like the data grid are now on their third or fourth iteration (I remember Visual InterDev 6.0 and it's 1.0 style implementation - urgh!) they have had time to develop to the point where they (look like) they are really going to work in the wild.

What a demo!
Out of the box Scott built a site with full security (login control, password reset and email page), customisable content using Web Parts, master-detail data pages (with paging and sorting enabled with simple checkboxes) using data binding in both 2 tier (direct to the db) or 3 tier scenarios (just plain public methods returning IEnumerable - no need for an interface or attributes!), easy internationalisation, support for navigation files (including treeviews and bread crumb trails) and Master Pages with Skins for easy site-wide look and feel manipulation. It truly does raise the bar about what will be expected as minimum website functionality in future.

The Full Notes
To save clogging up the aggregators and my front page, I've put my 3,500 word 'summary' of the slides and presentations here.

In summary I'd say that Scott is a great presenter.  He did a morning session on the basics, then the session I attended.  He spoke from 2pm to past 5pm and hung around until we were thrown out of the building to ensure that everyone's questions were answered.  It's obvious that he's pumped about the product and the platform that he's been instrumental in creating for developing web applications.

Well, he may not be blogging (yet) but there is an excellent interview with Pat Helland on TheServerSide.NET.  Pat proposes a solution to the debates about what SOA is and how it is defined.   Pat says we should focus instead on HST (Hooking Shit Together):

"Hooking Shit Together emphasizes the fact that I've got two things, they're separate, they're independent, and it's about: How the heck do I make them work together. That then takes me to messaging, that then takes me to Schema, that then takes me to the contracts and the flows between them while allowing them to be independent. I think there's far more in common in doing that inside an IT shop, and doing it across IT shops, than there are differences. So I actually like HST because it emphasizes what's in common about tying things together across machines, without regard to whether they're in the same company or not."

Pat will always have a special place in my heart because he taught me about Microsoft Transaction Server using an analogy about banana splits on an AVI movie that was part of a Visual Basic 5(?) multimedia (remember when that meant something?) CD-ROM.  The man's a genius.

Michael Platt, a .NET Architect with Microsoft UK, writes a great story about architectural thinking.  He describes being called by a distraught customer who claimed that their n-tier application was failing to scale past 50 users because 'Windows wasn't scalable'.  Without spoiling the story, the post highlights a couple of points for me:

• The need to avoid panicking and remain clear-headed when thinking about problems (e.g. don't invoke magic, witchcraft or the Windows platform as the primary reason for a problem)
• The evils of premature optimisation (though to be fair, this problem seemed to illustrate the evils of optimising without really understanding what's going on)
• The value of understanding software architecture.  Every now and then I devalue what I know about software architecture because it seems like such a well documented area - surely everyone knows about it?  Stories like this remind me of the value of knowledge and experience.

When I was clarifying the Microsoft Messaging Message I said I'd watch the PDC DAT 420 presentation on BizTalk and Indigo.  As my PDC DVDs arrived this week I didn't have to download the presentation I finally got around to watching it.  Here are the key points I took from the presentation:

• From a BizTalk perspective, Indigo is the vehicle that will provide secure, reliable transacted services (over more than just http).
• Indigo is an API, BizTalk is a set of tools.  Indigo will natively integrated into the next version of BizTalk after BizTalk 2004.  The demos showed a prototype Indigo adapter that worked with BizTalk 2004.
• BizTalk has been achieving reliable services through the BizTalk Framework 2.  In future this will be deprecated and replaced with WS-ReliableMessaging, which will be provided by Indigo.
• BizTalk will use the security model built into Indigo to enable WS-Security support for Username tokens and X509 certificates that can be easily configured through attributes, configuration files and policy.

The presentation helped me understand that Indigo is the plumbing.  A lot of business functionality will require the use of tools that take advantage of that plumbing, such as BizTalk.  So I was interested a few days later to see an announcement that the BizTalk Orchestration will ship as part of Indigo in Longhorn.  As the article points out, BizTalk seems to debut features that eventually become part of the plumbing. 

I really enjoyed the presentation, and not just because Scott Woodgate has such a pleasing accent to listen to (New Zealand, so similar to Australian with stranger vowels).   The BizTalk 2004 health and activity monitor combined with the Orchestration debugger were great features.  I can imagine that these would reduce a lot of the pain with administrating and developing complex service-oriented systems.  I'm hoping that I'll get to benefit from these tools on future projects.

Suresh Kumar has posted a newsgroup exchange from Don Box and Scott Woodgate that cover most of the points from the talk on how BizTalk will take advantage of the features provided by Indigo.  Suresh also has some other good posts on general BizTalk 2004 resources, BizTalk and Publish/Subscribe and XML Web Services in BizTalk 2004 including talk of a WSE adapter for BizTalk later this year.   If I could find his RSS feed I'd subscribe.

Feedster does a great job of turning an OPML file at a URL into a custom RSS feed.  I found this tip on Frank Arrigo's list of Australian .NET Bloggers (yes, I still monitor the Old Country).  If you have an OPML file visible at a URL it's possible to use Feedster to search them and then consume the results as an RSS feed.  Very cool.

Being able to consume a number of blogs in one feed certainly makes it easier to track a number of blogs in a news aggregator.

Here's the =2004}&sort=date&ie=UTF-8&limit=30&inopml=http://www.sneath.org/tim/ukbloggers.opml&type=rss">RSS feed for UK .NET bloggers and here's the =2004}&sort=date&ie=UTF-8&limit=30&inopml=http://radio.weblogs.com/0124955/gems/aus-dotnet.opml&type=rss">RSS feed for Australian .NET Bloggers.

Tim Sneath is maintaining a list of UK .NET Bloggers.  Hopefully we'll achieve his aim of having another Bloggers' Dinner after the success of last week's efforts.

Update:  Using the power of Feedster, you can track this list with an =2004}&sort=date&ie=UTF-8&limit=30&inopml=http://www.sneath.org/tim/ukbloggers.opml&type=rss">RSS feed for UK .NET bloggers.

Someone who's coming along to my .NET Exchange User Group presentation in Ipswich tomorrow night contacted me about finding a .NET implementation that handles SOAP with Attachments.  Here's some background and a possible solution (let me know if there's a better one).

The Problem
He needs to write a client to consume a web service provided by a finance group that are using Java to publish their messages in the SOAP Messages with Attachments (SwA) specification.   The frustration for his company is that even though Microsoft was a co-author on this specification they provide no implementation that supports SwA.  Omri Gazitts, an Indigo Program Manager has the definitive explanation about why Microsoft has co-authored four SOAP attachments specifications.  Dave Bettin has also been asking good questions about this issue.   Basically the requirements have changed as time progressed.  The main problems with SwA are that it uses MIME which provides no message header length so that the whole message stream must be parsed to find a text boundary between attachments, which is slow, and it doesn't work well with SOAP Headers which makes it hard to integrate with other WS-* specs such as WS-Security. 

Omri has another post that explains the future, which is MTOM that will be backward compatible with SwA.  Unfortunately as Omri says 'we have no plans to support "vanilla" SwA at this time, given that it neither imposes an infoset data model nor is as efficient as DIME - in other words it's the worst of both worlds'.  Where does this leave people like my colleague who need to consume a web service that uses SwA in the wild?

Possible Solution
Well, there is an MSDN magazine article from two years ago that has a lightweight SoapExtension implementation to convert SwA messages into a single SOAP message with the attachments written out as elements inside the SOAP Body (thanks to fellow RD, Christian Weyer for pointing me to this).  It's not too much trouble to convert this SoapExtension to work on the client.  The other part of the problem is an improved MIME parser.  I found the cpShere Email component project on GotDotNet  provides this.   With these two parts I think it's possible to kludge together something that works while we wait for a .NET implementation of MTOM in future.

If anyone has implemented or knows of a better .NET solution please let me know.

Tonight was my first bloggers dinner and quite a night.  I was the first one to sign up and the last one to arrive as Chris Anderson worked out.  What was most surprising was that other than Chris, Don and Ian 'I have no blog comments because I hand-coded my blog myself' Griffiths, I only met a few other bloggers (Chris has a better list).  A lot of blow-in blog readers!  But all are welcome and everyone's company was appreciated.

By the time I rolled up the only chair left was next to Don himself.  Luckily I'd boned up on things to talk about (John 'Policy' Bristowe also primed with a couple of last questions via instant messenge) and managed to garner the following:

• I started off with asking how do you communicate semantic intent with policy?  Don's response was: you don't.  You do it out of band with the telephone.  It's what humans are for.  There are a lot of people trying to apply formal methods to determine the structural correctness of the messages, but the intent is something that humans will work out.
• WS-Addressing.  I raised my beef about it being all well an good to have an opaque string as the address, but where was the protocol binding.  Apparently Indigo is going to use the simple URL style scheme, so a tcp address will look like tcp://domainname etc.
• The important of the Endpoint references is to provide a SOAP cookie style mechanism without having to get into the URL hackery such as query strings.
• Someone said "it just felt unusual to send text-based messages around - isn't that slow?".  Don's response was that the first time he had sex it felt weird, the next time better and now he couldn't imagine it any other way.
• What's different between the M4 and M5 releases of Indigo (Clemens seems to know a lot about this)?  Apparently the programming model has been refactored toto make it more unified and simpler (great news here).  The System.MessageBus.Servicess library has changed significantly because what was there was just not as good as it could have been.  I hope this means better terminology than DatagramPortTypeAttribute and DialogPortTypeAttribute, which always seemed a bit complex to me.
• What's the best way to approach Indigo today?  If you want to learn about indigo today in a way that will pay of in the future, don't worry too much about the high level details.  If you work from the wire level back up it will be most useful (this is least likely to change). 
• The focus with V1 of Indigo is proving that they can get great performance with messaging.
• We discussed Clemens' idea that because Indigo uses a streaming reader over the message body, it may be possible that on intermediary/endpoint may start sending onto another intermediary/endpoint before the full message is received.  Don mentioned that there's no reason to think that the message may ever finish.  Queue zen-like moment of poignant reflection and silence.  'Sort of like one giant Congo-line of a message throughout the Internet' I said.  Don didn't say it but I felt him think 'Yes, Grashopper'.
• We spoke about the fact that my pregnant wife who has always been worried about the size of my head, thought that Don was the only person I have on my PDC photowall that had a larger head than me.  Don said his head was larger than mine in more ways than one.  We agreed that his head was impressive, but he was leaning forward in the photo and clearly his head was not awesomely large (at least physically).
• I suggested Don try the Cinnamon Club for dinner.  We had the Axxiant Xmas dinner there and it was very good (though it was late and I had been drinking ... ).

Afterwards some of us moved off to the pub ...

In preparation for the London Bloggers Dinner with Don Box and Chris Anderson on Monday (see the report of this dinner, and what went on at the pub afterwards), I thought I’d get up to speed and make some notes of the key points made in Don’s interview on the ServerSide.NET.  To my mind this is the best interview I've seen or read about Indigo.

I'm posting these notes mainly so that I can have them handy in SharpReader when I'm offline and in case others might find them useful.  You can see the interview and download the full transcript on the ServerSide.NET site.

What is Indigo? 

• A connective tissue between programs.  It makes it simple for developers to put a message oriented façade on the edge of your application to allow easy communication with others.

Positioning Remoting and ASMX web services

• The problem with .NET remoting was that even though it supported interfaces, it really worked best when both ends of the pipe shared code such as assemblies or DLLs to integrate.  Sharing types like this turns out hard to manage because you often don’t have control over the deployment versioning and testing at each end of the pipe.
• ASMX Web Services (including WSE) are more flexible because they use WSDL and schema which is more adaptable to change than the OO style type interface used in remoting.  ASMX is proving more popular than remoting in the field because it is actually a simpler model and simplicity is a feature.
• Developers should go down the path of integration using messaging because it is simpler and more flexible, but support for remoting won’t go away.
• You don’t have to use XML Schemas and WSDL to describe structures and contracts, but that’s the practical way that everybody does it.
• The problem with components was that they tried to do the right thing separating interface and implementation, but it was too easy to ‘leak’ object concepts and require the sharing of types.

Do components have a place in the world?  

• Don argues that the world is simpler if you think about objects, which are great for programming software that gets integrated, tested and deployed, more or less as an atomic unit and services, which are the central abstraction for deployed, autonomous pieces of software.  Given this perspective he suggests there may not be a place for components.

What’s the difference between service-oriented programming and service oriented architecture?

• SOA is the buzz word du jour, system message bus will probably be next, whereas service-oriented programming is more focussed on the reality of the code.
• Service oriented programming is about four ideas:

1. Boundaries are explicit.  In Indigo you have to place attributes on your code that you want publicly exposed
2. Services are autonomous.  Indigo doesn’t assume you’ll deploy a whole system as a single unit.  Each service should be secure and reliable as a stand-alone unit.
3. Share schema and contract, not class.  It’s too easy to mix up interfaces and implementation, so it’s a better idea to separate schema and contract into two distinct ideas.  Schemas are defined in XML Schema language, and contracts, which are message exchange patterns, are defined in WSDL.
4. Compatibility based on Policy.  To know that services are compatible you have to care about other things than just the signature of the messages.  WS-Policy is supported in Indigo and provides a way to share this over the wire in an interoperable way.

Does Policy impose versioning restrictions?

• Policy is similar to marker interfaces in Java or C#.  You have to have a stable name that is immutable that is decoupled from the actual signatures.  Both parties have to agree on the name and it has very strict version requirements.
• WS-Policy provides a way to declare compatibility with any number of versions of a given policy assertion.  The logic engine is able to determine if party A’s assertion lines up with party B’s assertion.
• In the PDC bits you can see the first message sent on the wire is a ‘get policy’ message.

Can policy be used to create versioning schemas?

• There are three policy specs:

• WS-Policy which is the XML form for writing a logic expression
• WS-PolicyAttachment, which provides a way of taking those expressions and applying them to a particular domain
• WS-PolicyAssertion which contains a small number of concrete assertions that plug into the expressions defined in WS-Policy.

• WS-PolicyAssertion contains an assertion called Spec Version that can be used to determine compatibility at the naming level.  It’s looser than strict or nominal type equivalence that is used in C# and Java.  It provides some flexibility, not complete flexibility and Policy is used to do the version negotiation.

Are there situations where the Indigo approach doesn’t work well?

• In future the service-oriented programming model may be a practical way of doing things like cross-process, cross-app-domain or even within a single exe.
• The focus is making sure Indigo has a solid model that can be implemented and scale down well.  Objects took a ‘near’ metaphor and tried to stretch it out, Indigo is trying to take a ‘far’ metaphor and shrink it.  Indigo is striving to have very low performance costs within a single exe.

What are the performance optimizations being worked on now?

• The team is very focused on make sure that the I/O implementation is as well tuned as possible.  They want to make sure we can get bytes in, in XML and other forms, in to and out of an app domain as fast as humanly possible. 
• They want to make sure that it’s not tied to a specific protocol, such as HTTP.
• The idea is ‘I create a message here, I need it to appear in your program as fast as possible’.
• Performance is an important goal for the first release.

WSE and Indigo

• WSE is the vehicle to give developers access to implementations of the WS-* protocols as quickly as possible.
• The goal is still to integrate them into the .NET platform, but sometimes the platform has to ship before standards are set, which is why WSE will have a future post-Indigo.
• Post-Indigo WSE may be used to track protocol evolution on top of the Indigo infrastructure rather than just the ASMX infrastructure.

Will Indigo do anything new at the wire level?

• Indigo is about the software productization of ideas (such as the WS-* specs) rather than interesting new wire protocols.  For example, transactions on the wire will use WS-AtomicTransactions but within the service they are doing a lot to make transactions easier to use from managed code with extremely efficient implementations.

What's a Microsoft Regional Director?  Fellow RD Jon Box answers this well.  Clemens also has a good post. Basically Microsoft Regional Directors are independent developers and architects, volunteers chosen for their leadership in their local technology circles. You've probably seen many of them at conferences, read their blogs, or heard them on .NET Rocks.

I'm looking forward to continuing to spread the word about .NET, web services and Indigo in the coming year.  As shown on my presentations page, I'm doing a talk on WSE to the VBUG group in Ipswich on Feb 3, then a talk on Indigo in London to the .NET UK user group on Feb 23.

Russ Lewis blogs about David Sussman's ASP.NET 2.0 talk at the London DNUG last night.  I was off on a Microsoft Solution Framework training course with Axxiant so I couldn't make it, but Russ has done a great report.  In the comments he left on my site Russ says that it was one of my YAWPs (Yet Another WSE Presentation) that made him start blogging. 

Good to see that I'm encouraging the community in my own small way.  Keep up the blogging Russ!

From Ted Neward's blog comes the news that he's now the Editor-In-Chief for the new .NET community site TheServerSide.NET.  It's the .NET port of the TheServerSide, a vibrant Java community, but it's backed by companies such as Microsoft and Developmentor.  Ted's aiming to do the following with the site:

1. Create a place where the hard questions can be asked.
2. Promote J2EE/.NET interoperability.
3. Promote open source .NET projects.

Some good points already:

• An excellent sets of interviews, with transcripts, such as Don Box on Indigo, Scott Guthrie on ASP.NET and some guy from Borland. 
• A good set of articles from DevelopMentor trainers such as:
  •  Justin Ghetland on Unit Testing in .NET showing a Visual Studio Addin that still has the NUnit green and red traffic lights.  Any thoughts Jamie Cansdale?
  • Stuart Charlton on Autonomous Services dealing with SOA and the ROX holy trinity of relational, object and xml technologies.
• Pre-release chapters from the Microsoft Patterns & Practices group on Remoting and Managed Code Performance.

This looks like a great addition to the .NET community and already has some quality content.  The only downside, and this is a major one in my view, is the lack of an RSS feed - the front page only has an RDF feed that wont work in SharpReader.   What's going on there?

The Application Integration and EAI Architecture guidelines I mentioned in the last post seem to me to be an answer to Michael Earl's Plea to Microsoft Architects because they talk about problems that really matter today and how you can apply currently-shipping technologies to solve these problems.  The guidelines have contributions from a great bunch of Microsoft Architects (at least I think they qualify for this title) such as Maarten Mullender, Keith Short and Pat Helland (who have given some top presentations at TechEd and PDC recently).

Don Box has already responded to Michael by saying he'd like to see a list of top 10 MSFT bloggers who are focussing on helping people apply shipping bits to the problems that matter.  I agree, but think part of the problem is that many of these guys aren't blogging but are writing great content for MSDN Enterprise Development site (nice reorganisation) or the Patterns and Practices group.  For better or worse, blogging seems better suited to short, time-relevant information such as thinking about the design of upcoming technologies.   The problem is that blogging helps make a topic seem alive and current and creates a sense of community.  It would be nice to combine these benefits of blogging with high quality content already available.

Adding life to the Patterns and Practices content
I'd like to see some way of using blogging and the community to increase the value that comes from the Patterns and Practices and MSDN material.  Every time I go back to it I'm impressed with the content, but sometimes it seems impenetrable, and well, a little dull.  I wonder what can be done to bring it to life and get more dialogue going in the community? 

I know that the Patterns and Practices group are now doing webcasts (disclaimer: I haven't had a chance to participate yet). Perhaps there could be some more focussed community involvement or debate around the architectures or concepts?  I know that Shadowfax, the Patterns and Practices group's project to apply service oriented solutions with currently shipping technology, is available for download.  Perhaps there could be some key questions the group are looking for feedback on?

Perhaps there needs to be a pub architecture club, similar to the Extreme Tuesday Club (now open in Brisbane, Australia).  Either that or find some way to get the like of Pat Helland or Maarten Mullender to start blogging?

The ever-active Patterns and Practices group have released Guidelines for Application Integration focussing on EAI Architectures.   It provides useful coverage of all of many of the issues involved in integrating applications.

The guidelines covers the levels of application integration, such as business process, data and communications-level integration.  It defines the capabilities required at each of these levels.  It also covers security and operational considerations before finally showing how to map Microsoft Technologies to application integration capabilities.

The guide is useful because it deals with concepts from an abstract perspective rather than a technology-centric approach.  All of this is likely to be useful in a service-oriented (it's just not PC to say SOA anymore) world.  The architectural concepts and business issues identified are independent of the technical implementations such as Indigo.

Here's a Public Service announcement for fellow UK bloggers. The London .NET User Group will be hosting Dave Sussman speaking on ASP.NET 2.0 'Whidbey' on January 20th at the Cafe Royal  (not the MS swimming pool room in Soho).  For those that don't know, Dave is a co-author of the Addison-Wesley .NET Series titles - 'A First Look at ASP.NET v 2.0' and a 'First Look at ADO.NET and System.Xml V2.0' (sample chapters are available at both links).

The event is free but book early as it is likely to book out.  If you're interested let Ian Cooper know at meetings@dnug.org.uk 

I'm going to be presenting on Service Oriented Architecture with WSE and Indigo in February.

Doing some performance testing on my current project showed me the evils of premature optimization and the need to test performance first before 'optimizing' the code.

A month ago when I started on the project there was no working deployed version of the application in an environment I could run tests against, so I did some code reviews and eyeballed areas of the architecture that might be slow.  The project uses BizTalk 2002 Orchestrations to process a SOAP message that is passed around as a string.  All of this manipulation was based around the XmlDocument DOM model, which had been slow and memory-hungry in previous projects, so I wrote some code to test the speed of different approaches to XML processing.  Using the XPathDocument and XPathNavigator turned out to be 70% faster. 

However, when I analysed the entire web service method only 6% of the call time was spent in these methods.  So even if we implemented the change it would only improve the overall performance of the web service call by about 4%.  This may or may not be worth it, but it highlighted to me the need to get real performance figures to test my assumptions about where the bottlenecks were in the code. As Donald Knuth says:

"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil."

MSMQ kind of semantics can be implemented [in Indigo] in two ways:  They can be implemented inside a channel, or out in an application serving as an intermediary.

Indigo comes with a channel called the reliable channel that implements point-to-point buffering, point-to point-queuing - so that you can be on a laptop sending messages to me.  The messages stay in a buffer. Later your app goes away, I connect, the messages get sent to me, I reply, the messages sit in a buffer, later your app comes out and drains them.

Or I can build a queue intermediary.  Indigo V1 will ship with a queue class that you can put in an intermediary sitting at an address.  You can put messages to that and you can pull messages from it.

So those are the two options depending on whether you think of a transport kind of thing - so the semantics of me talking to you - or whether it’s a real thing that sits between your and app and the user of the app, where many people may be writing to the same queue and many apps might be reading from it.

Over time - Indigo is a long plan - in Indigo v1 we will be releasing classes, so that you can build queues.  Over time we will be implementing 'queue services' - full bore services that known about clusters and the whole thing.  Parts of MSMQ wont be in v1.  The programmer part will be, but the big old configurable server side wont be.

So, the programming model is there for V1, but a replacement for the enterprise aspects of MSMQ will have to wait until the future.  This would cover the apps where I've message queue in the past (basically a private queue that one app can post to independent of another that reads the messages off), though its not (yet) the replacement to Tibco I was hoping for.

What's the relationship between Indigo, MSMQ, BizTalk and  SQL Server "Yukon" Service Broker?  When and where is each technology appropriate?
Basically it depends what type of application you are trying to build and what environment it needs to run in.  Here's a chart adapted from the DAT406 session:

Technology	Indigo	MSMQ	"Yukon" Service Broker
Environment	Any WS-* compliant endpoint	Windows NT	Yukon on both ends
Application	Any distributed application	Any Async NT Application	Database applications
Message Store	In memory or DB persistent store (Yukon/SQL Server 2000)	MSMQ Message Store (NT File System)	Built into Yukon
Type of Message	Persistent and non-persistent dialogs	Reliable, Express and Transactional Messaging	Transactional messaging only
Protocol	Various	Various	TCP only

Where does BizTalk come into it?
BizTalk is a product that that builds upon other technologies in the Microsoft platform.  As BWill says in comments on a previous post, choosing BizTalk or Indigo will be a question of how much of the infrastructure you want to build yourself.  BizTalk has connectors for MSMQ currently, in future it may connect to Indigo or possibly to Yukon Service Broker.  BizTalk 2004 is currently in Beta it's on a different release schedule than the other products.

Where does Yukon Service Broker fit?
Here's Roger Wolter's answer from the DAT406: Building Reliable Asychronous Database Applications with Yukon:

The key to keep in mind is that Service Broker is a database application framework, not a messaging system.  Yeah we send messages to other databases, but if you're building a messaging infrastructure there's not enough in Service Broker to satisfy your needs.  If you are building a queued, asynchronous database application and you wanted to use reliable messaging to scale out that application, Service Broker is the answer.

There were some hostile questions in the DAT406 session about why it was necessary to put a messaging layer in the database.  John Cavnar-Jonson (who definitely needs a blog) calls it an 'abomination' in the Developmentor Indigo discussion list.  Personally, I think it's part of a Dr. Evil style plan from the SQL team - if they were to add a spreadsheet to the product then a great majority of the world's applications wouldn't need an OS letting the SQL team achieve world domination.  Seriously though, there seem to be several good reasons:

• Developers that know SQL can now  develop queued, asynchronous database apps.  The ability to have asynchronous queues is a very nice architectural feature.  Being able to achieve this with SQL syntax like BEGIN DIALOG ... FROM SERVICE ... TO SERVICE is pretty cool.

• It's all in the one box.  Everything happens within the database.  Backup, restore, installation, configuration, monitoring and security are all there in the one location.  So deployment of the database is deployment of the messaging system etc. (no need to hassle with MSMQ installs).

• The  message broker is the database.  It's easy to query the status of messages, processing the queues is as simple as writing a stored procedure, the database can efficiently throttle the queue processing resources and it's possible to farm out message processing work to another machine since all that is required to process a queue is a DB connection string.

• It's fast. The Service Broker is fast because there's no need for two-phase commits for transactional messaging, there's no need to cross processes to get to the messaging platform and if the send and receive queues are in the same database then it's very fast.

Neils Berglund from Developmentor has been teaching Yukon for a while to Microsoft employees (such as Tim Sneath) and has an excellent sample chapter on Yukon Service Broker that's available for free download.

How does MSMQ fit into the longer term picture?
BWill says in my comments that the Indigo team has shared the love and embraced the MSMQ team into its building.  John Cavnar-Jonson did some research at the PDC:

I completely disagree with the idea that Indigo offers all of the functionality of MSMQ. I discussed this exact question at the PDC with Steve Swartz, Mike Vernal, and Anand (whose last name I don't recall, but he's an MSMQ program manager).  Indigo will offer reliable messaging (which is a huge improvement over current web service technology), but it will not be a full-fledged message queuing system.

As I reported from the PDC, the message was:

"We are not building the uber queuing system - we are not a replacement yet for MSMQ - we have support for routing, but we aren't replacing CISCO, we support eventing but we aren't a replacement for Tibco."

I think there's more to be said in this space.  It's likely that this is about achieving an Indigo V1 release (primarily about unifying the three different programming models and baking WS-* specification support into the platform) and then targeting more ambitious goals with future releases.

Which parts of Indigo will ship in Whidbey and which bit will ship before or with Longhorn?
Basically, System.Transaction will be in Whidbey, the rest later.  I'm still digesting Don's WSV302 Indigo Part2: Secure, Reliable, Transacted Services and Jim Johnson's Transactional Programming on the Windows Platform presentations to understand this more deeply.

Given that the last version of WSE will be wire-level compatible with Indigo and that a future version of WSE is likely to support WS-ReliableMessaging, what are the benefits of Indigo other than the simplified programming model?
Even though I love WSE I'm following the words of Hervey and accepting that WSE is V.Last++.  This question was me fishing for what features Indigo will provide me with as an architect/developer that I can't get from WSE.  BWill mentions:

Advantages of Indigo over WSE: three off the top of my head are performance, integration, and support [it's part of the platform rather than WSE's 2+1 support policy]. I'm sure there are others. Also, note that there is no guarantee that every feature in Indigo will also be WSE.

So, no bites as to what the extra functionality of Indigo might be, so I'm still fishing (e.g. digging deeper into the Longhorn SDK Indigo Samples).  Of course, Indigo has learnt from WSE, so the Indigo programming model will also be nicer (though the WSE programming model is already small and well refactored).

Indigo is committed to supporting WS-* standards and interoperability, but what extra functionality will be available if the whole environment is made up of Indigo boxes?
I'm still trying to get a feel for what features and functionality might be available in Indigo.  BWill mentions the fact that Indigo will likely run faster in an all-Indigo environment. 

Indigo does offer Peer to Peer functionality
Robert Scoble tells us:

I stopped in on Don Box yesterday and he gave me a demo. Indigo is going to radically change how we think of Internet technologies. Imagine something that looks like a website, but that doesn't require a centralized server. Now you're getting your mind around what Indigo could do. Indigo is designed to take advantage of our always on, always connected computers.

It's not difficult to spot an Evangelist with a marketing strength is it :)? Sounds like a simple programming model on top of existing network stacks opens up opportunities to use the Internet for more than just the web browsers against a central web server.  I think I'll review WSV306 Indigo and Peer to Peer apps on the train this week.

On the 'death of objects' and the SOA 'paradigm shift'
Same started off by saying:

SOA [is] one of those paradigm shifts - it really does mean the death of objects at least as we know them.

Before I could suggest that we fine any blogger who uses worn-out expressions like 'it's the death of [technology X]' and 'paradigm shift', Sam later clarifies that he didn't mean objects but OOA/OOD: 

I really meant OOA/OOD - how do you now design/decompose system design/requirements into architectures and I think it's less now about classic OOA/OOD and tightly coupled object design and more about a loosely coupled collection of components under a service.

I agree that proclaiming the death of objects was a misstatement. It may be more accurate to say that components, which are based around sharing types between parts of the system, are likely to become less of a primary focus with the rise of SOAs.  I agree with what Bryan Noye's says:

I don't think SOA means the death of OOP or Components at all. Just like most people build components using OOP, I think most people will built SOAs using OOP and Components. They are not competing concepts but complementary.

SOAs are important where there is a need to share messages and interoperate with unknown others
For my mind, SOAs provide the most benefits when there is a need to share data/information and  interoperate with other groups, possibly on unknown platforms, that you have no control over. There will still be plenty of applications that are built in the current n-tier component Enterprise Architectures. Talking with Jim Johnson at the PDC he made the point that the benefits of SOA with external partners may also be benefits within an organisation or service boundary. I agree, but I think the technology platform (e.g. Indigo) and management tools (e.g. whatever Microsoft are planning here) have a long way to go before these benefits outweigh those of using existing component-based technologies.

Interoperating with others often means going for a lowest common-denominator approach which is always going to perform slower than when you can go with a binary format and control both ends of the wire (as Sam mentioned, using ASMX just doesn't give the same performance as current 'binary typed' systems, which is why Indigo will do special things if it knows it is working in an all-Indigo environment).

SOAs are currently still complex and difficult to build and manage
SOAs are currently still complex and difficult to build and manage. They are complex because the standards are still being implemented (on a recent project I was on it took a major international bank nearly 3 months to convince a market-leading J2EE vendor to adopt SOAP headers and honour the 'mustUnderstand' attribute).  Newer standards like WS-Addressing are still being worked through and implemented.  It's difficult because of the layers of the technology that must be understood in order to build the systems.  Just read Clemens' description of his latest FABRIQ project to see the level of technology understanding, skill set and experience you need to build an SOA project today. As the tools develop and experience and awareness of SOA's grow they are likely to get easier and simpler to build, in the same way that client server and then n-tier were once considered complex and difficult but are now considered main stream.

Services are about outside, Objects/COM+/Enterprise Services/MSMQ are for inside
It's useful to be aware of boundaries as Don Box demonstrated at the PDC.  There are parts of SOA that are designed to be used on public organisational boundaries and some that are better deployed within an organisational boundary or even behind a service boundary.  Clemens Vasters' makes the distinction between 'near and far'.

When using services outside an organisational boundary there are benefits to using open standards and working with contracts and schema rather than type, since it's difficult to control what is at the other endpoint.  Enterprise Services and MSMQ provide useful functionality that isn't yet covered in WS-* standards, but the problems with these two approaches is that they often share binary type information or require a Microsoft box or adapter at the other end.  This doesn't mean they shouldn't be used in SOA, just that they are better used inside the organisation where it's possible to have more control over the communication and the endpoints.  Within the service boundary there's still a service to provide, and it's here that technologies like components/COM+/ES/MSMQ are likely to be just as useful as they are today.

Gregor Hohpe's Enterprise Integration Patterns provides useful SOA guidance
I think that Sam Gentile is right, there are some architectural changes that need to move towards SOA and message-oriented systems.  Luckily Gregor Hohpe has written Enterprise Integration Patterns - Designing, Building, and Deploying Messaging Solutions a great book distilling his experience of these systems.  Hervey Wilson's reading it and its on Ingo Rammer's list of recommended books as well!.

Recently I've been reviewing the most efficient way of retrieving values from an XML document in .NET.  As Scott Hansleman mentions, the best way to find this out is to write some code and measure the performance, so I wrote some code that can be downloaded to test the performance of three different approaches:

• an XmlDocument using XPath queries with SelectSingleNode
• an XPathDocument with an XPathNavigator
• using the XmlSerializer to deserialize into a custom class

The results show that the XmlSerializer is fastest once the initial cost of creating temporary assemblies has been overcome.  In situations where the initial performance is most important then an XPathNavigator over an XPathDocument is the fastest.

Approach
Here's more detail about each approach:

Aaron Skonnard provides some excellent background on the benefits of the XPathDocument over the XmldDocument in his article .NET XML Best Practices: Part I: Choosing an XML API.

For the timing tests I used the code available in the EggHead Cafe article "High-Precision Code Timing in .NET".  Unfortunately the Ticks property of the System.DateTime class is only accurate to around 16ms even though it displays values to 100 nanoseconds.

Results
In this app I take a small XML document with no namespace and retrieve four values from it.  Here are the figures I got when running the console application for the first run and then a repeat run:

Runs	XmlDocument	XPathDocument/Navigator	XML Serialization
1	0.00543	0.00129	0.09020
1	0.00051	0.00035	0.00028

The first time the application is run the XML Serializer has to create temporary dynamic assemblies so it performs the worst, however on subsequent runs within the application it performs the fastest since it can use a cached copy of the temporary assemblies (As Scott found out, these assemblies are not cached in .NET 1.0 if you specify a namespace).  Daniel Cazzulino has some good background on how the XmlSerializer works).  In both the first and the second runs the XPathDocument/XPathNavigator approach is faster than the XmlDocument.

In the sample code I've also presented an ASP.NET web application that can host the same tests.  ASP.NET creates a new AppDomain for each website it hosts and is maintained across requests to the server until it is shut down or recycled. Since the XmlSerializer caches its temporary assemblies based on the AppDomain, using the XmlSerializer in an ASP.NET web application or web service application is actually the fastest technique for retrieving values from an XML document.

In situations where the AppDomain is created per-request then using the XPathDocument/XPathNavigator will be more efficient than an XmlDocument and the XmlSerializier.

Discussion
The XmlSerializer is the fastest way to retrieve values from a small XML file if it is possible to overcome the cost of the creation of the temporary assemblies.  In situations where the first retrieval is the most important, or where more complex XPath queries are used the using an XPathNavigator over an XPathDocument provides better performance than the XmlDocument. 

Thankfully it will be possible with .NET 2.0/Whidbey to use a tool (sgen.exe) to pre-create and compile Serializers.  Doug Purdy covered this in his PDC talk, see Scott's notes.  I believe this will make XmlSerializer the fastest approach to retrieving values from XML, but testing will tell.

Click here to download the sample code.

A month after the Indigo 'Kimono opening' at the PDC there's still a lack of clarity about what Indigo is, how it relates to other messaging technologies and what's the best way to start developing applications today.  While a lot of this was covered at the PDC my perception is that some of the message hasn't been ack'd successfully from the audience.  [Update: See my more recent post 'More on the Microsoft Messaging message' for some answers to these questions]

The Longhorn DevelopMentor mailing list had an excellent exchange yesterday on Indigo, which has lead me to highlight some areas where I'd like a clearer message:

• What's the relationship between Indigo, MSMQ, BizTalk and SQL Server Service Broker?  When and where is each technology appropriate? (Thanks to John Cavnar-Johnson for highlighting this)
• How does MSMQ fit into the Indigo longer term picture? Is MSMQ for within the Enterprise or will it focus more on outside the Enterprise messaging?
• In terms of EAI technologies, where should we look for this functionality - Indigo or BizTalk?
• Which parts of Indigo are going to ship in Whidbey and which bits will come in Longhorn?
• Given that the last version of WSE will be wire-level compatible with Indigo and that a future version of WSE is likely to support WS-ReliableMessaging, what are the benefits of Indigo other than the simplified programming model?
• Indigo is committed to supporting WS-* standards and interoperability, but what extra functionality will be available if the whole environment is made up of Indigo boxes?

I'm  know some of these were addressed at the PDC.  I'm still digesting some of it (for example, the PDC DAT406 presentation on the "Yukon" Service Broker shows how MSMQ, Indigo and Service Broker are positioned, though not BizTalk).  Other questions I'm still researching.

Many others have noted that the PDC Sessions are available online as streaming audio with synchronized PPT and demos, so who do I think I am posting it again? Well, here's my value-add: it's possible to download the Windows Media streaming audio and the demo videos using a free tool called Net Transport.  I'm using this to max out my broadband connection as well as letting me relive the PDC sessions I was too worn out to make when I was at the conference, on my daily four hour train commute.  Today I spent time on the train with Anders, tomorrow more Don and I'm considering using Doug Purdy's presentation as my new motivational tape to help me get out of bed in the morning.

To start each PDC folder has a file with a 0Media.asx file, which is an xml file containing links to all of the media files in the presentation.  Net Transport expands the xml and starts downloading the individual files.  It's even possible to pause the downloads.  Here's the link for Don's overview of Indigo for example  (you can test the accuracy of my transcription of Don's talk at the same time!).  Once you've downloaded the files, edit the Media0.asx file to use local paths to the individual files and launch it in Windows Media Player.  You'll have to move the slides yourself, but it makes it feel more like a slide night.

If I had more time I'd adapt early & adopter's script to download the slides to download all of the streaming content.

I've just listened to Ander's PDC presentation on C# 2.0 (TLS320).  C# is incredibly cool, the new features are great and it's amazingly useful to have Anders explain them.  Languages enhancements are as exciting for a developer as a better quality hammer is to a carpenter.

What's interesting about language enhancements is that to take advantage of them you have to teach your brain to think more like a compiler.  The Java language exam is probably the best example of this that I've come across.  Initially I thought that having to answer questions that the compiler would check was a waste of time.  Luckily a Java development mate of mine argued that it was important, especially in becoming a better developer.  Having done the exam, I have to agree with my mate that there are real benefits from being able to look at code and know how the compiler would treat it.

So listening to Anders it isn't just exciting to know that they've developed a compiler that knows how to compile impressive statements, but also realising that I'll have to train my head to do the same thing.  Luckily I find learning things like this a real pleasure.   Having new C# syntax to learn and play with is like a bright blue sunny sky for me after the grey clouds of my current project (arguing with a developer who stubbornly insisted that GOTOs were the best way to code a particular method).  Sure this looks a bit scary now, but I can't wait to get familiar with the syntax and to start developing with this extra power:

Some random points from Anders speech:

• Generics  These are great in general and I can't wait to be able to use them in anger.  I have a brilliant C++ developer mate who said he wont switch to another language unless he can use collections without having to cast.  Hopefully he'll be swayed.  I also like the use of the T.default and null checking that is thrown away by the JITer if it's a value type.
• Anonymous methods Anonymous methods seem similar to Java's anonymous classes.  Dare says: "[Anonymous methods] fix the lack of anonymous inner classes in C# by adding anonymous methods for use in the exact same situations most people use anonymous inner classes for".  It was good to understand how the implementation works under the covers to allow anonymous methods to access the parameters of the outer methods (Steve Maine waxed lyrical about this feature and lambda methods on the bus back from Universal Studios, now I finally understand his passion).  I love the idea of passing code as a parameter.
• Iterators I did once try and implement these in VB6 and it was a horror, so I'm glad the story is that writing iterators that can be consumed by the foreach statement is now easy.  Also, statements like foreach (Item x in Items.Subrange(10, 20)) are just cool.
• Partial types Anders explained there were two main reasons for this feature.  I was wondering since Juval Lowry couldn't really see much point in them when he did the .NET radio show pre-PDC. Basically they help people who generate code - the generated class can be regenerated without breaking code written in the other part of the partial class, and where there are long classes (though this is a bad idea in general).  It's also a key part of the XAML story. 
• Other enhancements Finally C# lets us have different accessibility levels on property set's and get's, so the set can be internal and the get is public.  This just feels more right.

Roy's at it again, campaigning for IntelliJ and asking Visual Studio to take notice and compete with the advanced developer experience provided by the IntelliJ IDE:

 "I felt like I was not coding alone ... if you decide you should have a new method ... you just type the method name and provide its parameters if necessary.  IDEA will prompt you with an unobtrusive popup with a suggestion to create a new method ... the method will be created in the corresponding class with the proper method signature (including correct parameters), so that you can implement it later"

I saw the same feature demoed by Erich Gamma, (the G in the GoF Patterns book!) when he came to London to talk about Eclipse.  I was so impressed the hairs on my arm stood up.  Forget clapping at the PDC, you know a feature is good when your body reacts with goose bumps.  As I mentioned in a previous post about refactoring, I think this feature would solve a lot of the frustration that Randy mentioned where Visual Studio's Intellisense clobbers any new methods or properties written in tests before the object exists.  It's the difference between thinking of the IDE as a text editor and thinking of it as a developer tool.

Luckily, IntelliJ are developing an IntelliC# add-in for Visual Studio that will bring its intelligence to .NET.  I can't wait to get it.

As Roy laments, what's Visual Studio doing?  I remember the glory days, back when Jim McCarthy (of Dynamics of Software Development fame) was working with the Visual C++ team making the IDE great.  Where's the passion, the spirit, the sense of competition gone?  For all the talk at the PDC there was noticeably little about Orcas, the Visual Studio version after Whidbey.  Visual Studio is good, but it can be even greater.  Let's hope that good things are in store.

Web services are being adopted in the wild
There certainly is a lot of material to cover in web services from the specifications to the implementation, so we only briefly touched on Indigo.  There were 30 or so developers there. Starting with the mandatory 'polling' questions I was surprised that around 60% of them had used web services already.  Of that group about a third had added some form of security (HTTPS, VPN or internal network).  There were some good questions in the group ('Can I use WS-Security in a PHP application?  Will Microsoft compete with Tibco and the EAI vendors with Indigo and/or BizTalk?  Has Don Box stopped using the term GXA?)

Blogs are a great way to learn about Web Services
I mentioned that blogging is a great way of tracking what's happening web services. I showed how:

• Omri Gazitts a PM on the Indigo team, talks about web services attachements and that DIME is still the best choice for today, but MTOM is Microsoft's preferred approach for the future.
• Hervey Wilson, WSE lead developer, has said that WS-ReliableMessaging is coming in a future release of WSE and writes great content about lots of other topics.
• Rebecca Dias, Product Manager for WSE, has mentioned as a result of feedback at the PDC, "Attachments will be abstracted away [in WSE 2.0] so that migration to MTOM will be more seemless(sic)"

How to get started reading blogs
First download a news aggregator like SharpReader or RSSBandit then download my modified OPML file.  It contains many Microsoft bloggers as well as other web service related blogs (including those on Matt Powell's list).

I also promised a link to the code that downloads all of the PDC powerpoint presentations.

As part of a new consulting engagement I started today, I found out that the latest version of the .NET Solution Build & Deployment Process & Tools has a visual designer for MSBuild Files.  The Tools are writtien by Andy Reeves, from Microsoft UK's Solution Development Centre and were the basis for the extensions included with MSBuild.  The download page also has more about why MSBuild was chosen over NAnt.  John Lam has also posted about the NAnt community's reaction to MSBuild.

The new engagement involves 4 hours train travel each day, so I'm going to have to kurb my blogging to get to bed.

Some ThoughtWorks experience with TDD and Refactoring
Dan North, a ThoughtWorker, has an article in the Java Developers Journal called Test-Driven Development Is Not About Testing,  his major point is that 'it's not about the tests; it's about seeing how little you actually need to do and how cleanly you can do it!'  Jeremy Stell-Smith, also ThoughtWorker, has a reply post TddIsOnlyOnePieceOfThePuzzle where he makes the following statements about TDD:

• TDD is a tactical thing.  It helps developers from adding unnecessary complexity and allows them to think about how the code will be used before other developers use it.
• It's important to refactor while using TDD to reduce duplication to keep the system clean.
• If a system has good automated testing and loose relationships between classes, then tests can give good feedback about the extent of damage that a change to the system might bring.
• Tests work best when they test small isolated areas of functionality that shouldn't break if anything else changes.  For this reason, unit tests are easier to manage that end-to-end integration tests

There are also some points that Jeremy makes that seem a bit 'religious' such as 'changing your mind later is about as expensive as changing it later'.  This just doesn't seem rational, but perhaps I'm just arguing for a shades of grey rather than this black and white style statement.  I know that TDD makes it easier to make a change, or to measure the impact of a change, but it's still better to make changes as early as possible because it is more expensive to make changes later.  However, I think that his assertion that 'tests must be as fine grained as possible so that most changes no matter how radical don't affect many tests' seems a fine goal, but is hard to achieve.

Early Microsoft experience with TDD
Chris Anderson points to Randy, a friend of his within Microsoft who's reflecting on his experience with TDD.  Randy says that it takes a lot of discipline to stop writing code before a test case that uses it and that Visual Studio is a hindrance when working test first because the intellisense stops being helpful when the writing test code for methods that don't yet exist (Roy wrote a good post showing how IntelliJ handles this situation in a much cleverer way, hopefull Microsoft will pick this up and copy extend it).  Randy also talks highlights how TDD can be difficult:

 ... it's a completely different ball game when you're changing the behavior of code as opposed to writing entirely new code. Say you've written a sizable body of code with inter-related classes and non-trivial implementations of whatever. Now suppose you want to make a fundamental change to one of the more central class's behaviors ...  substantial build breaks where methods no longer exist, because the functionality they provided has been made unnecessary ... doing test first in this case is tricky. You can start by changing the test that exercises the fundamental class in the new way, and then changing the fundamental class to pass the test. Unfortunately, then you've got build breaks across your entire project, and fixing those involve work that should also be done in test-first fashion. The way I did this was to exclude all the other files from the project, get the fundamental tests working again, and slowly re-introduce the other functionality, tests first of course, fixing/updating as I go. Maybe it's the same amount of work in the end, but it feels more cumbersome

From my experience and the developers I've talked to at the Extreme Tuesday Club, if the code is difficult to test then it can highlight weaknesses in the design of the code (such as it being too tightly coupled).  Put another way, designing code with testing in mind often leads to code that is better designed.  The pain that Randy highlights when working with inter-related classes encourages a designs that are more loosely-related so that they can be more easily tested. 

Mock Objects is an approach to making testing easier in large complex systems.  As the authors of the definitive paper on Mock Objects say testing can improve code "by preserving encapsulation, reducing global dependencies, and clarifying the interactions between classes."  More on that in a future post.

Browsing the Longhorn newsgroups, I found a post by David Solomon saying that the Inside Windows 2000/XP/Server 2003 Interactive Video Tutorials CD-ROM was included in the PDC bags.  I had seen this CD in its innocuous blue cover, but I didn't realise that has 12 hours of video presentations, as well as an e-book version of Inside Windows 2000 version 3, and retails for US$950.

The content is really good.  This is great stuff for me as I need to improve my knowledge of Windows internals.  I would have liked to have attended Dave's pre-conference but I felt I had to hear the Web Services story.  Now with the videos on my laptop I can go through the material at my own pace. I'll never be bored on a train again.

Chris Sells and Larry O'Brien are both having a crisis of confidence over their role as authors or information providers in the new world of blogging, Google and RSS.  I think its a good time to look at what value they add and what's important in helping developers learn. Says Chris in Time For An Exciting Career In Electronics:

what I'm best at in the whole world has either become completely unnecessary or incredibly easy. The skill that I spent most of the last 10 years learning was to look at a group of types, APIs, reference docs, headers, source code, etc., distill it down to a set of architectural intensions and then to weave a story through the whole thing to make sense of it for developers that weren't able to glean the intensions themselves (most often because they had real jobs). I call this exhaustive search for intention officially at an end. I'm not longer needed except as a gatherer for those not yet facile in RSS aggregators (which, luckily for me, is still a large number : ).

And Larry in 'Being run over by the cluetrain':

With the whole gamut of Internet-based communication (Websites, newsgroups and mailing lists, Google, email, and blogs), the typical path between technical question and answer has become much more direct. ... Independent authors and teachers have traditionally exploited the very inefficiencies that are being paved over by these technologies. The community no longer has the same incentive to pay money for magazines, books, seminars, and mentoring / consulting: they get the same substance faster and cheaper, if perhaps not with the same style, context, and specificity.

The mistake being made here is to think that the value of authors is based around converying factual knowledge.  The advent of encylcopedias didn't reduce the need for teachers.  There's a big difference between transferring information and helping people learn.   I think it's a great thing that technical authors can now move up the tree away from technical answers and onto higher order tasks like helping people learn how to adapt to the new technologies (that seem to be changing every three years).

I also think both of these guys sell themselves short.  Even though Chris knows about the intentions behind the software, there's always a market for someone who can tell a story and transfer this information to others.  At the PDC, most of what Don had to say has been written somewhere else previously, but I'll still pay for the experience of hearing him tell the story.  Heck, I've read most of Chris' online articles, but I still paid for his latest book (I wanted the convenience of the book and I wanted to have him tell me the whole story about windows forms, not just part of it)

I'd  also say the valuable skill that Chris has is not his own deep knowledge of the intent behind APIs, but his ability to transfer this knowledge of the intent to me.  Larry's book has the right idea - 'Thinking in .NET', not 101 code examples.  I want these guys to help me think, and learn how to think about the problem, not just to find the answer.

What's clear to me, after 19 Microsoft Certified Professional exams (including sitting the core MCSD exam for the third time), is that knowledge expires fairly quickly, whereas knowing how to learn continues to pay back.  This is exactly what Chris has written about in Learning How To Learn.  I can't believe he thinks he'll be replaced just because some of the MS tech team are blogging themselves.   Maybe he just needs a hug.

When I met the Longhorn Shell Team at Meet The Experts at the PDC I told them I had read Adam Bar's 'Proudly Serving My Corporate Masters', a great book on his 10 years as a developer at Microsoft (that's available to read online).  He left Microsoft frustrated with the fact that Windows never had a proper shell environment.  Well, it turns out that Adam's going back to Microsoft to join the Longhorn Shell Team.

Others have also been impressed with the new shell capabilities, code named MONAD, specifically the ability to deal with objects rather than text streams.  Larry O'Brien says it was the best demo he saw at the PDC, even though the demo was on a bus to venue. As he says 'sysadmin scripting: priceless'

Don Box is a great speaker, but as Scott Hanselman and Steve Maine and this post demonstrate, sometimes understanding and explaining the reasons behind his tenets can be difficult.  As an intellectual entertainment and learning exercise, I'm throwing my hat into the ring, trying to answer a commentor on Scott's website who asked what is meant by Don's tenet of 'Share Schema, Not Type' (actually it was 'share schema, not class'), or as I like to ask 'Why should I use web services (with schema and WSDL) over .NET remoting or DCOM?'.

Steve's Answer
Steve's answer goes like this:

• Types came from the land of COM and were all about binary implementations of interfaces.  Interfaces described the methods that could be called on an object, so they were all about behaviour.
• There were two problems with Types.  Firstly, since they just described behaviour, it was hard to make sure that all implementations of the interface produced the same behaviour (that they were semantically consistent).  Secondly, that once published the interface was immutable and hard to extend.
• Schema are from the land of Web Services and are all about describing the layout of a message. Schema communicate state and nothing about behaviour.  Contracts are the way that schema describe behaviour by specifying the format of input and output of messages that are sent to a service.
• Schema solve the problems of Types.  Schema can extend types by using the open content model.  Schema can ensure semantic consistency across implementations by specifying the schema of the input and output messages.

I like Steve's story, but I'm not sure it's the answer.  The open content model is one approach for extending schemas, though it still works like COM and requires that the newest version provide all of the content required for all previous versions, so I'm not sure how much of a victory this is for the schema approach.  Also, I'm not sure of the differences in terms of semantic consistency, between a COM interface that defines the types of input parameter and output parameters from a method, and contracts based on schema that defines the format of input and output messages of a service. 

Scott's Answer
Scott's answer was that types are unique and immutable and require the sharing of an assembly, whereas schema is a description of the XML content that acts like a contract between parties that can be used without the need for an assembly.  I think this emphasis on avoiding a binary assembly is closer to what Don was talking about.

What was Don thinking?
In Don's PDC Session “Indigo": Services and the Future of Distributed Applications, (slides here) session, Don's tenet was 'Share schema, not class: Integration based on message formats and exchange patterns, not classes and objects' (see Tenet 3 of  my transcript of Don's talk at the PDC, which now seems less comprehensible than I'd hoped).  In his MSDN article Don restates it as "Services share schema and contract, not class". 

"Classes are convenient abstractions as they share both structure and behavior in a single named unit. Service-oriented development has no such construct. Rather, services interact based solely on schemas (for structures) and contracts (for behaviors). Every service advertises a contract that describes the structure of messages it can send and/or receive as well as some degree of ordering constraints over those messages. This strict separation between structure and behavior vastly simplifies deployment"

So, I think the advice behind Don's tenet is to build services using the service-oriented approach (using schema and WSDL to bind the input and output message) rather than an object oriented approach (DCOM or .NET remoting).  This is because it is easier to evolve the structure of the input and output messages using schema rather than having to redeploying an assembly or a binary interface description (*.tlb).

Don's argument is that service-oriented approaches can evolve more flexibly because they can use "features such as XML element wildcards (like xsd:any) and optional SOAP header blocks to evolve a service in ways that do not break already deployed code."  I'm not entirely convinced that COM didn't provide a similar level of flexibility as the xsd:any wildcard (e.g. using As Object in VB).  The optional SOAP headers seems feasible.

What do I think?
I think that interoperability across binary platforms is the key reason to favour schema and contracts over classes.  Object-orientation and types are all about binary representations that are just too difficult to get all of the binary platform vendors to agree on.  Schema provides a machine verifiable mechanism that currently uses text representations which are the lowest common denominator and can be implemented across all platforms.

Within a platform, I think using schema and contracts (web services) over classes (DCOM/remoting) wins on pragmatic grounds because schema and contracts seem easier to deploy (no binary object copying required), debug (it's easy to sniff the packets on the wire to work out what's going on) and evolve (using schema and wildcards seem easier to work with than interfaces in COM, especially when with the XML versioning improvments in Whidbey).

Well, it's been an amazing PDC.  Before I came I thought it would be mostly about the sessions and the technology, but it's also about catching up with people, making new relationships and socialising.  So many times I've thought that I'll have to wait for the DVD to review the material and let it all settle in my head.  The time here has mostly been about understanding the general directions and seeing the motivations and directions for the technology.  As one friend said, "why go to the sessions to watch the PowerPoint when I can go to the Microsoft Product Pavillion and get a Product Manager or Lead Developer to take me through it?"

Some outstanding points;

• Meeting Chris Sells and Steve Maine in person.
• Catching up with all the bloggers shown on my photo blog roll.
• Chating with the Microsoft employees at the Pavillion, the sessions and in the speaker lounge.
• Seeing Jamie Cansdale get the opportunity to demo his NUnit Addin and other tools to Anders Hejlsberg (inventor of C#) and James Newkirk one of the creators of NUnit.
• Seeing my blog stats go through the roof.
• Spending an evening at Universal Studios and managing to ride only th escalators as I was so busy catching up with other bloggers that the rides had shut by the time I got there.
• The end of having to guess what Indigo is.

I'm really excited to see the new technology and to get on board the Indigo wave.

In the panel on Indigo a question was asked about how far Microsoft were going to go in making Indigo support pub sub messaging and competing with Tibco.  The answer was that Microsoft wanted to go all the way, but they couldn't do it all in V1.  The core infrastructure will be there in V1, but it will get easier to access in future versions.

It's 8:30am and there's a room full of geeks wanting to get deep down and dirty into the Web Service protocols.  Omri is the Product Unit Manager for the Advanced Web Services group, so he's the man responsible for the WSE team.  He demonstrated the secure, reliable and transacted demo that Bill Gates gave with IBM on 17 September.  The slides for this session are available.

Positioning Indigo's Protocols
The wire protocols that Indigo uses are key to interoperability story.  Omri is positioning Indigo as the Internet's L7 protocol, comparing Web Services as the top of an Internet stack above XML on top of HTTP on top of TCP on top of IP and onwards.  Microsoft see Indigo as supporting applications between mobile devices, P2P, B2B and EAI style applications.

The Secure, Reliable and Transacted Demo
The demo between Microsoft and IBM showed a commitment to interoperability, as well as a commitment to making sure the protocols involved in web services were royalty free.  The demo was a Supply Chain Management application involving a supplier managing inventory and ordering parts from an auto manufacturer.  It demonstrated security, reliability and transactions in an interoperable way.

MindReef  SoapScope - a better SoapTrace packet sniffer 
Omri showed the messages flying on the wire using MindReef SoapScope, the same tool that Scott Hanselman used in his Learning To Love WSDL presentation at TechEd 2003 (the funniest presentation I've listened to).  Werner also recommends it.  It looks like this is a better approach than MSSoapTrace, which is so buggy that it makes development a horror.  Are Microsoft going to fill this hole?

Web Service Design Principles

General-purpose, composable protocol framework.  Web services allow a building block approach, rather than a monolithic stack like DCOM.  The use of SOAP headers allows the various specifications to be improved and versioned in isolation, without having to redo everything.  Omri mentioned that making the specifications composable was a challenge.  He pointed out that WS-Attachement which was based on the idea of using MIME attachments, based on email, was a mistake that couldn't be combined with other SOAP specifications, like security that relied on SOAP headers.  It became clear that to add security to attachments they would have to reinvent many of the same ideas used to secure SOAP messages: a better model would be to put the attachment within the SOAP envelope so that WS-Security using SOAP headers would provide security.  Omri mentioned that they had learnt from their mistake and were proposing MTOM (message transmission optimization mechanism) as a SOAP-based attachment specification.  As Martin Gudgin says, DIME is dead.

Architecture is meta-data driven, policy based.   WSDL provides some meta-data about a server such as the basic information about messages and ports.  However, this is not enough to describe what the service needs.  This is where policy comes in.  Policy provides another way of expressing the extra metadata that a service needs ( Basically schema is not enough to describe what the service requires).  You can say things like I am a service that requires strong 128 key encryption.  Also, I support HTTP but I am inside a trust domain with you so I can go faster by UDP or TCP.  Deployment information and info about a service that can't be captured in WSDL is in policy.

Protocols are factored from both transports and applications semantics.  Basically web services place information from the transport into the soap message, so rather than using HTTP request response model based on HTTP headers, the information is placed inside the SOAP message to provide abstraction from the transport. 

Architecture allows intermediaries.  Having intermediaries is useful for routing and other reasons.

Uniform data model for protocols/content.  The same infoset is used for both the header and the body.  It allows for header and body-based content-based routing.

Broad industry partnership.  Nothing was going to get off the ground without interop.  Interop was necessary for a transformational change. Omri says that Microsoft knows interop is essential if Web Services are going to be a revolutionary technology.

WS-Addressing
Addressing is a core part of the architecture, the work horse of the stack.  It provides a mechanism to encapsulate identity information in an XML end point reference. There is a basic <Address> element that is the network address of the resource.  There's also the <ReferenceProperties> element that can be used for extra information.

The idea is to allow people to host more than one service at a particular network address.  So if there is a service facade it can host multiple services, without having to do ugly bizarre things with the URLs.  The addressing mechanisms are transport neutral and the internal resource organisation is opaque.

The header has to have a to and action.  It can also have a From, ReplyTo, FaultTo, MessageId (a unique id) and RelatesTo (that provides a way to correlate the message).

Attachments: WS-Policy, WS-PolicyAssertions, WS-PolicyAttachments
Policy provides a framework for making statements about resources.  The policy can say that the service requires (such as transports and security).  Policy can also be used at runtime to make sure that the messages meet these requirements. 

WS-PolicyAssertions provide some basic requirements that can be expressed such as TextEncoding, Language, SpecVersion (Uri asserting compliance with a namespace spec) and MessagePredicate (XPath statements about the message)..  WS-PolicyAttachments solves to the problem of how to attach policy to an endpoint (e.g. matching an address with a policy).

WS-MetadataExchange
This addresses how to get metadata from another endpoint.  There isn't yet a specification to cover this although the demo used a simple get to retrieve policy, WSDL and schema.  It was implemented in the bits using a simple GET mechanism.  Kevin Hammond mentions some more details of how Indigo is doing it in the current bits.

WS-Trust
This defines how to broker existing trust relationships.  It defines how to exchange security tokens using a security token service.  The specification details a RequestSecurityToken message that is sent to a trust service, that replies with with a RequestSecurityTokenResponse.  This is currently supported in WSE and Hervey Wilson, the lead developer has posted a code sample that demonstrates it.  WS-Trust is one of the building blocks of WS-SecureConversation.  

WS-SecureConversation
WS-SecureConversation defines a mechanism to allow clients to make multiple calls to a web service, without incurring the overhead of asymmetric encryption.  It involves using a SecurityContext token based on symmetric encryption.  Think of it as a security 'session' object.  The token can be received using WS-Trust to request one from a security token service, or from having one party create one, or through negotiation.

WS-SecurityPolicy
This specification allows for policy statements about security tokens.  It can be used to specify the types of tokens, the confidentiality or encryption requirements, the algorithms to use and the which parts of the message require signing or encryption.

WS-ReliableMessaging: better QoS, less code
Reliable messaging allows developers not to worry about an inconsistent network.  It provides ExactlyOnce and in order messaging that saves developers from having to write code to worry about lost, duplicated or delayed messages or endpoint failure.  Sent messages are stored in a persistent local store and are only removed when an ACK is received from the server, otherwise they are retransmitted.  Early and Adopter have more discussion. Rebecca Dias has a Starbucks metaphor for the specification-challenged.

Transactions
Transactions are described by two specifications, WS-AtomicTransactions and WS-BusinessActivity, based on the WS-Coordination.

WS-Coordination.  The common parts of the two-following specifications.  It deals with how to set up a context for the transaction, based on WS-Addressing.

WS-AtomicTransactions.  This is the standard two-phase commit (prepare, rollback or commit) using web services.  It uses locks to prevent others accessing resources.  It works best inside an organisation where you can trust the locks to be used correctly.  Don Box calls this the 'anti-availability protocol'

WS-BusinessActivity.  This is a more flexible approach that doesn't use rollbacks or locks.  Instead transactions can be reversed using compensating actions.  In this way the system continously moves forward, there is no need to roll something back (if money was credited into one account, but something else in the transaction failed, post a credit to correct it rather than using a transaction to lock and roll back the initial credit).  It also removes the need for complete agreement between all resources before committing.  Omri's example was a transaction that books a flight, hotel and car for the PDC.  If the car isn't available, he still wants to make the trip and use a bus or a taxi, but there's no need to cancel the booking.  This approach is best for loosely coupled (think external web services) situations, but isn't in the PDC Indigo bits (they think it's important, they just couldn't do it in time).

Omri has announced he will repeat his session on "Indigo: the Web Services Protocols and Architecture" which is great.  Martin Gudgin says that they are demonstrating the sample from the Gates/IBM demo (I want these bits in WSE!).  I saw the comments on the WS-Transaction specs that I'll blog on tomorrow (or I'll beg Drew, who does a very, very thorough job of session notes, to go).  I'd like to sponsor Drew to go to conference I can't make it to in future).

I was so busy catching up with people over lunch I got there late and was sitting in the corridor watching the TV again.  Even though I spoke with Omri last night at Meet the Experts, I didn't recognise his voice and the TV only shows the PowerPoint from the smaller rooms.  I thought that it was Don Box with a shot voice from singing with Band On The Runtime.

ObjectSpaces introduces a mapping layer that separates the business logic from the data access logic to reduce the amount of code to maintain.  It's a declarative mapping between objects and relational tables.  It makes sense when you have a strong or large business logic layer.  It will be available in Whidbey.  They will also have a nice User Interface in Visual Studio to help with the mapping.  I've heard from some ThoughtWorks friends that it's not as good as the Java versions or Neo, an open source product from ThoughtWorks (this guy may have been biased ;)

It works with mapping files to tranlsate from an object query/update to a SQL query/update.  The idea is that objects are responsible for saving their own data and can just call a .Persist method to save the data to the database.  One of the examples is showing how to use GetObjectSet to execute query strings against the objects.  It's a new query language called OPath that lets you write select statements such as this to return data:

It's being given by Luca Bolognese who has promised that his goal is to avoid PowerPoint hypnosis and aim to make sure that no one falls asleep.  He's Italian (yes, really) and a very funny speaker.  However, his accent can sound funny.  He's reading out SQL statements and adding an 'a' to every word, so when he's reading SQL statement it sounds like 'selecta ... froma ... exista'

Classic quote:  Someone asked why they ar eusing OPath rather than XPath.  He said 'we used XPath in the prototype and gave it to the group of programmers, and they came back and said it was too hard.  So we did a search and replace to switch the slashes with dots and they loved it'.  Actually it appears that the two are different problem domains.

I'm sitting with Peter Provost (by chance, the photographer of many of the photos in my photo blog roll), soaking up the blogging energy that he's emitting.  He's actually writing the source code as we see it (a true touch typing programming god!) so perhaps he'll post it later (Peter's notes are now available).

Update: Paul Wilson left a comment pointing to more examples in his article on ObjectSpaces. 

Steve Millet is talking about the improvements in the Indigo model for security tokens.  The good news is that the madness has stopped: when a UsernameToken is validated you only need to return a bool rather than the password.  WSE 1.0 and 2.0 require the password to be returned allowing WSE to work out whether they match.   This was uncomfortable for several reasons, such as the fact the password might have been hashed, or just the fact that sharing the password back with the framework feels like a 'boundary violation'.  I'm glad that we're seeing the end of this bizarre API practice.

Other interesting tidbits were that SAML tokens will be available in Indigo.  Now, if they were only in WSE ...

Existing WSE/ASMX applications are likely to have a good upgrade path to Indigo, with similar security attributes.  There are also extensibility hooks in Indigo to do custom security token handling, so there's an upgrade path for WSE (though this is almost certainly having to write code).

The message from Joe Long a PUM with XML Enterprise Services is that there's an easy migration path from existing applications to Indigo.  The changes are mostly using different namespaces and attributes.  Most of the code inside methods stays the same, except for .NET remoting where instead of using 'new' inside a method the developer has to explicitly state that the object is created remotely

Key Questions

• I have an app - how do I make binary work with Indigo?
• Now how do I move the code to Indigo
• If I start writing a new app today, what should I do to best set myself up to move to Indigo?

There are two approaches to integrating:

• Teach the infrastructure new protocols. Put WS-Security into the existing infrastructure like DCOM and MSMQ
• Teach new infrastructure with existing protocols.  The problem is that these protocols don't interop or work over internet.

Various points

• ASMX code migration is basically just a matter of changing the namespaces and attributes, the code within the methods just work.
• .NET remoting requires similar basic changes.  Is .NET Remoting dead?  No, it's used inside the same process, creating objects and treating them like local objects.  The only difference is the new - you have to declare whether you're doing it locally or remote.  Channels or sinks are not supported, but most of the time these were used to overcome limitations in the platform.  Now things are better.
• There will be a new version of WSE (WSE 3.0?) before Indigo is releases.  The latest version of WSE will also be supported.

WSE code migration

•  Migrating WSE code to Indigo may require a non-trivial development investment.
• There will be whitepapers saying how to migrate.

The key PDC message, or welcome to the Service Oriented World

• Objects are appropriate within boundaries
• Services are appropriate across and within boundaries

What to do today

Build services using ASMX

• Enhance your ASMX service with WSE if you need the WSE feature set and you can accept the support policy

Use object technology in a service’s implementation

• Use Enterprise Services if you need ES rich feature set, you are communicating between components on the local machine and have performance issues with ASMX or WSE
• Use .NET Remoting if you need to integrate with an existing proprietary protocol, you are communicating in-process, cross app domain
  Use System.Messaging if you need the reliable messaging and queuing features in MSMQ

Things to watch out for today

ASMX

• Avoid or abstract using low-level extensibility such as the HTTP Context object

.NET Remoting

• Avoid or abstract using low-level extensibility such as .NET Remoting sinks and channels

Enterprise Services

• Avoid passing object references inside of ES
• Do not use COM+ APIs – use System.EnterpriseServices
• Do not use MSMQ APIs – use System.Messaging

I saw Chris Sells pull his laptop out and start coding with someone on the Windows Form table.  Anders was talking to a bunch of people about C# and all just about all of the Indigo team was there.  I could name drop and say that I enjoyed talking with Ingo, Don, Omri and others, but I wont as that would be showing off.  I got to meet Brent Rector who autographed my copy of his book (the first time I've ever had a book signed!).

It was such a lot of fun.  Once I'd exhausted all of the questions I could think of I sat down to eat (I needed real food after missing lunch and trying to get by on sugar in many different forms).  At that point I noticed a big whiteboard sign: Longhorn Shell Team.  No one was talking to the guys so I went up and said 'are you going to make the shell a first-class citizen in Windows so that my Java friends will stop buying the Mac for its Unix shell'.  Happily the answer was yes.  This is great as it's been a long time problem with Windows that it doesn't have a decent shell architecture.  Adam Barr wrote about this in his book 'Proudly Serving My Corporate Masters'.  The shell guys were really pumped and visibly passionate about what they are up to (creating a new scripting language, manipulating the system through objects with properties rather than text).

The fatigue is showing: I got the session code and the room number mixed up so went off looking for a non-existent room and have ended up sitting in the corridor watching the television.  I met Doug at the Microsoft pavillion before the session and he's in fine form (so excited that he's losing his voice).

Some key points:

• Serialization is not the same as XML programming.  Xml programming is using XSD to get CLR type.  Serialization is about saving a CRL type to XSD or to another CLR type.
• Indigo uses a new attribute [DataContract] instead of [Serializable].  This is part of the general shift from the framework doing things invisibly, or by magic, to the new approach where the developer must opt-in.  Developers must mark up the class with [DataContract] and the members with [DataMember].  Like other attributes in Indigo, these attributes don't care about the CLR's visibility because serializing into XML is not the same as controlling access to objects.
• It's possible to deserialize one CLR type into a different CLR type provided that the serialization class and member names match.
• [DataMember(VersionAdded = 2)] allows you to support versioning in schema.

How to version schemas
A group is added to the schema called 'UnknownData'.  The previous approach that Doug mentioned at TechEd was the open content model. The downside of the open content model is that you lose the checking in future versions.  UnknownData uses the open content model, but it includes a version number which means you can continue to extend the schema.  This works in an interoperable way.

I took extensive notes from Don Box's WSV201 Indigo overview here.  I put them on a separate page because of their length.  Between these notes and Tim Sneath notes you've basically got a transcript.  Here are some questions I've still got

What shall we do about WS-ReliableMessaging and WS-Transactions and WS-Federation before Indigo ships?
Indigo is meant to ship with Longhorn, which Don admitted was still 'a few weeks away' yesterday.  So what are we going to do in the Interim?  Since we know Hervey and Keith from the WSE team have been working all summer on WS-Federation, WS-Reliable Messaging and WS-Transactions I'm assuming these bits exist today within Microsoft.  Will we be able to see them or will we have to wait for Indigo?

How does Indigo fit with existing EAI technologies?
We got the message yesterday that Microsoft are building the service bits that if they didn't write, we'd all write again ourselves (or have already written).  I'm still interested to see how for Indigo will go.  As Don said yesterday:

"We are not building the uber queuing system - we are not a replacement yet for MSMQ - we have support for routing, but we aren't replacing CISCO, we support eventing but we aren't a replacement for Tibco."

I'm looking forward to understanding exactly how far Indigo will go, and where we might still need to use MSMQ and Tibco.

Is Remoting really dead in applications or does it just suck at Interop?
There's been a lot of discussion about whether Remoting is dead.  I'm not a big user of remoting, preferring WSE, but I'm not sure that it's fair to say that remoting will have no future.  As Ingo mentions, Indigo will support whichever method you want. 

Brent Rector's book that was given out with the CDs shows that Indigo has RemoteObject services which are an improvement on the .NET remoting model.   The same decision matrix is involved with RemoteObjects vs. Web Services as with .NET Remoting and Web Services.  RemoteObjects are useful where both ends of the wire share the same Indigo platform and when you need to marshal the object across machine boundaries.

There's some great stuff in this road map presented by Brad Abrams and Jeffrey Richter.  It's clear that Microsoft have been doing more work on the patterns, speed improvements (the team must love being able to tune a V1.1). The session was an overview of the trolley load of goodies that come with the Whidbey release of the CLR. Brad Abrams mentioned his goal of making sure that the framework uses consistent design patterns. An example of this is the WinFX poster that shows how all of the technologies relate.

How .NET is being adopted

• 60% of the fortune 100 have .NET
• 70M machines have .NET Framework
• NET has had about 100% growth a month this year, recently growing to 300% growth per month

Brad mentioned that blogs and .NET user groups are helping to contribute to the success of the .NET framework (along with over 400 books).

Trustworthy Computing
Trustworthy Commitment is big deal for Microsoft. A few years ago security was just a feature of the product, with a single feature team. Microsoft now understand that security is a horizontal foundation that is thought about in each spec and code review. It's all part of the Trustworthy computing model. Some visible impacts: 1000's of hours of review, external parties have come in to audit the security code. The goals is to help developers build secure apps on top of the platform. This is where the Prescriptive Architectural Guidance comes from.

Base Innovations
These are the most important parts of the system:

• It works seamlessly on 64-bit support. Writing managed code is an investment that will pay off in future.
• Performance. Making the runtime faster to start up. Also raw throughput on ASP.NET - fastest response time possible (optimising the environment and code paths)
• Edit and Continue is here.
• 70's innovation - coloured consoles. This could open up a whole new field for games developers.

Generics
Added new IL instructions and changed metadata tables to get support for Generics, so VB can share it as well. It's also in C++ (why this is better than templates I'll have to look up). Generics have also been added to the common language runtime, which means that all the 3rd party languages can adopt generics (e.g. Eiffel).

Data Access
ADO.NET has no model changes. This is so incredible that it's worth repeating: Microsoft have not invented a new way of doing architecture in the Whidby version of the Framework. (this is extrordinary given the changes that have happened in the past - where did they shift the ADO guys too - surely they would have been itching to do some improvements), the focus is advanced features and performance.
System.XML is core to the platform. There are some performance improvements (from 50 - 100+% for different areas especially XSLT). Also support for Xquery for better support.

ObjectSpaces
ObjectSpaces. Treat rows and columns as an object graph. Based on an xml file that defines the mapping between the relational and the object representation. It basically creates an domain/business object wrapper around the database.  Having to remember the ordinals for rows in a columns is annoying. However, with ObjectSpaces the first things is to set up a connection

ObjectSpace os = new ObjectSpace(myConnections, myMappings);
foreach((customer c in os.GetObjectSet<Customer>( "city = Seatle")){
Console.WriteLine(c.name);
}

Data Acces - System.Data.SqlServer
The integration with Yukon looks like it uses the attributes, such as :
[SqlTrigger("EmailReview"), "Reviews", "FOR INSERT")].

The example Jeff showed how easy it is to write a stored procedure that sends and email when a new review is posted to a database. This used to be a horror in previous version of SQL Server. Just knowing that extended stored procs wont bring the machine down in a bunch of flames is big deal. This could seriously impact consulting revenues!

ASP.NET 2.0
A major release for ASP.NET. The team went away and looked at common code and controls that teams did. The goal was to reduce the plumbing code by 70%. This has been achieved with page framework, 40 new controls

Casini - the old Personal Web Service is back. This times its 100% managed code. It picks a random port each time it's run to protect from someone trying to hack into it (don't they trust developers to lock down their machine )

Where did all the code go?

Building Blocks
They build a range of "Building Block" APIS
Membership control (username/password, resetting the password)
RoleManager (control access based on role)
Personalisation (customize the layout of the site. You can define a class (e.g profile containing name and zip code) that is associated with the logged in user).
SiteNavigation - tracking how the users move between pages
SiteCounters - useful for sites that are paid based on behaviour such as view
Management - IT department get a page or email if there's something going wrong on the site.

The provide an abstract Provide Model Design Pattern that controls the storage of the data behind these controls. Very, very nice.

Page Framework Features
MasterPages - eack. Sounds like some of the FrontPage guys escaped and let loose the nasty FrontPage themes into the ASP.NET page
Themes/Skins - separate the UI from the logic so that it's easy to skin without changing code (like the themes in DasBlog)
AdaptiveUI - all of the controls will render to small handheld devices

Control Buckets (over 40 new controls)
Leverage the previous features to do tings like Security, Data, Navigation, Web Parts. These controls know how to talk to the underlying controls. One example was the bread crumbs links at the top of page.

Innovations on the Web
ASMX performance is being improved through making sure the server side requests per seconds is much better. Secondly there's a smaller working set required on the client to call a web services. They've also notice that web service calls must be asynchronous (the button shouldn't stick down while the web service is working). You need to use a thread pool. This is a little complicated for some developers with the IAsyncResult pattern, so that now this is much easier - this should be the main stream way to call webservices. It ends up just looking like an event.

.NET Remoting - authenticated and encrypted channels. I wonder whether this is WS-Security compliant? It doesn't have to be since remoting is about two .NET machines rather than interops.

System.Net - this has better 'network awareness'. For example Outlook 11 detects the type of network and adjust the experience based on that. FTP protocol support has also been added.

Client Tier with Windows Forms - System.Windows.Forms
Lots of developers wanted to move to Windows Forms, however deployment of client applications is still too difficult. .NET started to make it easier (each assembly has it's own metadata). Whidbey is concluding this story : click-once. It should be as easy to deploy a forms project to clients as it is to deploy a web server. 

XP Theme support has been added. Finally you can look like Office (why is this always a couple of months after the Office release?).  Apparently the Office team will be here this week showing how to make the Outlook interface in 100% managed code.

Longhorn Related

Windows Forms app will work great on Longhorn.  There will be a two-way interop with Avalon.  You can use Avalon markup and mentions win form controls, or you can use win forms on Avalon.

This session was on the Patterns and Practices Group at Microsoft.  It was hosted by Jackie Goldstein and included James Newkirk (originally ThoughtWorks, and now Dev Lead for the Patterns group.  Aside: he's writing an exciting looking book on Test First Development in .NET) and another guy from the team. 

Easier to Contribute
Several participants mentioned wanting to make it easier to contribute changes to the patterns and application blocks. So that if a company makes changes they don't need to maintain them separately in their own versions.  The team is looking at using the community workspaces.  James Newkirk mentioned using the Adapter pattern to switch between the shipped source code and any local revisions.

Improve the Help Files
The documentation is currently very class based, what the members and classes are, rather than on how to use them.  Sometimes the samples are too simple and don't show how to use the advanced features.

Some of the QuickStart/JumpStarts are too difficult, especially the User Interface Process (UIP) model based on the Model View Controller pattern.  One participant mentioned integrating the help files into the Visual Studio collections.

An MCT trainer asked for Microsoft Official Content (MOC) that used the patterns group.  The MS people said they are trying to get out there and integrate it with MOC courseware.

Conflict between RAD perspective and Architecture
Visual Studio promotes a RAD, drag-and-drop, visual-designer backed RAD tool.  One participant said they'd like a Visual Studio Add In to help integrate the patterns.  Often the keynotes at the PDC are the quick RAD solutions and this comes across as what Microsoft thinks is Best Practice.  Guidance is harder to market.  One of the Microsoft guys said that marketing the RAD features was a more important goal for Visual Studio than marketing the architecture.  Marketing the Architecture is harder.

Integration with the Visual Studio and Language Teams
Apparently the community around the patterns group (bloggers and speakers like Scott Stansfield from Vertigo) had forced the the Visual Studio and Language teams to work closer with the Patterns Group.  There is a disconnect between internal Microsoft Groups.  The Patterns group said they didn't know that the ASP.NET Starter Kits existed until after they were released).

FX Cop for Architecture
Someone asked for an Architecture Cop like a FX Cop.  Apparently someone in the patterns group also had this idea.  I'd like to see improved checklists as a first step on this one.

Why isn't there more focus on the GoF Patterns?
James Newkirk said they will highlight them as they use it.

Brad Abrams mentions at the Rotor BOF session mentions that Rotor (a free, fully functional implementation of the ECMA #335 standard for a common language infrastructure.) has moved from a skunkworks projects to being fully managed by the CLR team.  When code is checked into the CLR there are tests that determine whether it might break anything  in the Rotor Unix build.  They are looking to release a Whidbey version of Rotor after Whidbey has shipped (someone mentioned September 2004).  Other points:

• There's also a community site at http://sscli.net 
• There's a rotor list at the developmentor site.
• Intel does stuff with Rotor (some of the guys were in the session)
• It's focussed on academic, non-commercial use.  For acedemics and those that are interested in getting under the covers and understanding how .NET works.  The license forbids re-releasing or using it in a commercial product.
• I know a lot of people who use it to run the CLR on Mac OS X
• It's popular with grad students working on Rotor for thesis topics like garbage collection tuning.
• It's for people who want to 'fix' who the CLR, for example Chris Sells got some research funding to look at deterministic finalization.

Brad Abrams wanted to know whether anyone was interested in using the JIT compiler and making it available in the Rotor distribution - you couldn't modify it but it would allow for better performance.

It's interesting to see this effort from Microsoft based on building a community of interested people.  Quotes: 'If you blog it they will come'

Tim was talking and Don was typing. I guess Marting was blogging (I couldn't see - I was on the floor at the back with the power outputs).

Aside: recently I've been watching lots of these presenters using Windows Media. It has an excellent - play at 1.4x button that means you can listen to an hours presentation in 43 odd minutes. It's great - the presenter's pauses go away and it's surprisingly understandable. However Tim Ewald is the only guy who this doesn't work for!

You can play at the Raw XML level in SOAP messaging
You can, but it's messy - you have to handle the SOAP processing rules (headers, actors, etc) yourself. The benefits are that you can play in XML rather than object land - you can use transforms, Xath queries etc.

Various ways of using the objects discussed in the first part of the day were mentioned. Essentially similar to Don's MSDN TV presentation.

But working this way in a live demo is hard
Much fun was had trying to map a WSDL schema back into an ASMX page with the correct WebMethod parameters. Don eventually made Tim go back to the PowerPoint slides while he had a go. By now (near the end) they still haven't got it working (showing that this is a hard core thing to do. Evidence: MSDN does it but required Tim Ewald to implement it . This is gratifying, as I spent a night trying to generate my own provider of the Amazon web service so that I could demo the Amazon web service at talks where there was no Internet, but I gave up after a couple of hours hassle.

SoapExtensions
A description of how to use SoapExtensions to ease the problems of processing the Mandatory headers in SOAP messages. See Tim's article Mandatory Headers in ASP.NET Web Services in May 2003 MSDN Magazine

The Do's of .NET Web Services

• Build web services using ASMX - it can support objects or XML focussed web services equally well.
• Enterprise Services are good within your service, but not exposed externally.
• .NET Remoting is for cross-application domains. Ergo, only with two .NET end points.
• Use WSE for advanced protocols support

The Don’ts of .NET Web Services

• Don't use SOAP formatter for .NET Remoting. This seems like an emerging theme of this PDC. Don't wrote "ever" in horror-film style red writing (the pen tool in PowerPoint) after this point. Remoting is about the .NET type system and not interoperability.  It's funny that in this sesssion we're being told about the limitations of Remoting, while upstairs Rocky Lhakota is talking about how to do it.
• Don't marshal object references in ES/COM+. A no brainer.
• Use RPC/encoded in WSDL. See Yasser's article RPC/Literal and Freedom of Choice on MSDN for the rationale (also see my post to the WSE group on why rpc/literal is not supported in .NET).

WSE is 100% Goodness
Tim showed how WSE provide filters in the input/output filter that serialize/deserialize objects and SOAP headers.  They built a sample project that modified the filters that are applied by default.  He also showed how to use the RequestSoapContext inside the ASMX WebMethod and how to set the client up to call this method using WSE.

WS-Addressing
Don mentions that interop with Tibco was one of the motivations.

WS-Security
5 slides, 20 minutes.  Brain is dead now.

The XML and Web Services Perspective continues with Don, Tim and Martin.  Heavy going after lunch,but much better than the first session.  This session had more useful content that matched the audiences level of understanding (a very advanced audience).

The future of Remoting: SoapFormating SOAP stack
SoapFormatter is dead.  It is the one part of the .NET platform that we'd recall if we were available.  It is the SOAP stack underneath .NET remoting.  .NET remoting works in situations where you have .NET on both sides of the pipe ('living the COM dream without reference counting' as Don say). is one part of the .NET platform that we'd recall if we were able to.

What is SOAP about?
SOAP is primarily about extensibility, it is designed to allow us to evolve services.  The service provider and consumer may evolve at different times.  It is important that the server is able to evolve the service without disrupting the established clients.

SOAP 1.2 will be the last version of the protocol.  Ever.
There's wont ever be another version of the SOAP protocol.  SOAP 1.2 is the last version that we will ever need.  Don justifies this with two arguments.  Firstly, no one gets a third chance at it.  SOAP 1.1 is here and will be with us for a long, long time, we wont get rid of it.  SOAP 1.2 is here and we should move to it but this will happen slowly (like the conversion of IPv4 to IPv6).  It wont be practical to support more versions of SOAP.

SOAP 1.2 is in the next version of .NET, would have been in earlier (Windows 2003) if it was in a final state as a specification.

SOAP is not just for serialized object graphs
SOAP can be used to serialize an object graph, but this is a lifestyle choice, a subject decision.  There's nothing about SOAP that says it has to be like this.

SOAP Headers and SOAP 1.2
SOAP has headers that are extensible.  They are meant for the ultimate receiver rather than any intermediaries.  The headers have the mustUnderstand element which means that it must be processed successfully, as well as an actor tag that specifies which intermediary is designed to process the message.

Because the headers must be processed before the body is processed, most of the SOAP stack implementations buffer the headers and then stream the body.

SOAP 1.2 adds the s:relay attribute which means that the header is targeted for the current intermediary, but that it can be ignored.  However, if it isn't processed it shouldn't be removed from the message - it should be passed on to the next intermediary (presumably changing the actor tag on the way).

There's also a well-known URI 'http://schemas.xmlsoap.org/soap/envelope/next' means the next intermediary should process it.  As Don says, 'everyone plays the role of next - it's the Iunkown of SOAP'.

WSDL
Don made us stand up and recite 'WSDL sounds really fun, please tell us how it works'.  WSDL is more difficult.  Don mentions several times that many of the flexibility points in WSDL were because they weren't sure that Web Services would end up using XML Schema as the type descriptions, and SOAP as the binding.  WSDL provides for other type descriptions and bindings, but XML Schema and SOAP are the most common.

Some very funny stuff when they went through Doc/Literal RPC/encoded choices in the WSDL.  Basically at each attribute Don said 'there is no other way'.  Good laugh, obviously everyone in the room understood that document/literal is the way to go.

Don also joked about the '.NET WSDL parser'.  If you run the WSDL.exe tool before adding the binding then if it's successful there will be a 'no classes generated' message, but at least no exception.

AXM Security and Web Services
Tim did some great debugging of IIS and the directory to solve a permissions problem.  When it failed first time, Don made the joke that it was not a failure but was 'locked down by default'.  Don also made the point that the way the problem was attacked was to change the server until the client message came through (through giving anonymous access to the virtual directory), rather than the better approach which is to give the security credentials to the client (through using the proxy object).

Polymorphic data and serialization
Tim made the point that WSDL generation mandates knowledge of all types at compile time.  If you want to use derived types then you have to use the [XmlInclude] attribute so that the SOAP receiver understands where it must look to see if a type has a derivation somewhere.  The alternative is that the SOAP stack would have to look through all of the referenced assemblies.

Obviously there's a lot of camaraderie between these guys, all having worked at DevelopMentor together in the last millenium. While this adds for a good feeling between audience and presenters there's also an element of pranks and mucking around that sometimes has seemed more fun to the presenters than the audience.

At the most interesting point in the session, where Tim finally got away from the keyboard to talk passionately about Schema, the point was sabotaged by a sniggering Martin and Don typing behind Tim's back on the screen. This was OK, but we didn't get the benefit of hearing the end of Tim's point. I don't mind the presenters having a good time, but not at the expense of the audience.

Note: I missed breakfast this morning which may have contributed to these feelings.

This session was slow-moving in the morning. All of the content has been available on MSDN or shown in MSDN TV episodes (e.g. Passing XML data inside the CLR). However, as a colleague mentioned, the first 2 hours of this talk were the full day of a previous PDC pre-con session, so it shows how things have developed.

Here are some points that stood out:

• The XML stack is built on protocols in the following ways: unicode + uri, XML 1.0, XML Namespaces, XML Infoset, XPath, XML Schema and XSLT/ These map very nicely to System.Text, System.Uri, System.Xml, System.Xml.XPath, System.Xml.XmlSchema, System.Xml.Ssl
• The XmlReader API shows three things that are not part of the XML Infoset specifications:
  - IsEmptyElement (that can tell the difference between <tag></tag> and <tag/>)
  - QuoteChar (returns the quote character used around an attribute, such as single quote ' or double quote ")
  - WhitespaceHandler (gets or sets how the whitespace is handled - an enum that uses All, None or Significant). Tim mentioned this was useful in the MSDN xml work he's been doing.
   
• The XmlWriter is a stack-based approach to writing the XmlDocument.WriteEndDocument will close up any opened elements (stitch the patient up now that the operation is over).
   
• Implementing an XmlReader over data is a difficult job, using it is much easier.
   
• The design of the DOM classes is poor because it was written by people who wrote parsers. Some particularly nasty parts: having to create elements off the document, and having to use ImportNode to bring a document. Don also mentioned that XmlElement.InnerText is a shortcut for XmlDocument.CreateTextNode. It made me feel much, much better to know that Don thinks this was a crap API.
   
• Tim's rules for structuring schema documents: 
  - use BlockDefault="#all" to forbids substitution, extension, restriction
  - use only global elements declarations, so that the one element means the same thing regardless of where it occurs in a document (e.g. <person> means the same thing wherever it is in the XML document). See Ruminations on XML Schema from Don's blog in May.

My appetite has been whetted for Doug Purdy's XmlSerializer presentation 'A tale of two type systems' at the PDC next Tuesday.  I was talking with a customer today about how to version schemas in web services.  It's a problem they haven't addressed yet as everything is still in the first version.

I've seen Doug's TechEd presentation on loosely coupled web services and his MSDN TV episode.  I understand that changing the namespace is not a versioning mechanism - it changes the type system.

Doug's suggestion then was to use the open content model and author version schemas with a version attribute and places in the document where we can put any content.  Then clients can use the switch statement and decide how to handle the message using the highest version number they understand.

What I really miss with this approach is that it loses all of the goodness of schema type checking once the open content elements are used.  It's like you can have schema versioning only for the first version of message, once extra content is added it isn't described or validated.  I like the idea of a schema validation saving me write application code to achieve the same result (or in the case of XML firewalls, having them do the validation).

Radovan argued against this open content approach, saying that subtyping would be better. In his Doug says in his post:

The approaches that I suggest are the best ones using the existing Xml Serialization engine provided in v1 and v1.1 of the Framework. There are ways to "seal" off and extend an open content model in such a way that you don't really need to switch on a version element (although you may want to include the version information in the instance document regardless). But these mechanisms are not supported by the XmlSerializer currently.

I'm really hoping that more schema versioning options will be part of the surprises Doug's got in store in his Tuesday  talk "Indigo": Using XSD, CLR Types, and Serialization in Web Services should be informative.

Here's a good post to the microsoft.web.services.enhancements newsgroup on creating X.509 Certificates with the Open SSL toolkit that can interoperate between WSE and Java.

Open SSL is an open source tool that I've seen used in a couple of production situations and I mean to look into it more.

Understanding what .NET and the C# compiler are doing under the covers can be both useful and interesting.  While on holiday last week I was (geeking out) and re-reading Applied Microsoft .NET Framework Programming by Jeffrey Richter.  What I enjoyed about Richter's book was that he goes down to the Intermediate Language (IL) level to demonstrate the points.  He shows how to use these tools to better understand string handling in .NET and how the switch/case statement works under the covers.

When an assembly is compiled, the compiler examines the code for literal strings and places them into a metadata table.   This table is loaded by the framework as a hashtable (according to Richter) allowing strings to be compared based on hashtable references.  This is obviously much faster than comparing character values in strings.  The C# language compiler uses these techniques to make the switch/case statements more efficient. 

Here's an example that can be worked through using ILDASM and Reflector to make it clear what is happening under the covers.

using System;
public class StringIntern
{
  public static void Main(string[] args)
  {
    Console.WriteLine( 
      new StringIntern().ProcessBlogPost(args[0]) );
  }
  public enum PostAction
  {
    ReadNow, StudyLater, ReadWhenever
  }
  public PostAction ProcessBlogPost( string author )
  {
    switch ( author )
    {
      case "Don Box":
        return PostAction.ReadNow;
      case "Chris Brumme":
        return PostAction.StudyLater;
      default:
        return PostAction.ReadWhenever;
    }
  }
}

To see the string table metatdata that is created when this application is compiled, run the ILDASM tool with the /ADV switch, as follows:

ILDASM /ADV StringIntern.exe

Then choose the View -> MetaInfo -> Show! menu option.  The file that is displayed lists the metadata contained in the assembly.  The 'User Settings' heading shows the string metadata table contents: all of the strings used in the assembly.

User Strings
-------------------------------------------------------
70000001 : ( 7) L"Don Box"
70000011 : (12) L"Chris Brumme"

You can output this information to a file using the following command line:

IDLASM /ADV /METADATA /FILE:outputfile.txt Assembly.exe

This is a technique that Microsoft uses internally to discover code that might be susceptible to a SQL injection attack (where SQL strings are concatenated into SQL statements).

The String.IsInterned method can be used to work out whether a strings is stored in the assembly's metadata table. The method returns null if the string isn't in the metadata table, otherwise it returns a reference to the string (effectively the memory address of this string).  This allows string comparisons to occur with memory references that are much faster than character by character comparisons.

This method is used by the C# compiler to make the switch/case statement more efficient.  Using Reflector (and it's fantastic code IL tool tips that explain each IL instruction when you move the mouse over them) it is possible to see what is happening in more detail.  Here's the pseudo-code for what the C# compiler does with each switch/case statement:

1. load the strings used in the case statements from the metadata table
2. go to the default case if the switch variable was null
3. check whether the switch variable IsInterned, storing the resulting reference
4. compare this reference to the switch string with the references to each of the case statement strings.

public PostAction ProcessBlogPost(string author);
// Load the author parameter into a 'local variable'
L_000c: ldarg.1 
// Go to the default case if the parameter is null
L_000e: stloc.1 
L_000f: brfalse.s L_0032
// Check whether the string parameter is stored in the meta data string table
L_0012: call string.IsInterned
// Store the result (either null or a reference to the string in the metadata table)
L_0017: stloc.1 
L_0018: ldloc.1 
// Get the reference to the string stored in the metadata table
L_0019: ldstr "Don Box"
// Compare the reference to the switch string with the reference to the case string from the metadata table
L_001e: beq.s L_002a
// if they are not the same, repeat with the next case statement.
L_0020: ldloc.1 
L_0021: ldstr "Chris Brumme"

Alan Cooper introduced personas as a user interaction/product design technique a few years ago.  Here's a good definition:

"A persona is a user archetype you can use to help guide decisions about product features, navigation, interactions, and even visual design. By designing for the archetype—whose goals and behavior patterns are well understood—you can satisfy the broader group of people represented by that archetype." 

They are a very effective method of getting developers to focus on a particular person when building software, rather than a class of user (e.g 'How would Steve the crazy trader use this trading screen', instead of 'What features does a trader want?').  Microsoft have been a keen adopter, especially in the core Windows group as this presentation highlights (they go to the point of developer posters and even a live email account for their personas, and rate product features using a feature-persona priority matrix).

What I hadn't realised until recently was that they are using personas in the Visual Studio group.  They are often backed up by more hard-core usability work (such as this presentation BradA mentions on Describing and evaluating API usability at Microsoft). Walter Moise, a contract developer with Microsoft, writes about the personas:

"Every group has personas. In the developer tools, we have Mort, Elvis, and Einstein. These guys are respectively, the typical VB developer, the typical C# developer and the typical MC++ developer. We have a detailed description of them in very specific vivid detail, which include not only the relevant job habits, but also their lifestyles (how they have fun, what they do on weekends). These get very specific, that you wonder if they are talking about a real person.

Mort is your most common developer, who doesn't have a CS background, may even be a recent newcomer, and doesn't quite understand what the computer is doing under the covers, but who writes the dinky IT programs that make businesses run. Elvis, more knowledgeable, cares about code quality, but has a life too. Einstein writes some serious-ass piece of code like device drivers, wants to get things done, needs to be able to go low level and high level, needs a language without restrictions to get his job done."

Mort seems an unfortunate name for the VB.NET developers.  I can't imagine anyone aspiring to be a Mort.

I find it interesting that there isn't much talk of personas within the XP/Agile world.  The agile processes are 'people friendly' but this is mostly if those people are developers.  Admittedly personas may have more impact on consumer shrink-wrapped applications, but I believe that the investment is still worthwhile in enterprise software projects (it's not a big investment for the potential pay-back).  There's still a lot (too much) responsibility placed on the client representative to know what the clients actually want.  Personas seem like an incredibly easy, but powerful tool that would fit well into the XP/Agile arsenal and help ensure that what the developers built is likely to "delight the customer" (to steal Steve Balmer's phrase).

Somehow I'd failed to update my RSS feed for John Bristowe's site so I'd missed the last few months postings.  He's been done a heap of work with WSE.  John's presentations helped me get started with WSE.

The post that most caught my eye was on custom policy assertions.  This is a great piece of functionality that lets developers hook their own XML Security tokens into a WS-Policy xml file.  So you can write statements like 'only let in a SOAP message that has an X509 digital certificate and one of my own XML tokens' in an XML file rather than having to write any custom code (I know I bang on about this point, but it's such a good one).  There's very little documentation (say, none) on this one (I relied on Reflector, the copy and paste button and a judicious set of breakpoints to work out what was going on).  John also comments on using policy to describe roles. 

The observant amongst you may notice that my new site redesign has a resemblance to John's.  I'm hoping that John will be at the PDC so I can buy him a beer.

Why is always more important than what ...Technical people are smart and used to figuring out things on their own. ... What they can't figure out is what was going on in your [the designer's] mind when you wrote a feature, and knowing that [the designer's intention] is the most important part.

As an audience we can use this time before Microsoft open the Kimono to start thinking about what Microsoft's strategies are in each of the areas (OS/dev tools/Web Services) and what design challenges they have to think about.  This helps generate questions so that when we're listening to the endless days of PowerPoint we have a reason to listen and engage.  As Dare has mentioned there's plenty of opportunities to ask the Microsoft developers questions at these conferences (like at Tuesday night's Ask the Experts).  Doing some work before hand to develop understanding and frame some questions is a good investment.

Tim Sneath has already started the questioning, asking will Yukon kill the business tier and put in the database?  Mehran isn't convinced, but it is the dialogue that is important. 

Background to Indigo:

• Read the speculations, and more speculations and  work out what is known from the session outlines

• Try and understand the WS-* standards that Indigo will be based on, especially WS-ReliableMessaging.

• Look at the EAI vendors and work out what problems they are currently solving that Indigo will try and solve.  I suggest the CapeClear's whitepapers site especially the Web Services and EAI paper on Web Services technologies and it's impact on applicaiton integration and Web Services Messaging Strategy.  Jorgen Thelin CapeClear's Chief Scientist has recently joined the Indigo team. 

• Get up to speed on the Architectural buzz-words: Message Oriented Middleware (MOM), Service Oriented Architecture (SOA).  Roger Session's latest ObjectWatch newsletter has a crack at defining What is SOA.  Clemens is a great source of information, as is Ingo's recent architecture briefings: 'SOAP is not a remote procedure call' and one on using other message patterns than just request/response.

• Enterprise Integration Patterns - a site by Gregor Hohpe from ThoughtWorks, describing 65 patterns to do with Enterprise Integration in a vendor-neutral way.  Going through this material should help fill us in on the space that Microsoft are trying to move Indigo into.  For example, since we know Indigo is a message bus, why not find out what a message bus is and how they can be usefully used in projects.

Specific questions I've got based on WSE:

• We know that WSE is a small 'bouquet of flowers' for us as developers, but what have Microsoft learned from WSE that they are going to put into Indigo?
• Will Indigo have the same modular, open approach to its architecture as WSE?  Will it be made of the same 'pliable building materials'?
• Gregor mentions that 'config file hell' is a major problem in Enterprise Integration projects.  Will Indigo be doing anything to address this (like WSE's very helpful Visual Studio Add-In and user tools?

It's a good thing to hear Microsoft people asking what they could do better.  It’s important to note that the current situation isn't necessarily a criticism of anything Microsoft are currently doing.  Blogs are making a big difference to the Microsoft community (both the MS blogs and others), the conferences like the PDC are great, the material on MSDN (MSDN-TV, the full presentations from TechEd 2003), the MSDN newsgroups with responses from the development teams, the patterns and practices group are all great things.

I think Larry nailed it when he says that the Java community has grown organically, without central sponsorship, and that communities involve people communicating with each other.  I think it's a tough ask to say what Microsoft could do itself to overcome these obstacles.

What I like about the Java communities that I've seen so far:

• Open source development plays a large part in this.  The Java developers seem to be comfortable using Open Source technology.  They are comfortable finding and learning about new Open Source tools, or even writing them if they are not available.  I was surprised at the XTC group to find out how few of the Java guys dislike J2EE.  The view I got was that when they need any Enterprise Software they just download an Open Source piece of software that handles the function they need.
• There's more interest in how you develop software, with ideas like XP or Agile Development and the tools: Nant (which MS are finally catching up on with MSBuild), CruiseControl, NUnit and others.
• Here's an excample of the kind of community event that I think it is exciting.  A group of open source developers hold a  weekend in Amsterdam, which from reports sounds really interesting.  Or a weekly GeekNight (admittedly, sponsored by ThoughtWorks, a development services company) where developers get together with their laptops and code.

Here are some issues I think the Microsoft communities  have (based on my very self-focused point of view):

• All of the technology comes out of a central organisation with long release cycles, making it hard to dynamically incorporate new technologies.  Although the PDC is great for 'opening the kimono' it's hard to create a community around this (while the PDC bloggers is good fun, it's also pretty tiring when there's no concrete information about the new technologies). 
• The communities are mostly sponsored by Microsoft.  They focus is on everyone getting closer to the source, rather than establishing a community in the place where they are.  For example, the MVP program and Partner programs seem to be about connecting people up to Microsoft so that they can take that information and push it back out again.  This is another complex point to overcome - 'how can MS create communities without making them look created'
• There's so much good information available that it breeds a kind of laziness amongst developers.  We don't need to go out and learn about it, it will come to us when we need to.  I admit this is a tough criticism to overcome (don't do less, but find ways to encourage developers to engage and do more).
• Microsoft is good at making complex technologies simpler so that they can be accessed by a wider audience.  We've seen it already in areas like relational databases, OLAP technology, distributed transactions, hosted environments and we'll see it again with many of the new technologies like Indigo with its message bus.  The problem is that sometimes Microsoft over-evangalise the wrong message.  This leads to the situation where too many VB projects were using MTS when they weren't getting any benefit from it (or the converse now where COM+ seems to be ignored in favour of 'cooler' new technologies).

So, how could Microsoft improve? Here are some random thoughts:

• Provide more opportunities for developers to speak to each other.  Provide events where the focus is on getting the audience to talk rather than the traditional User Group Expert-Lectures-With-PowerPoint-For-2-Hours.  Perhaps use this strategy adopted by John Bristowe when he ran the .NET group in Melbourne:

"An independently run volunteer user group, MDNug holds informal gatherings once a month where the focus is not on one speaker or subject but on several. To do this, members break into groups gathered around whiteboards. In small groups members are more likely to ask questions, present ideas and professionally critique the work of others, says Bristowe." Source

• Encourage more local dialogues on practices and architecture.  Why not have a 'developer hypothetical' evenings with MCS consultants where a problem is raised and the group (or teams) have to come up with architectures?
• Provide developer drop-in centres - space where teams of developers could come and work on small public projects (e.g. improving DasBlog - why not sponsor Clemens a trip to London to talk to a developer group who would implement new features, similar to Erich Gamma's campaign for Eclipse addin developers)?
• Encourage more open projects.  What about having someone like Eric Gunnerson managing public projects for development tool add-ins.  Things that the MS team didn't have time to do, but were requested by customers?
• Encourage more public discussion about the direction of future tools.  So not just about the bugs in the product today, but the future directions and the advantages/disadvantages of various approaches?
• Provide more opportunities to talk with MCS developers about architecture and approaches.
• As I was writing this John Porcaro a Microsoft Marketing guy posted about this issue, saying its about:

"letting the customers--the ones who just love your products or services--do it for you. It's about giving them something to talk about and making it easy for them to do it (even rewarding). It's giving them permission and rewards and social standing (the MVP program). It's removing barriers and maybe just as important not standing in their way."

These are just my thoughts, I'm interested in what others have to say (especially on how things could be improved, this is the tough one).

Continuing the theme of what the .NET community can benefit from the Java community comes news there will be a version of IntelliJ for C#.  IntelliJ is regarded as the best Java IDE by many people. Here's Martin Fowler's response:

"I know lots of ThoughtWorkers who are dying for this tool. While Visual Studio has a lot of nice features, the programming capabilities seem primitive when you've used a tool like IntelliJ or Eclipse. Of course, I find I really miss the refactoring capabilities, but there's much more than just the refactoring that makes IntelliJ such a stunning tool. Having a tool like this as a plug in to Visual Studio will make .NET programming much better."

I also noticed today that Martin Fowler is coming to the PDC to be part of an architecture discussion panel on What is Service-Oriented Analysis and Design?

Larry O'Brien suggests that Microsoft and Microsoft developers look at what they can learn from the Java community.  The key point for me is when Microsoft will support proper refactoring tools within their development environment.  Martin Fowler, who wrote the first major book on the topic, defines Refactoring as:

“…the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure.”

When combined with the practice of Test Driven Development (supported by automated testing frameworks like NUnit) Refactoring is an essential tool in any developers tool kit.  If you take the position that software development is a craft then Refactoring is like a painter discovering a new type of brush, or a carpenter discovering a new type of hammer.

The Eclipse project, an IBM backed project to develop a new IDE, they have superb support for refactoring, as shown on their help page.  While there are products available for .NET such as C# Refactory, they are still too buggy to use reliably.  I'm hoping that we'll be seeing the addition of Refactoring to Whidbey at the PDC.  Actually, after Googling on it I found that Alan Dean, fellow UK-based Microsoft Developer and PDC attendee has done the hard work of reading the PDC abstracts which state:

Visual C# "Whidbey" includes improvements to the code editor and debugger that cater to the code-focused needs of the C# developer. With support for refactoring in the code editor, advanced visualizations in the debugger, and more, Visual C# "Whidbey" supplements its modern syntax and component-oriented features with new and powerful productivity-enhancing IDE features. Source

The other point that I enjoyed Larry's article was that Java and Open Source have a much better sense of community:

With GotDotNet and other sites in "The .NET Code Wise Community," Microsoft’s presence is palpable and can be somewhat stifling, like a sanctioned school event where the principal is sitting in a corner reading a magazine.

This is definitely something I notice when visiting the Extreme Tuesday Club in comparison to the Microsoft User Groups.

Good to see that Hervey Wilson the Development Lead for WSE has a blog.  Hervey helped me out a couple of times on my last project to implement WSE in a multinational bank.  He's a great developer who really cares about how people are using his product.

Lest I be accused of falling for the 'link to a Microsoft person and expect everyone to know and care who they are' trap that Cameron Purdy spelt out, here's some interesting content that Hervey's already mentioned on his blog:

• A design goal of WSE was to open up the product so that it was easy for other people to open it up and use it in different ways.  He higlights the x509 certificate support that can be used by any project that needs a  managed object model to access the Certificate Stores regardless of whether they need Web Services or not.
• WSE can use this open approach because it has a short maintenance cycle and can have breaking changes across releases.  Herevey mentions other products (I'm thinking Indigo) have to think harder about what goes in the product.
• Hevery links to an article on David Stutz's site (ex Microsoft developer, including Rotor) about software as pliable building material (sounds like a metaphor for Service Oriented Architecture), where David talks about Indigo as part of the 'wave of network integration standards'
• He's been working on making WSE interoperate with other vendors against the OASIS Web Services Security scenarios.  Interestingly he notes that other vendors were pretty lacking (I can believe this - my recent project had a Java team also working on Web Services Security and the client went through a terrible time trying to drag a major Java vendor through the WS-Security specs (they ignored the 'mustUnderstand' SOAP header for goodness sake!)).  I hope Microsoft are helping OASIS push the other vendors to support the standard.  The WS-* standards are only useful to the extent that all vendors adopt and implement them.
• WSE is already being used by customers to handle transactions worth millions of dollars.

Sorry to blog about blogging, but here's a humorous rant from Cameron Purdy on some annoyances in .NET and Java blogs. Some highlights:

• There's too much terminiology for the sake of it which creates an artificially high barrier to entery.  Chris argues the terminology "is all a bunch of stupid made-up acronyms designed to keep 'new' people out by scaring the sh*t out of them".
• "half of [the .NET blogs] are by Microsoft employees trying to make it appear that there is a .NET community"
• "I despise the implication that I should be able to keep track of everyone on a first name basis that currently or has ever or may ever work at Microsoft or with Microsoft products, and further that I should be able to remember past, present and future project names that have been picked from a plethora of idiotic bags such as "river names," "mountain names," "city names," "country names," "old girlfriend names," etc"  Actually, here's a list of the Microsoft code names for those who are trying to keep up.
• a very funny spoof of Don Box's blogging style (here's a thoughtful discussion of the reasons behind Don's style I found by Stu Charlton, who's blog has many interesting comments on .NET community, SOA Architectures and software architectures amongst other topics)

Here are the presentation slides from my Test First Programming with .NET talk at the .NET London user group.  As I finished the talk, where I was demonstrating the Nunit and the NUnit Addin when the organiser said the guy who wrote it, Jamie Cansdale, was in the audience.  Talk about the choir preaching to the minister!  Luckily Jamie is a really nice guy.

We had a chat at the pub afterwards where Jamie said he understood my concern that the Nunit addin didn't have the 'green bar' of the original GUI.  He lamented that his girlfriend didn't really understand the work he did (a common problem in development), but that she did understand the green bar.  In my talk I shared that both my wife and mum know about the green bar.  I compare its affect to the sound of a bell to a Pavlovian dog. Jamie said he'd had to switch back to the GUI after the addin broke when Nunit was upgraded, and it made him realise he missed the green bar as well!

The good news is that Jamie's working on a way to provide a view of the tests inside Visual Studio, so as part of these changes we may see the green bar appearing in the Addin.

Just as I was singing Ingo's praises in the last posting my SharpReader icon turned yellow with news of Ingo's a new MSDN article on role-based security with WSE 2.0.  The article is mostly on using X.509 tokens together with roles and policy files.   The latest project I worked on used the WSE custom token managers to authenticate a SAML token as well as a custom XML token (a substitute for the ASP.NET Session Id HTTP Header, but for web services).  However, I wasn't sure whether you could use the same technique for X.509 Certificates as this seemed to be handled automatically by the WSE framework. 

The solution Ingo demonstrates is to derive a class from WSE's X509SecurityTokenManager and override it's AuthenticateToken method, calling the WSE implementation after doing any custom code, as follows:

public class X509RoleBasedSecurityTokenManager:
  X509SecurityTokenManager
{
 protected override void AuthenticateToken(X509SecurityToken token)
 {
    base.AuthenticateToken(token)
    // do some custom work, like setting the token.Principal
 }
}

Once a TokenManager's authenticate (for binary tokens) or validate (for Username tokens) method has been fired (it's hooked up using the AppDomain's config file, usually the web.config file, or using WSE Visual Studio add-in) then the policy file is applied.  This allows you to make declarations in the policy file about the roles the authenticated/validated user must be in to access the method, without having to write any code on the [WebMethod].  Ingo uses the WSE Policy Editor tool rather than writing the Policy XML by hand (good choice), but just to make it real, the policy file would have the following:

<wsp:Policy wsu:Id="CertificateRoles" xmlns:wsp="http://schemas.xmlsoap.org/ws/2002/12/policy">
  <wsse:Integrity wsp:Usage="wsp:Required" xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext">
    <wsse:TokenInfo>
      <SecurityToken xmlns="http://schemas.xmlsoap.org/ws/2002/12/secext">
        <wsse:TokenType>wsse:X509v3</wsse:TokenType>
        <wsse:Claims>
          <wse:Role value="Accountant" xmlns:wse="http://schemas.microsoft.com/wse/2003/06/Policy" />
        </wsse:Claims>
      </SecurityToken>
    </wsse:TokenInfo>
    <wsse:MessageParts Dialect="http://schemas.xmlsoap.org/2002/12/wsse#part">wsp:Body()</wsse:MessageParts>
  </wsse:Integrity>
</wsp:Policy>

This snippet defines a policy that says a soap request must contain an X.509 certificate security token that plays the role of Accountant in this application and that the soap:body of the request must be digitally signed with this certificate.

As I've mentioned before, this is is a great idea as it separates the code from the security settings. 

In the article, Ingo maps from the incoming certificate to some server-side certificate-role mappings to look up the roles that the user (represented by the certificate) is authorised to play in the application.  In his case he stores mappings between the certificate thumbprint (think Certificate Id) and roles in the configuration file.   I was thinking that it would be better to carry the role information inside the XML that represents the X509 binary security token (for example, SAML tokens have a collection of attributes that can be used for this value).  Perhaps it is better that each application map the certificate to local application roles as Ingo demonstrates rather than carry these with the tokens.  If the roles are defined with the tokens it means the issuer/creator of the tokens has to understand what roles the application's roles, which might be too tight a coupling between the token issuer and the application.

This was another good article from Ingo (he's certainly got his finger on the pulse.  I wonder if WSE and Service Oriented Architectures are part of his new book?).  It's great to see more articles on using WSE 2.0.  I can't wait for the other articles that Matt Powell mentioned.

One of the questions I've had with .NET is where has the COM Scripting Control gone.  Tonight Ingo Rammer helped me see the answer - there's no scripting control, but there is the ability to dynamically compile code.  So instead of using VBScript or JavaScript back in the land of COM, it's not possible to use and .NET language (VB.NET, JScript.NET) to script the application.  This is both a plus and a minus (it's more powerful, but also hard to learn to user).

Check out Ingo's presentation on Extensible Application with Scripting, CodeDom and Reflection.

if(this.Request.UrlReferrer != null)
{
    //Use the property's value.
}

Using a decompilation tool it's clear that the UrlReferrer property is doing some lazy evaluation of a URI member variable that represents the referrer.  The problem is that the .NET code is checking for the wrong type of exception.  If the URI object experience a problem in its constructor (such as a null or empty referrer value) then the documentations states that a NullArgumentException or a UriFormatException will be thrown.  However, the actual code only catches an HttpException:

try
{
   if (text1.IndexOf("://") >= 0)
   {
      this._referrer = new Uri(text1);
   }
   else
   {
      this._referrer = new Uri(this.Url, text1);
   }
}
catch (HttpException)
{
   this._referrer = null;
}