I'm currently pondering whether the new server-side capability of Excel 2007 could radically change the face of many financial application.  Excel is so widely used in finance that many companies could describe it as their platform.  I'm currently involved in writing a web application for a group who use Excel extensively and I frequently think if we could work more harminously with this spreadsheet we could build the application for much lower cost but equal or higher business value.

In the keynote last night they demonstrated the Windows Compute Cluster Server 2003 running Excel Services to perform Monte Carlo simulations using Microsoft Excel.  Initially I thought this was implausible, imagining the response if someone asked, "Can we set up a cluster server in order to run my Excel spreadsheets?", but on reflection I think this could make a great deal of sense.  Even though I'm suspicious about the efficiency of this kind of solution, viewed from a different perspective I think it could represent great business value. 

It's amazing to think that Windows and Office could lead to a scenario where a front-office trader could write software that can be executed in a cluster - an arena that previously seemed the domain of C++ and hard-core developers.  On the one hand it scares me, since there will be less custom C# development for me to do, but on the other hand I like it a lot since re-writing functionality that Excel already implements takes a long time and doesn't give as big a "bang for the buck".

Obviously I need to do a lot more research on Excel Services.  There's a good introduction here and some more technical details on how you can call Excel through web services.

I did a presentation on Indigo to the VBUG London group last night.  The turn out was pretty good, although it was obvious that many English football fans were missing (though they probably would have enjoyed the User Group more than the humiliating defeat).

Off the top of my head, here are the things that keep standing out for me about Indigo:

• The power of separating choices about the binding from the code that a developer writes.  Being able to switch from the tcp with a binary encoding, to reliable messaging over HTTP, to MSMQ just by changing the configuration file is great.
• The simplicity of the programming model.  The majority of the functionality that required Enterprise Services, such as Instancing and Sessions, are available using attributes on either an interface/class or a method such as ServiceBehavior or OperationBehavior.
• The fact that the details of the protocols are hidden under the covers.  Instead of focussing on the WS-* protocols, the focus is on the functionality required.  This has taken me a while to get my head around after having used WSE, which is powerful, but involved having to understand a lot of the details of the underlying protocols.

I'm looking forwad to some of the Level 400 sessions at the PDC, such as Kenny Woolf's presentation on Channels.

As Mehran noticed, I couldn't help but show off the Refactor! tool that's shipping for free in Visual Basic 2005.  This was in response to an attendee who believed that Visual Studio was 'a multi-megabyte bloated version of notepad'. 

Strong bodily reactions are a great way to measure the impact of new technologies.  While many US technical conference crowds like clapping when they see new features demoed, I think the best reaction is a more modest bodily reaction. I'll never forget the time I shivered after seeing Eric Gamma demonstrate the Refactoring support in Eclipse a couple of years ago.  One area of .NET 2.0 that is generating strong bodily feedback is ASP.NET 2.0's support for Custom Build Providers.

First there was Fritz dropping his jaw when he created a custom build provider that took his custom XML metadata file which he dropped into the app_code directory that automatically generated the strongly-typed classes for him.  Tonight I discovered that Kirk Allen Evans had a similarly-sized bodily reaction when he created a Custom Build Provider that automatically created XmlSerializer classes that allows you to drop a schema into the app_code directory and automatically generate the XmlSerializer class.

I'm with these guys: Custom Build Providers are cool (bodily reaction: light goosebumps).  I'm passionate about finding ways of driving application development through the use of metadata files and the Custom Build Providers are an innovative way that ASP.NET provides to support this kind of development.  The only downside I can see is that the creation of these classes is slightly 'automagical' and requires that developers understand what the framework is providing under the covers.  However, I think this initial learning curve is more than compensated for by the productivity that Custom Build providers enable.

Kirk Allen Evans has an interesting post about proposing a distributed architecture for a client where he finds, after doing performance testing, that the speed difference between different transports reduces as the amount of work done within each call increases.  Based primarily on this finding he's recommending ASMX web services as a better strategy than using COM+/Enterprise Services for his client.

I remember coming across this arguement at TechEd 2004 in Amsterdam where Christian Weyer spoke about some Thinktecture research which showed that the difference in speed between Web Services and other distributed technologies (remoting, enterprise services/COM+) was significant if the work done in the distributed call was minimal, but once a reasonable amount of work was being done the performance difference became less and less important.  Worrying about the speed of the transport without thinking about the amount of work done in each call is a little bit like worrying how fast a car can rev if you have it in neutral.

Of course, this is all keeping with the long-known guidance from the Indigo team, which is to start with ASMX and only go to other options if you need specific features that they provide.  Richard Turner's recent white paper provides more details.

I'm on the Ask The Experts stand here at TechEd (back tomorrow and Wednesday at 2pm) and the first person I spoke to today asked me how to solve a socket access permission when using a when using a tcp service with WSE 3.0 Tech Preview.  Working through this also answered a second common question people have, which is 'what's the default port number that WSE uses when "soap.tcp://localhost/" is specified as the service address?'.

The exception in questions was:

An attempt was made to access a socket in a way forbidden by its access permissions.

This occurred when I was trying to run the TCPStockService sample application and here are the key lines:

Uri address = new Uri("soap.tcp://localhost/tcpstockservice");
// This starts a TCP-based listener if there isn't one already started.
SoapReceivers.Add(new EndpointReference(address), typeof(StockService));

Digging through reflector it turns out the the constructor of the Microsoft.Web.Services3.Messaging.SoapTcpTransportOptions options sets the defaultPort:

So, 0x1f91 in Hex turns out to be 8081 in decimal, so by default the WSE tcp transport listens on port 8081 if no port is specified.  Now the exception message made more sense, since I had another application listening on port 8081.  Changing the port number, or stopping the process that was listening on port 8081 solved the problem.

Thanks to the MSDN Product Feedback centre I can send a suggestion to improve this error reporting straight through to the product team.

It's been a long time since I last posted.  I've been busy with a mix of presentations, work and life, such as:

• Helping Kalido ship version 8 release 2 of their two core products, the Kalido Dynamic Information Warehouse and the Kalido Master Data Manager. 
• Speaking at the Microsoft UK Architect Council and Architect Forum about Services on the Microsoft Platform today.  You can find the slides on the Connected Systems Architect forum page.
• Presenting to the London .NET User Group on Programming Indigo.
• Facilitating a 'park bench panel' on Smart Client vs Web Development at the Microsoft UK MSDN Roadshow in London.
• Doing plenty of behind-the-scenes or out-in-the-community work with local User Groups.

Some technical topics that have been taking my time:

• We set up CruiseControl.NET in my team at Kalido.  This just rocks.  It monitors source safe for checkins, checks the code out, runs a Nant script (these are great: you can compile the code using a solution file (instead of hassling with the vbc.exe or csc.exe directly) to build the code, then it runs our NUnit tests as well as providing coverage reporting with NCover and even duplicate code checking using Simian.  It all comes together with a tray icon that monitors what's happening on our build server and lets the whole team know if a build fails.  Tracking down the reason for failure is easy since there's a central web page that pulls together the log file from all of the tools (including comments from the source safe checkin box - finally, an easy way to see these comments!).  It's so exciting I'm rambling even while blogging about it.  The key point for me was that it only took one developer a couple of days to set up, even with no previous experience of any of these tools.
• Schema design.  I'm working on a new WSDL interface and have been playing around with various strategies to build the interface most efficiently and effectively.  More on that in future posts.
• Consolas.  Like Scott Hanselman, I love this font.  I've embarrassed myself my gushing about this to colleagues who are less aesthetically inclined than I am.  It's a mono spaced font, which is part of the Longhorn font set, especially designed for reading code on screen.  I've also set it up as the Windows Console Font as well.

I've also been enjoying time with my daughter who's now walking, starting to 'talk' and generally just getting into trouble (I spent last Saturday morning in the accident and emergency section of the local hospital after a walking accident.  The doctor said 'ah, we always see a lot of head injuries from kids who've just learnt to walk').

If you like a free day of training on Microsoft technologies presented directly by developers with experience using technologies then sign up for the DeveloperDeveloperDeveloper day that's being held on Saturday 14 May at Microsoft's Thames Valley Park campus.  Don't wait to sign up - we're 75% full already and based on similar events in the US we're likely to sell out completely.  Microsoft have graciously provided the venue and are handling the registration and logistics, but all of the speakers are independent community developers! The www.developerday.co.uk site has a full overview of the event, the agenda and sessions and the speakers involved.

There are three different tracks with 6 presentations in each.  Here are a sample of some the talks from developers I know that I'm looking forward to:

• Alex Homer and Dave Sussman will be talking on SQL 2005 and another secret topic
• Ian Cooper, who runs the London .NET User Group, on Patterns of Data Access in .NET (here are my notes from a previous version of this talk I heard)
• Phil Winstanley, ASP.NET MVP, will be talking about preparing for ASP.NET 2.0
• Craig Murphy, Jedi Scrum Master and XML Web Services MVP, on Managing Iterative Development with Scrum
• I'll be covering What's new with Web Services in .NET 2.0.

I'm also looking forward to hearing about custom attributes in .NET, refactoring, test driven development, debugging tips and writing custom FxCop rules.

As well as the presentations it's also a great chance to network with other .NET developers.  For instance, I know that Jamie Cansdale is likely to be there, so if you've got any questions/comments for him on his fantastic TestDriven.NET addin there's an opportunity.

A big thanks to Mike Ormond and his team (Mike Pelton who first posted about the event) for providing the venue and logistics support.

I've just landed back in the UK after a three-week round the world holiday to Australia and back via Redmond for the Indigo Software Design Review (SDR) last week.  The SDR gave me a chance to play with a slightly updated Indigo build than the publicly available Indigo Community Technical Preview.  I was going to list the highlights of my week but then I saw that Omri Gazitts, from the Indigo Team, posted a list of his favourite Indigo features that maps pretty closely to what I was going to mention.  My key take-away was that the team have done a great job designing Indigo so that it is feature-rich and easy to use with good extensibility points throughout the system.

While I've been away Mike Taulty (Layers of Indigo, IInputChannel and IRequestChannel, Channel encoding and filtering messages) and William Tay (Message Tracing, Logging and Activity Management and the details of the default Indigo bindings) have both been doing some excellent job of kicking the Indigo tyres and exploring the CTP builds.

I'll write more when I get a fresh VPC image installed with the CTP bits.

As Rebecca Dias mentions, my article ("Why WSE") covering the high-level reasons to use WS-Security has been published on MSDN.  It covers the provides benefits WSE provides, such as end-to-end message-level security, content-based routing, and policy through leveraging the WS-Security, WS-Addressing, and WS-Policy specifications. 

Let Rebecca know what you think of it.

If you want a longer article on the same material I'd recommend fellow-RD and web services enthusiast, William Tay's piece on Solving Real World Business Problems with Web Services Enhancements in .NET

Microsoft Research in Cambridge have released the WSE Policy Advisor for Microsoft Web Services Enhancements (WSE) 2.0.  The Policy Advisor is an an unsupported tool that acts as a security diagnosis tool for WSE2 policy files (think of it as an FxCop for web service security policy files).  It analyses the policy file for common security vulnerabilities, provides a description on the risk and remedial advice.  It can be launched as a stand alone application or from the policy tab of the WSE Settings Visual Studio add in.  If you are intersted in WSE 2.0 and Policy then  download the Policy Advisor and run it against the sample files that ship with WSE 2.0 and send the research team feedback.

I've been a fan of using policy files to secure web services with WSE for a long time.  As Clemens says, authoring a policy file by hand is pushing things too far.  In combination with the WSE Settings add-in the Policy Advisor provides a great service for anyone wanting to understand and apply policy files, without having to get too focused on the XML angle brackets. The help file contains a list of all the problems the Policy Advisor can detect and is an excellent learning resource if you want to learn about the purpose of many of the policy elements.  For example:

This policy accepts messages with unauthenticated or elements. (Alarm)
Risk: The message is authenticated, but authentication does not cover and . Those elements are often used to implement replay protection, and should thus be authenticated. Otherwise, an attacker may intercept a message and generate a series of slightly different messages that will be accepted as distinct, genuine messages from the original sender. (The risk may be mitigated if the transport provides integrity protection, or if the recipient implements replay protection using other authenticated elements.)
Advice: Insert wse:Timestamp() and wsp:Header(wsa:MessageID) in the element in the assertion.

It also has warnings about the evil that using unencrypted UsernameTokens, though I'd highlight Keith Brown's excellent MSDN article on Securing the Username Tokens with WSE 2.0 as the best source of guidance in this area.

Here's how the tool integrates with the WSE Settings Visual Studio addin:

Below is a screenshot of the report that the Policy Advisor produces, in this case it is reporting against the secure conversation sample that ships with WSE 2.0.  The top part of the window describes the report, the bottom tree view highlights all of the issues found and the relevant policy for each problem.

Mark Fussell, the WSE Program Manager posts that since the MTOM specification has moved to a full recommendation it means that it can be implemented in WSE 3.0.  This is great as it means that we'll have a way of solving the problem of how to send binary data in an efficient, interoperable way that plays nicely with the other WS-* specs .

John Lam, together with Dominic Baier, shows how to implement a hex encoded SHA1 hash of a password plus a scope URI, as suggested by Keith Brown an improvement over WSE's default password handling for UserNameTokens.  Nice work.

In a recent Opinari email (I can't find it on his website), David Chappell defends his belief reuse is one the fundamental benefits of Service-Oriented Architecture and that SOA will be more successful than previous approaches such as object-oriented or component-oriented development.  His rationale is that services operate at a higher level of reuse than binary classes, and they solve many of the human problems that prevent reuse, such as having to have one single view of business entities and forcing reuse through 'draconian' human policies.

Here's my summary of his three major arguments:

1. Service orientation will lead to better reuse of business-oriented software because it is about sharing application functionality at a higher level than binary classes:
   • Evidence for this comes from the fact that the most successful examples of business reuse comes from packaged applications such as SAP and PeopleSoft
   • Attempts to share business classes, such as San Francisco, haven't been successful.
2. Previous approaches to re-use have failed due to human reasons rather than technology:
   • It is hard to agree on a definition of a business entity, such as customer. 
   • Even if there is agreement about the 'one right view' then it is likely to change over time.
   • It is difficult to force developers to reuse software - given a choice (or allowed to get away with it) most developers will reinvent rather than re-use.  Where reuse has been achieved with previous technologies, it was mostly through 'coercive management structures'
3. Service-orientation overcomes some of the human issues involved with reuse because:
   • Services don't require a single definition of business entity, just conformance with a particular schema. A service-consuming application may have a different view of a business entity as long as it can map to the service's view of that business entity.
   • Services can better handle changes over time.  Service-boundaries are a better way of insulating consuming applications from changes to the internal implementation than base classes and binary inheritance as there is a cleaner separation of external interface from internal implementation.
   • Since services are more explicit with their boundaries, and often these boundaries are physical, it is harder for developers to find ways around them.  This is similar to Clemens' point that well designed services do not share databases.  David's argument is that these design decisions make it easier to enforce reuse.

I like his focus on the human problems that plague reuse.  I also agree that sharing schema is a better approach than sharing class (part of the four tenets of service-orientation), and that using interfaces rather than inheritance leads to more flexibility to cope with change over time (a well-known principle).   I find the argument that service-orientation will make re-use easier to enforce less persuasive, since this seems more to do with deployment than the design philosophy (a component-oriented application could also reuse components that encapsulated a private database).

In an interview with Mary Joe Foley, Joel Spolsky says that Indigo will make building applications easier, but it won't lead to a whole new class of applications.

If you look at the different aspects [of Longhorn], like Indigo, I'm not really even interested. It's just a big communications architecture that makes it easier for programmers to build communications things. But there's no application you can't build right now because you don't have a good communications architecture. It might be harder (without it), but it's not going to enable you to build a whole new class of applications.

I think there is some truth in this statement.  As Steve Swartz has said, Indigo is a replacement of the infrastructure and an evolution of the programming model, through simplification and a unification.   As Clemens has demonstrated that you can use existing Microsoft technologies (ASMX, MSMQ, Enterprise Services, and Remoting) to do many interesting things today.  If you need to build a service oriented application it’s worth starting today and not holding off until Indigo arrives.  However, I think Joel underplays the benefits of making the programming model for service oriented applications simpler. 

The problem with building service oriented applications today is that it takes talented developers, like Clemens to do them.  As Don has mentioned, one of Indigo's goals is to make things so easy that he will have to join MS to survive. 

The applications that are developed on Indigo may not be ‘a whole new class of applications’ (though I think it’s highly likely that the killer apps of the future will be based on top of communication technologies), but the fact they are easier to develop them should lead to more of these types of applications being developed.  This is in the same way that VB enabled an increased number of COM applications that could have been built with C++.  It’s an interesting argument as to whether VB lead to ‘a whole new class of applications’ being developed.

There are several questions that I still have about the impact of Indigo:

• Where will the business imperative to build applications that communicate more come from?  Is it that there is a demand today but businesses aren’t developing the applications because of the technical cost? 
• Will the increased ease of development encourage businesses to build applications that they haven’t considered building today?
• Will the pool of distributed application developers increase, or will Indigo just be adopted by the developers who are already building distributed applications?
• Will the increased simplicity of Indigo mean that the architectural concepts behind service orientation will be more widely used?  For example, I’m puzzled as to why MSMQ doesn’t get more love – message queuing is an important architectural design and MSMQ works very well with a very simple programming API – but I don’t see it used as widely as I think it could be.

Thoughts? Comments?

I was suprised to see COM's IUknown interface makes an appearance in the WS-Transfer specification.  If you look at the second example the WS-Addressing MessageID is 00000000-0000-0000-C000-000000000046 the GUID for IUknown. 

Here are the code samples I used my webcast on WSE Messaging last Monday.  The webcast will be available for download from this link soon [update: it's available by clicking on the register button and getting through the login page].

Here’s the code from the demonstrations.  The major sample – a suggestion service – is based around Keith Ballinger’s talk at TechEd San Diego and my version of this talk at TechEd The sample shows:

• A Windows Form client that sends a message to an ASMX web service.
• The webservice logs the message to a file, before sending a one-way message onto another Manager service using WSE’s support for sending SOAP messages over TCP
• The Manager Service is hosted in a Windows Form client that receives the message using the WSE ISoapInputChannel and its in-memory queue.  This means the messages aren’t displayed until the manager explicitly requests them.  The Manager Service also acts as a publisher of these messages to any service that has subscribed.  The implementation shows a simple Publish/Subscribe model using SoapServer/SoapClient.  When the messages are retrieved from the memory queue they are published to all of the subscribers.
• The Boss Eventing Service is the final Windows Form client.  It sends a Subscription message to the Manager Service and receives notifications when the Manager Service retrieves a new message.

Other resources I mentioned/recommend:

• I thoroughly recommend Aaron Skonnard's Hands On Lab on WSE Messaging (available on this page).  It provides some great walk-throughs showing all of the key features of WSE messaging (including how to build your own lightweight chat app using SOAP over TCP).
• Simon Horrell's MSDN article on WSE and messaging.
• For a full-blown WS-Eventing implementation, check out John Bristowe and Co's Plumbwork Orange project on GotDotNet.

Simon Guest lists his top 10 tips for web services interoperability between .NET and IBM WebSphere and BEA WebLogic [via Christian Weyer who comments that things will points will go away with .NET 2.0, see his series 'The web service empire strikes back' for more details]

If you'd like to understand more about how to do messaging in WSE and would like to see the  three levels of messaging within WSE in action then why not register for my first MSDN webcast here?.  I'm going to be presenting it on Mon 9 Aug at 1PM EST (GMT -8). 

I'll be covering some of the demos from the Keith Ballinger presentation from TechEd San Diego that I presented in Amsterdam, along with some new material.  Here's the abstract:

Using messaging systems to support application functionality allows technical solutions to better match business problems.  WSE 2.0 provides messaging implementations that range from low-level explicit messaging through to high-level, more transparent models.  This webcast demonstrates various messaging levels within WSE including a custom WS-Eventing sample and how message, network addresses, intermediaries and queues need to become first-class citizens of your application. Learn how messaging can enable you to create powerful web services applications that cross machine and network boundaries.

I've put these details on my updated presentations page as well.

private ISoapInputChannel channel = null;
private void Form1_Load(object sender, System.EventArgs e)
{
   // Get our channel that's listening for incoming messages
   channel = SoapTransport.StaticGetInputChannel(
      new Uri("soap.tcp://localhost:8088/admin"),
      SoapChannelCapabilities.ActivelyListening);
}

private void button1_Click(object sender, System.EventArgs e)
{
   // When we're ready, process the messages off the in-memory queue
   SoapEnvelope message = channel.Receive();
   String messageBody =
      message.GetBodyObject(typeof(String)) as String;
   this.listBox1.Items.Add( messageBody );
}

In the Form_Load event the channel is retrieved from the SoapTransport.StaticGetInputChannel method, based on the channel listening on a particular network address (in this case using tcp) with particular capabilities (the options are ActivitelyListening, meaning the channel should create a listener at that address, or None, meaning the channel should connect to an existing listener).  As messages are received they are placed in an in-memory queue managed by this channel.  When the Button_Click event is fired the next message can be retrieved from the channel by calling the Receive() method which returns a SoapEnvelope from which the body can be unpacked.

Clemens' session on his ProseWare application at TechEd Amsterdam last week was one of the best conference sessions I've seen.  Proseware is "an industrial-strength, robust, service-oriented example application that newtelligence has designed and implemented for Microsoft".  The application clearly demonstrates how to go about building services today with currently shipping technology, reinforcing that there's no need to wait for Indigo to start building service oriented apps!  I'm hoping that we see a public release of these bits soon on MSDN.

Points that grabbed me:

• Guidance on where to use messaging patterns: Use the OneWay pattern where there is no intelligent immediate reply or no reply is needed, use Request Response pattern where a message asks a question that can be answered immediately (in under a second) and use the Duplex pattern when a message asks a question that can be answered later as the service has capacity (e.g. anything that takes over a second).
• Clemens showed how to achieving 'near enough' reliable web services with HTTP and services that use MSMQ transactional queues behind the service interface.  If there were any problems placing the message onto the queue then an exception would be returned inside a SOAP fault response.  He optimised this further by using a void response type on the web service method, even though it was not marked as OneWay, so that if there were no problems placing the message on the queue then the web service response message would be small.
• ProseWare is based around a repository of XML schema files which he uses to dynamically generated the 'message' classes in the application using pre-build steps.
• All of the service projects shared these schemas rather than having any project references linking to the binaries (services are autonomous).
• The pre-build step inserts ISerializable attributes onto the message classes so that they can work with Remoting.
• These message classes are used as the only input parameter to all of the public web service methods.  These classes leverage XML Schema's support for allowing any element or any attributes to come through, which is a powerful way of allowing for future extension to the message.  Dare goes through this in one of his previous posts.  The XML Serialization attributes look similar to this:

[System.Xml.Serialization.XmlAnyElementAttribute()]
public System.Xml.XmlElement[] Any;

[System.Xml.Serialization.XmlAnyAttributeAttribute()]
public System.Xml.XmlAttribute[] AnyAttr;

• Clemens showed how to use the properties in COM+ 1.5 in XP/2003 to set the home directory for an application, meaning it is possible to use a .NET config file to store the config. He's blogged about this previously here and here.
• He created his own object pool to create something similar to the ADO connection pooling but for COM+ objects.  He simply pops a component out of the pool and pushes it back when he's done, avoiding the excessive overhead when calling new in a COM+ environment (which has to create the pipeline connections).  He has also blogged about this JIT activation pooling here and here.
• He mentioned how LRPC, which is used under the covers in ES, is the fastest way to go cross-process on a single machine.  In order to get this benefit you need to use the JIT activation pooling in order to avoid having the performance gains wiped out by the cost of creating ServicedComponents (again, something Clemens' previously blogged).
• The nice part was that Clemens had done the hard yards and built an application installer that handled created the SQL Server and Windows user and group accounts.  He even showed where he'd found bugs in the OS and how he'd had to work around them.  Information like this is priceless (well, worth a lot of contracting dollars) if you're ever working in a situation where you need to achieve these outcomes.

David Chappell presented a barn-storming presentation based on the idea that the future is services, that services will be called by business processes and that we need to look for a platform that will manage business processes.  He argues, convincingly that we can't expect App Servers to perform this role.  The answer comes with Business Process Platforms.  Here he positions BizTalk 2004 as the answer and goes so far as to claim that it will be the major product at future TechEd's and that getting close to business processes (through BizTalk) could be a key part of keeping your job as a developer since business processes are much harder to outsource than simple services.  You can also read David's whitepaper on BizTalk 2004.

Application Evolution
There are four waves of applications: Mainframe, client/server, multi-teir and now service oriented.  They share the idea that there is a database at the back end, but the key with services, it that they are designed with the idea they can be consumed by other applications, not just humans.  The reason why we can have services today is that we have web services  - an agreement on soap and other specs - amongst all of the vendors.  It's a huge change that we are at the start of - 4th generation.

Soap is like TCP for applications.  It took years between the start of TCP and its ubiquity: the same will happen with SOAP and web services.  It may be five years away.

To think of SOA as just about soap is folly - the reality is that going forward we will see some apps exposing their services via SOAP, but we will also see other diverse ways.  Not all apps will be SOAP.

Who calls services?
SOA talks a lot about how to build about how to define and build services and miss the point of 'who is going to call these services?'

David proposes three groups: UI (portal, asp, jsp, win forms etc), other services (we will have composite services) and business process (some central business process platform that will manage relationships between services).

The need for a Business Process Platform
Where should we build these business processes?  Is it in an app server such as J2EE container?  No, an app server all by itself is not the right place to build service-oriented service.

We are seeing a new kind of platform designed to support business processes.  In other architecture shifts we've seen new platforms - mainframe to client services produced RAD tools, the shift to tiered apps produced App Servers, and now with services we need to support business processes that drives services.

Requirements of a Business Process Platform
What do we need from a business process platform? Something that manages communications with other applications, business process implementation, scalability, modifiable business rules, process monitoring, tools for working with trading partners, cross-app authenticaiton, human interaction with business processes.

Rules change much faster than processes change - so separate out the rules from the processes.

Business Processes: Your job may depend on them
Business processes are more immune from outsourcing than the services.  So as developers we have to start caring more about business processes.  In five years time we'll need to be closer to the business or move to Bangalore.

BizTalk: A Business Process Platform
David mentioned that he had previously avoided BizTalk because he thought integration was messy, boring and 'on the side'.  However, he thinks it will move to the centre of application development.  If you believe in the move to service orientation then you have to believe that business processes that drive those processes are fundamental, therefore BizTalk will be the centre of it all.  It is about to go mainstream as the service-oriented world becomes a reality.

It's not about B2B and EAI.  These are just subsets of the larger space of business processes. 

BizTalk Engine
This is built fundamentally built around the concept of a message.  It doesn't mean only asynch, you can use RPC, but what is processed here is messages.  The incoming message comes in and is processed by a receive adapter - software that knows how to talk to a service or application (there are lots).  Here is the difference between AppServers.  AppServers just support SOAP, but not the diversity of communication technologies.

The message is processed by a receive pipeline.  It does many things, including converting it to XML. The message then comes into a MessageBox (a SQL Server database) that other engine parts subscribed to (e.g. show me all messages from this organisation).  An orchestration (the BizTalk term for a business process) retrieves the message.  It may publish a response to the MessageBox, then back through a send pipeline and a send adaptor.

BizTalk Adapters
Microsoft provides adaptors for MQSeries, SOAP and SAP.  You can make your own or buy them.

Tools
Platforms need tools.  You can build your own adaptors in the Microsoft.BizTalk.Adapter.Framework namespace (notice, a namespace, demonstrating it is a .NET application).  There's also a pipeline designer, biztalk editor (used to create XSD schemas) and a mapper (mappings and xslt transformations between schemas).

Some customers are simply happy with the mapper.

Process implementation with BizTalk Orchestration
Orchestrations compile into .NET assemblies.  It has simple shapes like if-then-else statements, loop, send, receive and parallel actions.  The process logic is simple and doesn't require a high-powered language (e.g. you don't need C# or VB.NET).  Using a graphical language is a decision that Microsoft and many others have done.

Another advantage of a graphical language is that you can use it to communicate with people that understand the business domain.

This visual language can also be authored in Viso for use by Business Analysts.  This is something that will grow over time.

Orchestrations are another reason for preferring Business Process Platforms over AppServers (which implement processes through lower level code rather than graphical representations).

State
Business processes involve people, so state may need to be maintained for a very long period of time.  So we can't use in-memory.  The Business Process platform needs to manage it.  BizTalk does this by storing state automatically and reloading it if needed.

Scope
Business processes involve transactions over long periods of time.  So BizTalk avoids locks (services shouldn't let others take locks on their data), it uses long running transactions that use compensation.  Biztalk uses scopes to manage transactions that can be atomic or long-running.

Correlation
If I send two purchase orders to a service (ERP app), how can I get the correct invoice response from the service?  You can't use request response synchronous calls because the real world doesn't work that way.  So you put in a GUID in the message, but how would this work if you can't alter the response from the service?  You could match on particular fields.  This is what BizTalk does - you define fields that should be used to match responses. 

Scalability Support
BizTalk host instances enable request to be automatically load balanced across orchestrations and MessageBoxes.

Modifiable Business Rules with the Business Rules Engine
Rules change more rapidly than processes - it makes sense to separate them.  If you are building a business process in BizTalk then you can bake process and rules together in an orchestration.  However, if your process has volatile rules, you can build the process as an orchestration and put the rules in a set of rules defined by the rules engine.  This is worth doing so that you can change and redeploy the rules easier than deploying the orchestration.

BizTalk provides a business rule composer that allows business rules to be expressed in a more natural way.  You define terms (sort of like creating an object model) and then the business process rules (like script glues object models into an app).

Process Monitoring with Health and Activity Tracking
There are two levels of monitoring: technical  and business level.  The Health and Activity monitoring tool (HAT) shows the technical side.  The business level if Business Activity Monitoring (BAM) that shows real-time information about running processes (as separate from Business Intelligence data, which is not real time). BAM is based on views on a tracking application.  Business people can use Excel to do this, developers have interfaces they can use.

The Goal: Business Process Management
There's no agreement yet on what a Business Process platform should have, but we are getting a picture.  It has to communicate with other apps, scale out, support business process implementations, workflow with human beings, modifiable business rules and process monitoring.  BizTalk 2004 is Microsoft's implementation of this.

Why is SOA more well known that BPM (Business Process Management)?  Names are confusing in this case, but there's a change coming to the way we build software.  It implies more than SOAP, it requires service for building business processes.  Biztalk is a founation for building, managing and monitoring business process, in the world today and the service oriented world to come.

Pat Helland's just finished a great presentation on services (a highly polished of the version he gave at the PDC and in other locations available online).  The highlight was him singing 'Mr CIO Guy' - a 'speculative retrospective' based on what the future could look like if the Harvard Business Review's article stating that there was no more competitive advantage in having IT were true - all to the tune of American Pie by Don McLean.  Don Box was on bass and David Chappell was on piano.   It received a standing ovation from the audience (a first for TechEd?).

The video will be available on www.pathelland.com in a couple of days.

Here's the list of all of the Birds of a Feather presentations planned for TechEd Europe.  I'm going to be talking on "Service Orientation - what does it really mean" next Thurs 1 July at 18:15, Room R.  The BoF sessions  are often the best value sessions at TechEd since they are a chance to talk with other practitioners about the practice of software development, rather than simply features of products.  It's worth planning them into your schedule as the demand has been high and the room being used is likely to fill up.

I'm going to be hosting a BoF on "Service Orientation, What does it really mean?".   It will be a chance to go over the great definition debate, to look at what problems SO is trying to solve and how to do it today.  Here's the blurb:

Service Orientation receives much hype, but what does it really mean? Is it always the best approach? does it mean message orientation? is it necessarily tied to web services and XML? how do we architect SOA solutions? How de we partition?

Here's the full BoF schedule (the conference site list is a little out of date):

After writing about how WSE 2.0 can use policy and config files to secure web services with no lines of code, I was thinking about how 'magic' it seemed and had an aha! moment when I realised that this demonstrated the power of the Pipes and Filters pattern or aspect-oriented style approaches.  I believe that these approaches will play an important role in service-oriented applications in future.  Here are some recent quotes that back this up.

In my post about using Policy with WSE to create secure web services with no lines of code, I mentioned that this seemed like a good practical demonstration of an aspect-like approach, similar to the Pipes and Filters patterns from Gregor's book.  The policy filters are hooked into the incoming and outgoing messages pipelines are able to ensure that messages to and from the service conform to a particular policy, including retrieving tokens and signing and encrypting.  This means that the security of the service can be configured outside the service code, making for cleaner implementations.

Harry Pierson mentions that WS-Policy is aspect-like in his interview on TheServerSide.NET:

We do a lot of thinking around aspects in Web services, we just don’t call them aspects Web services, we call the Policy. All of the work around Policy is very aspect-like. If you’re spinning up a Web services with Web services enhancements, the new 2.0 stuff, there’s a Policy that defines, okay, if you want to talk to me, you have to be encrypted and digitally signed ... now I can communicate that to whomever is sending me a message. ... the WS-I engine can actually now say, okay, I’m expecting messages that [are signed and ecnrypted]  so I can actually enforce to make sure that those things have actually occurred before it ever reaches the business logic. So, that’s very aspect-like, and that’s going to be very critical going forward around service oriented architecture

Ted Neward writes about a recent presentation on Shadowfax and mentions it uses the concept of a pipeline of interceptors. He mentions that Shadowfax uses this approach to deliver functionality such as tracing, authorization, duplicate detection, instrumentation, authentication, authorizations and transactions.  Chris Garty started an interesting thread about this on the Shadowfax message board, where it was revealed that the Shadowfax team spent some time Gregor Kiczales (one of the creators of AspectJ).

This same pipeline/interception approach has been used in WSE, ASP.NET, Remoting and COM+.  Indigo will implement the same kind of approach using Channels and ChannelProviders.  I'm going to keep reading around this area to understand more about this approach and where it how it can be used successfully.

Celebrating the launch of the Web Service Enhancements (WSE) 2.0 at Tech·Ed 2004, Benjamin Mitchell and John Bristowe talk about the advanced Web services specifications that it supports, focusing on WS-Security.

I'll be on .NET Rocks! tonight talking about WSE 2.0 and building web services with Microsoft technology today.  I'll be joined by fellow Regional Director and Commonwealth Citizen, John Bristowe.  If you can't listen live, it should be available streaming on Monday.

Update: As Carl has mentioned, the streaming/download version of the show has been posted.

my previous post, Mark Naughton asked an excellent question about how he'd apply WSE 2.0 security to a particular scenario.  The answer highlights how to determine which SecurityToken to use in your environment, how to encrypt a UsernameToken with an X509 certificates with code and policy as well as handling authorization with X509 certificates and determining how to distinguish tokens received by a service.

Martin's scenario

An End User uses a Web-based UI Application (ASP.NET 1.1).
The Web Application talks to a Web Service (ASMX) for data storage and other processing.
The Web Service needs to identify the End User and the "direct" calling application (The Web UI App), since there may be more than one "direct" calling application.
We also want to sign and encrypt the Soap messages in both directions.

Since we're talking about adding security to a web service, we'll need WSE 2.0 installed on all of the calling applications. 

What's the best SecurityToken to use?
The first thing to consider is what SecurityTokens are applicable to the scenario.  Aside from custom xml or binary tokens, the three options that WSE supports out of the box are as follows.

Username Name and Password

• For - easy to construct, no distribution problems, WSE handles automatic Windows authentication and the construction of the SecurityToken.Principal for authorization.
• Against - on their own they are only as secure as the passwords.

X509 Certificate

• For - cryptographically strong, can carry extra information (e.g. certificate name). 
• Against - need to manage the distribution to clients.

Kerberos Ticket

• For - powerful cryptographically, inbuilt WSE support for authentication and creating the SecurityToken.Principal for authorization.
• Against - your application and the service you access must be running on computers joined to a Kerberos realm (e.g. not good over the Internet) unless you implement a custom security token service to issue service tickets.

So any of these tokens can identify the end user or the application - it's a matter of working out which one works best for your situation.  If you can handle the distribution and installation of X509 certificates to all of the calling applications, I'd suggest using them to sign and encrypt the message.  In your scenario, the ASP.NET web server could create a UsernameToken to represent the End User of your web application.  For best security, I'd suggest encrypting the UsernameToken with the X509 certificate (hiding the username and password/password digest).

The code would look something like this:

// ... Assume we have an X509SecurityToken and a UsernameToken, and a reference to the web service called proxy.

// Add both tokens to the SOAP envelope

proxy.RequestSoapContext.Security.Tokens.Add( x509token );

proxy.RequestSoapContext.Security.Tokens.Add( usernameToken );

/* Encrypt the username token with the X509 token.

 When encrypting, WSE looks for XML elements with an Id attribute from the http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd namespace, which the username token uses.

The "#" indicates the Id with this value is local to this message */

proxy.RequestSoapContext.Security.Elements.Add(new EncryptedData( x509token, "#" + usernameToken.Id ));

// Encrypt the message body with the X509 token to ensure no one can read it.

proxy.RequestSoapContext.Security.Elements.Add(new EncryptedData( x509token ));

// Sign the message with the X509 token to ensure its integrity

proxy.RequestSoapContext.Security.Elements.Add(new MessageSignature( x509token ));

You'll also need to decide on what token type to use when sending the signed and encrypted response.  Again, I'd recommend using an X509 certificate for the most cryptographically strong security.  The downside is that you'll need to install the certificate on each of the clients.  If you can't handle this install requirement then you are stuck with UsernameTokens.

Distinguishing the tokens on the service
Using the combination of a UsernameToken and an X509SecurityToken to represent the identity of the end user of the calling application and the identity of the calling application itself makes it easy for the web service to work out which token is which.  The web service has to search through the tokens in the RequestSoapContext.Security.Tokens collection to locate each token.  If you decided to use two username tokens for example, you would need to distinguish them somehow.  For username tokens you could achieve this through the username values, or perhaps by giving them well-know identifiers in their securityToken.Id property.  For X509 certificates you could use the certificate name.

Performing authorization
If you use a UsernameToken encrypted with the X509SecurityToken, and you don't want to send the password as plain text, then you'll need to create your own UsernameTokenManager.  This is responsible for authenticating the user and creating the SecurityToken.Principal object which can be used for authorization.  For the X509SecurityToken you can create a custom X509SecurityTokenManager and in the AuthenticateToken method, after calling the base class's implementation, then create your own generic principal and attach this to the SecurityToken.Principal (Ingo Rammer wrote an excellent article on this last September for MSDN, but it's disappeared.  You can find my notes on it here).  The benefit of doing this is, rather than just testing for the certificate name inside the service code, is that WSE Policy validation input filters inspect the SecurityToken.Principal when verifying the policy assertion.

Wrapping it all up with policy
As I've indicated in previous posts, using policy and configuration to avoid writing security code within your service is an excellent idea.   You could do that in this situation as well, except that signing the username token is a little tricky to indicate in policy (you'd have to hand-craft the policy XML file as this is a little way-off the WSE Security Settings Tool territory).   In a confidentiality assertion you'd need to modify the MessageParts elements in the policy file to indicate the UsernameToken's Id attribute.  I'll leave this as an exercise to the reader (as my daughter is about to wake up again), but Aaron Skonnard shows how you can use XPath 1.0 to achieve this in his excellent article, WS-Policy and WSE 2.0 Assertion Handlers.

My TechEd conference-buddy John Bristowe has a blow-by-blow account of my CTS302 Securing Web Services with WSE 2.0 session at Teched.  Michael Earls has some notes and a couple of photos from the repeat session (which was a little fast because it turned out to be 45 minutes rather than an hour).  Aaron Skonnard mentions my first session in his TechEd trip report on his new PluralSight blog:

Benjamin Mitchell's session on Web services security using WSE was excellent. His was the clearest presentation I've seen on general security concepts along with concrete code examples.

That's going straight to the pool room. 

After covering so many other peoples' talks it feels strange to read coverage of my own talk.

In the last post I showed how it takes only 1 line of code to ensure that a web service client signs all messages with a UsernameToken by creating a send-side policy with the WSE 2.0 Security Settings Tool.  In this post I show the same feat can be achieved with an X509Token without writing a single line of code.  I also show how this functionality powers WSE's support for automatic secure conversation without having to write any code, something that blew me away the first time I saw it.

X509Tokens can be located through Policy and Config
In the last post I covered how the PolicyEnforcementOutputFilter checks the send-side policy when processing output messages through the Pipeline and attempts to find a matching token to fulfil the policy.  In the case of UsernameTokens, this means searching the SoapContext.Security.Tokens collection or looking in the PolicyEnforcementTokenCache (hence the one line of code).  However, with X509Tokens it is possible for WSE to locate the certificate without a single line of code.  The Security Settings Tool allows you to configure which X509 certificate you would like to use and stores an identifier for this key in the policy file.  This information is combined with the the <x509> element in the Microsoft.Web.Services2 config section handler that specifies which certificate store to find the token in.  So the combination of the policy file and the config file gives WSE enough information to find the correct X509 certificate without writing any security-related code within the service.

Policy saves code on the receive-side as well
Policy files can be used to save writing code on the receive-side as well.  On the receive-side the PolicyValidationInputFilter is used to validate that the incoming message meets the assertions defined in the policy file.  The policy file can perform checks such as whether the message is signed and/or encrypted with a specific token type or token as well as whether particular message parts have been signed.  If an incoming message does not satisfy these assertions then a security fault exception is raised before your service code is even executed.  As with send-side policy, the WSE 2.0 Security Settings Tool can help you author this policy, saving you from paying the XML angle bracket tax. 

The samples provided with WSE 2.0 have examples of solutions that rely on code and the same solutions using policy.  Comparing these solutions side-by-side highlights the many benefits of using policy instead of code to perform receive-side validation.  The first is that it keeps your service code much cleaner.  Second, it saves you having to remember to make the same calls at the start of each service.  Third, you can change your security configuration without having to recompile the code.

Putting it all together: automatic secure conversation
The best example I've seen of the power of no-code security through policy and configuration files is the support in WSE 2.0 for automatic secure conversation.  WSE supports the WS-SecureConversation specification that defines a SecurityContextToken that is a fast, light-weight security token that can provide message-level secure communication across multiple calls between a client and a service.  It's fast because it is based on a shared symmetric key, rather than an asymmetric key (which is over 1,000 times slower to process).  WS-SecureConversation builds upon WS-Trust which defines the notion of a Security Token Service that receives RequestSecurityToken messages and returns the issued SecurityContextToken as part of a RequestSecurityTokenResponse message.  WS-SecureConversation uses these mechanisms to request and retrieve the SecurityContextToken.  While all of this may sound a little complicated, it is possible to achieve all of this in WSE using the Security Settings Tool.  Using the ideas presented above, if you use X509Tokens then all of this can be achieved without writing any code.  This is the first demo I showed in my TechEd presentation.

Here's my take on how it performs this magic under the covers (feel free to chime in any time Hervey).  On the send-side, the PolicyEnforcementOutputFilter loads the policy file which specifies that all sent messages must be signed and encrypted with a SecurityContextToken.  I think that WSE makes an assumption that the web service can act as a SecurityTokenService and issue SecurityContextTokens (This is enabled on the service by adding the automaticSecureConversation element to the config file).  So when a SecurityContextToken assertion is found in the policy file WSE loads the SecurityContextTokenManager class and calls the LoadTokenFromSecurityTokenAssertion() method.  This method retrieves the tokens that will be used to sign the request before calling the RequestTokenFromIssuer() method that sends the RequestSecurityToken message and unpacks the SecurityContextToken from the RequestSecurityTokenResponse message sent back from the token issuer (which is often the same location as the service).  The PolicyEnforcmentOutputFilter then uses this SecurityContextToken to sign and encrypt the outgoing messages.

Phew, that certainly was a lot of digging with Reflector.  But it illustrates how powerful policy can be: you can request tokens from a token issuer and use them to sign and encrypt messages without writing a single line of code.  This blew me away the first time I saw it working (I didn't believe it until I saw the wire-level traces).  I pinged John Bristowe and Christian Weyer asking 'how does this work?  It seems like Magic but I know it can't be'.  When I thought about it more I realised that this was a demonstration of the power of the concepts such as aspect oriented programming or the Pipes and Filters pattern from Gregor Hohpe's Enterprise Integration Patterns.  More on this in a future post.

Making more of a good thing: custom policy assertions
As well as using the built-in WS-SecurityPolicy features that WSE enables with its Security Settings Tool, it is also possible to create your own custom policy assertions as John Bristowe has demonstrated.  Aaron Skonnard also has more about custom policy assertions.  WSE has great extensibility hooks that let you write code that uses your own policy assertions, allowing you to write validation code in one location that can be hooked into your service through the config file without having to reference it in your code.

While playing with the WSE Security Settings Wizard I discovered that the generated policy requires a DerivedKeyToken to be used to sign the messages rather than the original security tokens.  This is a good thing, but isn't obvious from the wizard screens.  I thought I'd provide some background on what derived keys are, why they are useful and how to ensure your WSE services use them through code or policy. 

Derived Keys: what are they and why are they useful?
Using a derived key is a good thing as it means a different key is used to sign and/or encrypt each message.  Changing the key each time makes it more difficult to perform a ciphertext-only attack.  The DerivedKey can use many different algorithms to generate the key.  WSE 2.0 uses the algorithm defined in the WS-SecureConversation specification, which creates the derived key by performing a SHA1 hash over the original key (in the case below, a reference to the key), along with the combination of a label and a nonce value (a unique value that's seen only once).  Here's what a DerivedKeyToken based upon a UsernameToken looks like on the wire:

<wssc:DerivedKeyToken wsu:Id="SecurityToken-d06a92f7-990c-43a4-a996-f8f3a359e450" wssc:Algorithm="http://schemas.xmlsoap.org/ws/2004/04/security/sc/dk/p_sha1" xmlns:wssc="http://schemas.xmlsoap.org/ws/2004/04/sc">
   <wsse:SecurityTokenReference>
   <wsse:Reference URI="#SecurityToken-6c059c7a-c748-44b1-b957-4b927dd2a8f3" ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#UsernameToken" />
   </wsse:SecurityTokenReference>
   <wssc:Generation>0</wssc:Generation>
   <wssc:Length>16</wssc:Length>
   <wssc:Label>WS-SecureConversation</wssc:Label>
   <wsse:Nonce>mDRVPaA7343zsrMjD+CatA==</wsse:Nonce>
</wssc:DerivedKeyToken>

See Martin Gudgin's MSDN article on Using WS-Trust and WS-SecureConversation for further information.

Ensuring compliance with the WSE generated policy
To comply with the server-side policy that the WSE Security Settings Wizard generates you can either create a derived token in your code or use a send-side policy.  The advantage of using the send-side policy is that it requires fewer lines of code and provides the potential to change the security configuration in future without having to recompile the code.

Signing a message using a DerivedKeyToken with code
Creating a DerivedKeyToken requires two more lines of code than would be required to sign a message with a standard user name token.  The first extra line creates the DerivedKeyToken and the second line adds it to the RequestSoapContext.Security.Tokens collection so that it will appear on the wire when WSE sends the message.  Here's the code:

// Create the username token and add it to the message headers
UsernameToken usernameToken = new UsernameToken("TechedClient", "TechEd2004!", PasswordOption.SendPlainText );
proxy.RequestSoapContext.Security.Tokens.Add( usernameToken );

// Create a derived token, based on our username token
DerivedKeyToken derivedToken = new DerivedKeyToken( usernameToken );

// Add the derived key token to the message headers
proxy.RequestSoapContext.Security.Tokens.Add( derivedToken );

// Sign the message with the derived key token.
proxy.RequestSoapContext.Security.Elements.Add( new MessageSignature( derivedToken ) );

Signing a message using a DerivedKeyToken with send-side Policy
The simplest way to ensure that a message is sent using a DerivedKeyToken based on a UsernameToken is to use the WSE Security Setting Tool to create a policy file on the client specifying that output messages should be signed with a UsernameToken.  When WSE sends the message through its pipeline the PolicyEnforcementOutputFilter will create a DerivedToken based on the UsernameToken and sign the message with it.  For this to work, WSE needs to know where to find the UsernameToken to base the DerivedKeyToken on.  WSE first searches for a matching token in the SoapContext.Security.Tokens collection and if it doesn't find any there it looks in the PolicyEnforcementSecurityTokenCache. 

So you can either use the first two lines of the sample code above, or the following:

// Create the username token and add it to the message headers
UsernameToken usernameToken = new UsernameToken("TechedClient", "TechEd2004!", PasswordOption.SendPlainText );
PolicyEnforcementSecurityTokenCache.GlobalCache.Add( usernameToken );

The advantage of using the PolicyEnforcementSecurityTokenCache is that the username token can more easily be attached to any message leaving the client, regardless of which service is being used (e.g. it is independent of the webservice the message is being sent to, unlike the SoapContext which is related to a specific webservice address).  The second advantage is that if you later decide to sign with X509SecurityTokens rather than UsernameTokens then you wouldn't need to change any code or recompile (the UsernameToken in the GlobalCache would simply not be used).  I'll write more about this topic in a future post.

As Rebecca Dias notes, I'm repeating my CTS302: Security Web Services with WSE 2.0 talk tomorrow at 12:15 in room 33ABC.  

Yesterday's talk was so crowded that firemarshals shut the door (and many attendees had to touch elbows with the person next to them).  Even Keith Ballinger, the WSE Program Manager, was left out in the corridor!

If you were at the talk yesterday I'd be grateful if you complete the session evaluation form on the conference CommNet.  These evaluations are extremely valuable and all of the feedback is noted.

I'd like to extend the 'being at the conference through blogs' experience and invite any readers to leave questions on this posts that I'll answer in the session and a subsequent post.  What would you like to know about securing web services with WSE 2.0?

When developing with WSE it is often useful to be able to see what is going out on the wire.  Changes in the WSE 2.0 release mean that is no longer possible to use tracing tools such as tcpTrace and MSSoapT.  Christop Schittko has a good post on the background to this problem and shows how to use the inbuilt-WSE trace capabilities to get around it.  There's also another solution, which is to use Mindreef SOAPScope's WebProxy, and as I was writing this I noticed that Mike Taulty has also posted his WSE 2.0 trace tool, which has become my new default favourite.

Background
The previous approach to tracing in WSE was to listen on a one port with a tracing tool, then forward the request onto another port.  WSE 2.0 now checks to make sure that the WS-Addressing To header in the SOAP message matches the address in the HTTP Header.  This means the existing tracing tools will return a fault.

The SOAPScope solution
SOAPScope has implemented it's own version of System.Net.WebProxy that allows proxying to localhost.  It also means that the address in the SOAP envelope matches the address in the HTTP header.  To use it you just need to add the following to the client's config file:

<system.net>
   <defaultProxy>
      <proxy proxyaddress="http://benjaminm:5049" bypassonlocal="false"/>
      <module type="Mindreef.Net.WebProxyEx, MrTools, Version=3.0.0.0, Culture=neutral, PublicKeyToken=90f6595dbbe888f3" />
   </defaultProxy>
</system.net>

Mike Taulty's solution
Mike has written a new WSE 2.0 Trace Tool that uses custom input and output filters into the WSE Pipeline. These filters copy the message into a new SOAP envelop and post it a tracing 'web' service listening on the a tcp port, using the SOAP Messaging (soap.tcp:\\) support within WSE 2.0.  Using it requires adding the following to the config file:

<microsoft.web.services2>
  <filters>
    <output>
      <add type="WSETracingFilter.WSEOutputFilter,WSETracingFilter, Version=1.0.0.0, Culture=Neutral, PublicKeyToken=1c1f2f7177e1ff79" />
    </output>
    <input>
      <add type="WSETracingFilter.WSEInputFilter,WSETracingFilter, Version=1.0.0.0, Culture=Neutral, PublicKeyToken=1c1f2f7177e1ff79" />
    </input>
  </filters>
</microsoft.web.services2>

It's very nice! I'm going to be using it tomorrow in my talk at TechEd.

Don Box and Doug Purdy did a 'keynote' for the Connected Systems Track.  They started out by asking what questions the audience wanted to see.  A great set of questions were proposed and the answers contained some of the most valuable content in the session.  Here are my notes on their answers, and some they didn't get time to do.

How does WSE 2.0 fit in with the Indigo direction?
It lets you use the protocols we have today.  WSE takes your ASMX investment and keeps you in the game as we do this protocol work.  If you don't track the protocols it may not be so important. 

Indigo will be the primary technology for using the WS-* specifications in future.  WSE takes your ASMX investment and lets you add support for those specifications today.

What's the future of DIME?
MTOM.  DIME was an experiment - we were on the wrong track that didn't support security.  Microsoft got together and did PASWA that became MTOM.  It will be in Indigo and other MS technologies.

WS-Security vs. SAML?
There are many different kinds of tokens that may be used, such as Username, X509 certificates and Kerberos tokens.  Don said it was unlikely that a token type, like SAML will become the 'single token format to rule them all'.  No definite answer on where the SAML support was.  As I learnt on Saturday, trying to implement SAML support is a non-trivial exercise - it would be nice if there was a clear statement from Microsoft on when it will be supported in the platform (so that you don't have to share my dll in order for us to use it when we talk).  I think it will be part of the identity management work in future.

How successful is WSE at interop?
Microsoft do bake-offs with WSE where they get all the vendors in a room and try and make the specifications work.  There wasn't a definite answer other than this.

How do you talk SOAP from a Windows Service?
Don's answer was that you do the hard work to host ASMX inside a service then put an ASMX façade and call into the service with ES or Remoting.

I thought this missed the point that the recently released WSE 2.0 supports Soap Messaging, which allows you to implement SOAP messaging over TCP.  I think this would be a far easier way of hosting SOAP within a windows services.

What is the technology to replace COM+ in the long term?
ES investment will keep working.  Deployment, interception and synchronization are being brought forward into Indigo.  Many of the ES semantics are a direct correlation with the Indigo model.  Doug mentioned that ES programmers will be the most prepared to work with Indigo when it ships.

Is there an issue with the verbosity of web services payload?
Don's answer was that Indigo will 'negotiate up' and switch to a faster way of communicating if the other endpoint uses Indigo.  How they do this is to be seen (there were comments at the PDC that the first Indigo call will be a policy request to see if the other end is an Indigo endpoint).  Don mentioned that the industry is having a hard time defining binary protocols that allow user definition are difficult.  Binary protocols that support user defined structures are hard.

How do we discover services and determine policy at run time?
Don mentioned that UDDI was a solution you could use today.  In a show of hands only 4 of 200 attendees were using UDDI (2% adoption?).  According to Don it makes some hapy, but some customers want more.  They want a more flexible model for describing things without having to use the tModel (which is hard to grok).  There also other groups want to discover services on devices, so WS-Discovery is where Microsoft are headed.  It is a small spec that is easy to understand that can be easily implemented.

Will we need to continue to be plumbers to do web services security?
I thought this was a great answer:

For a while.  WSE makes it easier, but if things go wrong you'll need a plumber.  We have not done our job in Indigo if people have to understand the protocols.  Your common developer needs to solve business problems, not the protocol problems.  Indigo is adding value without focussing on the protocols. 

No matter how good WSE does, since we are ironing out the interop you'll still need to read WS-Sec.  Now at least we only need plumbers when things go wrong.

What's the migration path to SO?
This was really the content of Richard Turner's talk in the track, he's also written a great post on detailing prescriptive guidance on preparing to upgrade to Indigo.  The basic message is don't do tricky things.  If you are doing something that was hard to figure out, maybe that was for a reason.  So things like SoapExtensions or custom message sinks in Remoting are not going to upgrade well.  There's lots of material out there that shows that Microsoft have a 'good story' on upgrading from various technologies.

Unanswered questions:

• What is the MSMQ equivalent for COM+?  Is MSMQ going away?
• How does EIF fit into SOA?
• Strategies for native to managed interop?
• Will Indigo support mailslots?
• Are there any application blocks for SOA?
• How do we handle events across appdomains?
• Security - you have authentication and authorization - what about any attacks through the channel - filtering content? Content-awareness in firewalls?

I'm with all of the 'Blue Shirts', speakers and the Microsoft staff, in the keynote overflow room, sharing the experience of watching Steve Ballmer on a video screen.  Here are some key points:

• He's looking trimmer. A gasp of 'Atkins!' went around the room.
• Key messages - do more with less.
• The next 10 years are going to be even greater than the last.
• Only Pfizer spends more than Microsoft on R&D.
• Remember 10 years ago TCP/IP was a separate business to the OS.
• Integration is the key. How many data access layers does Microsoft need.
• How can we narrow down the skillset required to know how to use the products. Integrate to reduce the overhead required to use the platform.
• Windows XP SP2 has taken priority over Longhorn recently.
• It used to be 'features, features, features' now it's 'listen, listen, listen'.
• Watson is one of the biggest advances in computing. Being able to send crash reports to Microsoft means there is a statistical way of rating the issues that users are having.
• Integrated innovation and customer responsiveness to do more with less.
• Security is key focus.
• Spam is too cheap to send - we need to add cost and burden. Using techniques like making the sender prove who they are.
• Interoperability has been a key focus. Microsoft has done more than most people have ever given them credit for in integration. Microsoft is absolutely behind the XML stack as an open standard. The 'best and most important thing to happen to our industry'. It's an 'architected' way of doing interoperability. The old way was writing XML to connect each system.
• Microsoft Office beta web services - allow Office to be a smart end client to web services.
• Becky Dias gets on stage.
• WSE 2.0 is released! Also the Microsoft Office Information Bridge are entering Beta.  Basically web services integrated with Microsoft Office task pane.
• She's clicked on someone's name in Outlook. A task pane has come up with a form that lets her do a stock trade, calling a webservice and gets an ID back again, all without leaving Outlook.
• Demoing policy. Not sure if the audience are getting this. But it's very cool. We don't have to right code anymore. Definitely should have spent more time polishing the WSE Settings Tool wizard screens.
• .NET has more than 50% of the US market. Customers think it is 66% more reliable, 70% think it is faster, 2.7x people think it is more secure.
• The VSIP program has been increased. Oracle and SAP and TibCo will use Visual Studio for their platforms.
• Visual Studio 'Team System' - now trying to do more as part of the software development life cycle. Group development, modelling, testing and deployment.
• It looks like we now get bug tracking within Visual Studio.
• Showing a Whitehorse style screen that diagrams the deployment of the application and can check to ensure that it will work in that environment.  It produces 'build errors' when you compile it.
• Can specify that the system passes build rules, static analysis and unit tests. 
• Finally we have unit tests that are part of the build system (this got a clap!).  Now we know what James Newkirk has been doing at Microsoft!  Rewriting NUnit!
• Also includes code coverage tools as well (another clap)
• There's also a security version of FxCop that is built into 'Visual Studio Team System' based on Microsoft Research's work on Secure Computing Initiative.
• But wait, there's more .... load testing as well (more claps!)
• Back to the Information worker.  Steve has the feeling that SharePoint Team Server, Portal Server, Office and Live Meeting haven't been as well adopted as they should have.
• There will be advances in searching as a result of 'strong competition' (see Google)
• Why choose Microsoft over Linux or Java? More integreated innovation, better responsiveness and trustworthiness, partnerships, choice (more applications, better interoperability).

Overall I was a little disappointed that 'Crazy Steve' didn't make an appearance.  There was no sweat, no ranting, no cheering with the crowd.

There was a panel discussion with John Bristowe, Scott Hanselman, Joe Homnick, Joe Lindsay, Terry Mohn, Ted Neward.  the highlight for me was that there still isn't good tool support, or a good story from Microsoft, on how to manage services once they have been deployed.  Other points:

• Scott mentioned that his company provide services to 30% of all online US Banks and 50% of them rely on web services. So they are out there and being used.
• An excellent point was made there isn't good tool support (or a good story from Microsoft) on how to manage services once they are deployed. The tools to ensure that services are up and running and providing guaranteed levels of services are not here yet.
• There was some discussion about whether we should care about angle brackets or not. Scott made the point that we should focus on Infoset. John Bristowe made the point that it can be useful to understand the specs and know what is happening on the wire.
• Ted mentioned that editing WSDL is too hard and no-one at the event other than Scott had actually done it. Scott made the point that most of us had edited HTML files because we did not like the way FrontPage formatted them, so why hadn't anyone done the same with their WSDL?
• Some good points from the more business-focussed members of the panel that they are more interested in tools making developers productive than in having developers that understand the plumbing in web services.
• Discussion about what the definition of SOA is. Answers included 'components with a longer wire', 'objects with explicit boundaries', 'hooking shit together'.  I had a useful discussion aftewards with Joe Homnick that helped me see that SOA is most useful as a concept when talking with businesses about how to architect projects.  As I've mentioned before I'm still not convinced that SOA is definable, but it is based on top of good architectural concepts such as encapsulation, data hiding, loose-coupling, service discovery, messaging patterns and asynchronous processing.  So if we need to call this SOA in order to get all excited about these topics then it may be a necessary evil.
• I followed with the question 'what problems is SOA trying to solve?' answers included high-level re-use and interoperability.
• Discussion about location transparency not being a good idea. Ted suggested the original idea was really about being able to write code without having to know the location of the components, rather than hiding the fact that this was an expensive cross-network call to a component.
• Discussion about the fact that Amazon uses Web Services well but that XML over HTTP is more popular than their SOAP implementation. There was some discussion about alternatives to SOAP such as REST, but they were seen as more limited compared to SOAP. The key for me is that tool vendors need to implement great tools for SOAP so that it is easy to use as these competing approaches.
• The more business focussed members of the panel made the point that the technology is not the main thing, it's supporting business functions. For them the point of the Interoperability work is that it les them go out and buy products that they know can talk together.

Ted Neward gave the keynote at our WS-Interoperability Day even today.  Here are some of the points he made:

• It's important to learn from the past, in terms of previous distributed computing approaches, when looking at Service Oriented architecture and web services.
• One of the key things that made HTTP a ubiquitous approach to solving interoperability problems was that it was simple.  For me this highlighted the fact that it is essential that good developer programming models be developed over the top of the WS-* standards so that it's simple for developers to take advantage of the functionality without necessarily having to be a plumber. 
• It is important that vendors continue to work together and not start developing their own non-standard features.  Ted mentioned that vendors are currently being driven to ensure compatibility in order to make money from the new approaches, but once a market it established they may start fracturing away by adding their own non-standard implementations.  Dare covered this in his post detailing the SQL standards.
• Ted made the point that Objects work better when they are deployed together because they need to talk together.  Good objects are well factored and make a lot of calls.  This means that the same techniques does not extend well to situations where objects are distributed.
• The fastest call that can be made to a network is around 1000 times slower than a local in-process call.  This can raise to 10,000 times slower when the network is involved.  To illustrate what this means, Ted used the analogy that this was similar to his 20 minute commute to work taking 70 days (his math).
• SOA is just a new way to get developers to listen to the worn-out message that  'Distributed Objects Don't Work'
• Interoperable systems work best when the architecture is built from the centre out.  If you start at the .NET and work towards Java for example it's likely that you'll find you've implemented something that is incompatible.
• The problem with starting from the inside out is that WSDL, which provides the definitions, is hard to work with and you really need a tool to author it.

I flew into San Diego last night where Michele Leroux Bustamante had agreed to meet me at the airport.  The plane was an hour and a half late and I was wondering if she'd still be there or even recognise me.  Turns out I needn't have worried she'd be in the group with the three laptops and the wireless router:

That's Anant Kadiyala, Michele and Heinrich Gantenbein getting in all the preparation time they could before our interop demo.

I'm presenting a TechEd session in the Connected Systems track on how to use Web Services Enhancements 2.0 to secure web services.  TechEd looks like being a mini-WSE festival, with Aaron Skonnard doing the pre-conference on Sunday, Keith "My blog has gone" Ballinger talking up a storm on messaging over multiple machines and networks, plus loads of other applied web services talks.

In my session I'm going to cover all of the security features in WSE using as many code demonstrations as I can fit it.  I'll cover the basics such as security tokens, signing and encrypting before moving on to more advanced topics such as token issuing and establishing secure conversations.  I'll show how WSE 2.0 allows all of this to be done with code as well as policy and configuration files.  The best part is that straight afterwards you'll be able to go and do some Hands On Labs authored by Aaron Skonnard covering these topics.  How's that for in-depth educational experience?

Here's the official description

CTS302 Connected Systems: Using Web Services Enhancements v2.0 (WSE) to Secure Web Services
Wednesday, May 26 2:00 PM- 3:15 PM, Room 10
Speaker(s): Benjamin Mitchell
Track(s): Connected Systems, Developer Tools and Technologies, Security
Web Services are being used to cross application boundaries, especially between enterprises. Such interactions need to be secure. See how to use WSE v2.0 and the security protocols that it implements to secure Web service interactions within and beyond the Trust domain

For those interested in doing some background reading before the event, I strongly recommend WS-Security Drilldown in Web Services Enhancements 2.0 by Don Smith.  Rebecca Dias thinks its the best article she's read on WS-Security and I'd agree.  Also Hervey Wilson's blog mentions some truly magical features that will whet your appetite for what you could see.

Are you confused about how all of the WS-* specifications fit together?  Hard to get your head around the WS-Alphabet soup?   3Leaf have a great Web Services Architecture Timeline graphic that makes it easy to see how the specifications have been created over time [via Ken Brubaker]

Although there are a lot of specifications, they have been built in a composable way to support secure, reliable and transacted web services.

Martin Gudgin points to the Working Group Draft of the WS-I Basic Security Profile.  I believe that the web services security will play a major role in increasing the adoption of web services in future.  It's great to see the WS-I behind a push for greater interoperability in web services security.

Web services security enables message level security, through signing and encrypting parts of a message, so that secure communication can be established even across public networks such as the Internet.  Increasing support and compliance for this security infrastructure across platforms and development languages will lead to even greater adoption of web services.

I'm hoping that they develop test cases that can be used to prove compliance to the standard, similar to those already developed for the WS-I Basic Profile.  In preparations for this Saturday's Web Services Interoperability Day Michele Leroux Bustamante has had to assume the role of 'human interoperability tester'.

Before TechEd San Diego I'm going to be presenting a SAML token issuer sample with Michele Leroux Bustamante as part of the Web Service Interoperability Day this Saturday.  The event is a chance to actually see interoperability happening, rather than just watching PowerPoint slides.  We'll be focussing on showing real code demonstrate WS-Security (now an OASIS standard that will be implemented with the release of WSE 2.0) and WS-Policy.

Getting the demos ready has been an international collaboration.  John Bristowe has been waving the WSE/Policy 'Pom Poms'.  Chris Haddad is doing the Java implementation with OpenSAML.

After the demonstration there's a panel discussion on the state of web services standards that should be very lively.

While it is easy to see the benefits in the idea that services expose explicit boundaries, there is still some distance between this and understanding where service boundaries should be drawn and what it should look like.  A collection or recent posts from three Microsoft Architects highlight that services work best at the business level where they can share the same logical data model and single database engine.  It's nice to see the SOA discussion break through the academic debate to this practical guidance level.

Ramkumar Kothandaraman, a Solutions Architect on the .NET Enterprise Architecture Team, talks about  implementing the service-oriented tenet that services should have a clearly defined boundary when there are issues about searching or updating the shared data.  He proposes that it can be beneficial to share a database between related services and to expose these services as a collection or business services that share the same logical data model, rather than individual services. 

I believe this is similar to Pat Helland's session at the PDC where he spoke about business services encapsulating database-resident data and that each business services' data should live in a single database engine.  If the data can't fit into a single engine then look for ways to split it into two business services but don't have a service span database engines.  If the service needs to scale then consider partitioning the data between individual database servers rather than trying to merge and replicate.

Michael Platt, a UK-based Field Architect from Microsoft, shares the view that services need to be exposed at the business level.  He provides a useful, pragmatic definition of service oriented architecture as a model or abstraction of loosely coupled business-level functionalities or services.  I also enjoyed his reflections on a recent meeting discussing SOA:

There was a lot of discussion about what SOA was: explicit boundary’s, autonomous, etc which reminded me a lot of the COM definitions but with different words. Alas there was no discussion as to what a Service should look like, how big, what level of functionality etc; this was left as an exercise for the designer.

In terms of guidance on how to architect services, he mentions a story of the good old 'component oriented architecture' days where a client had 3000 components in a system, as a warning not to go overboard again.  Michael also invokes his 'First Law' as design guidance for a services which essentially says that that services should be coarse grained because of the huge communication and network stack they have to go through.

I've been blog-lite while I prepared and then gave my Indigo presentation to the London .NET User Group.  The audience seemed to enjoy it and I had a good time presenting it.  Some quick points:

• The selling point of the Indigo Service model is that it is simple and easy to use.  The benefits for developers is that they get all the benefits of enterprise-quality features such as transactions, reliable messaging and security without having to write any code.  Letting Microsoft do the plumbing allows developers more time to write code that focuses on solving complex and interesting business problems.
• I took on board Clemens' experience and started talking about the 'why' of distributed technologies before delving into how Indigo works.  Luckily this group was very switched on.  I got some good questions about the performance of Indigo across AppDomains/Processes/Machines.
• Someone asked me to provide some background on why Enterprise Services are a better choice than Remoting for communicating between components on the local machine.  Clemens' has some more information on it here and here. 

Doing a presentation where you don't have the current version of the software to demo is a challenge (the PDC Indigo bits are from an older, M4 build, which has been substantially refactored in the now-being-coded M5 build, so there's not a lot of value in showing demos with the PDC bit).  In order to get some code samples I had to transcribe screen shots from the PDC videos (difficult since Steve Swartz didn't use word-wrap in Visual Studio).  I'm filled with (even more) respect for Don Box and Steve Swartz's presentation skills after realizing they managed to do four Indigo presentations at the PDC without even compiling, let alone running any of the applications.

Given the PowerPoint dependence and Clemens' reports on the complexity of the topic I decide I'd have to resort to audience bribery.  I drew on my experience in TheatreSports and brought several bags of Minties from the Australia Shop at Covent Garden (to my horror I discovered today that they are made in New Zealand!) and threw them to the audience whenever I detected the signs of PowerPoint-induced lethargy.

John Udell has written an article on WS Alphabet Soup based on his discussions with John Schewuck, an architect on the Indigo team.  Here are the interesting points:

• Indigo may ship before Longhorn to enable people to provide developers with access to the functionality provided by the newer WS-* specs.  I know the message from the PDC was Whidbey < Indigo <= Longhorn, but I guess haven't really considered it will come before Longhorn (I thought this was what WSE was for since Indigo is a 'baked in' part of the .NET Framework).  Not that I'd complain if it came earlier.
• There are a lot of Web Service (WS-*) specifications, but this is a good thing because they are composable.  You only need to implement the specifications you need.  The alternative would be a single specification that tries to do everything.
• The WS-* specs are inventing a lot of what has been provided before in DCOM and CORBA.  As Don Box says, 'The entire motivation for defining SOAP was to get the industry out of the "you need my runtime to talk to me" business that DCOM and Java RMI so happily got us into in the 1990's.'
• The advantage of using SOAP as a message description format  is that it provides extensible SOAP headers ("an accordion-like construct").  Previous network communications protocols was their dependence on fixed-offset dependencies that broke if you had to extend or modify the packets.
• Microsoft is wise to spend time worrying about the usability of its APIs.  This was my initial reaction at the PDC - 'Wow, they have done a lot to make this really easy for developers'.
• The key to reducing the cognitive overload caused by the specifications is an easy programming model (this acknowledges the reality that not everyone gets as excited as John Bristowe about a new specification).  Indigo's use of attributes in the Service Model is a major plus here.  How awesome is it to be able to write code like this to add WS-Security (signing, encryption), WS-ReliableMessaging and WS-AtomicTransaction support:

[Service]
[Security(Disabled=false, 
 SecurityProfileName="MyDynamicProfile",        
 ClientAuthentication=UsageMode.Required,
 IntegritySupport=UsageMode.Required,
 ConfidentialitySupport=UsageMode.Required)]
[ContextPropagation(typeof(ITransaction)]
[Connection(DeliveryAssurances.Full)]
public class MySecureReliableTransactedService
{
    [ServiceMethod]
    [Authorization(Role="Clerk")] 
    public void CreditAccount(string acc, decimal amount)
    {
        …
    }
}

The Loosely Coupled blog also has more commentary on the interview with John.

Matthew Adams left a comment on my last post disagree with my characterisation about services being more to do with messages as opposed to remote procedure calls (RPC).   It was a good question that I thought was worthy of a lengthier response.

What I was suggesting in the original post is that overtime there will continue to be a move towards services that use messages, and in particular SOAP messages based on the XML Infoset, rather than RPC style calls over time.  SOAP because it gives a broader reach for interoperability because of it's lowest common denominator approach and associated standards.  Messaging because it gives more flexibility in terms of what is sent and the message exchange patterns that are supported. 

The Great SOA Definition Debate
I've been trying to get away for getting into a debate about what services are, mainly because I haven't found a useful definition of service that says more than 'it's a great idea to have blocks of code that do something' and when designing these blocks of code, encapsulation and loose coupling are good practices.

By Matthew's definition a service is something that can do something, at a particular addressable location on a particular protocol, with a particular message format, in a such a way as the consumer doesn't have to care about how that something is done.

Using this definition of a 'service' it is true that the service doesn't have to be implemented as a 'message' end-point.  It is possible to implement these 'services' in ASMX web services, in-process .NET classes and remotable objects.  The problem is that these choices can impact the amount that the consumer has to know about the way the service is called, if not how it is implemented.

Don's Definition of Services
I like Don Box's definition of a service as a 'program that you communicate with using messages'.  These messages are SOAP messages based on the XML Infoset (a logical view of an XML document - angle brackets and text representations may or may not be involved).

The beauty of using SOAP messaging is that it is a lowest-common denominator of solving the problems involved with providing a service:

• How do we address the service, which protocol should we use to talk to it? (WS-Addressing)
• How do we describe the format of the messages that are sent and received? (SOAP)
• How do we describe the format of the contents of these messages (WSDL, XML Schema)
• How do we describe the pattern of the messages that are exchanged? (WSDL)
• How do we describe the other characteristics we need to successfully talk (WS-Policy)

What's wrong with RPC end points?
My main issue with RPC end points is interoperability and flexibily.  RPC-style calls to RPC-end points in Matthew's terminology calls can do all of these things, but they are more likely to do it in a less-interoperable way that requires greater knowledge of the underlying platform, protocol or programming model (for example you may need to agree on the binary representation of the parameters used in the procedure call which may mean sharing DLL's or worse, an entire development platform).

Of course any remote procedure call can be thought of as just a particular kind of message.  So it's possible to create services that are based around RPC-style calls made within a message (which is how ASMX web services work today).  Within web services there are two types of SOAP message format - RPC and document.  As Yasser Shohoud points out in his article RPC/Literal and Freedom of Choice, RPC formatted SOAP messages are just a subset of the document formatted (I'd argue message-based) SOAP messages .  As he says, ASMX web services show it's possible to use a document style message format but with an RPC programming model.

[Update: See Shawn Smith's post on this topic.  In the last section of this post I wasn't very clear about the distinction between SOAP message formats and message exchange patterns.  My point was that it is a better idea to implement more flexible approaches to both the message format and the message exchange patterns because the more flexible approach can accommodate the more rigid approach.  In this sense the SOAP Document format can accommodate the SOAP RPC message format and the asynchronous message exchange patterns can accommodate RPC message exchange patterns (synchronous request/reply).

Ultimately, it's not worth getting too fussed up about this as both the ASMX and Indigo programming model will support you whichever way you'd like to work.  You can write services based around asynchronous messages or using synchronous request/response (stop-and-wait) RPC-style calls.  As I said at the start of this post, over time I believe there will be a movement away from synchronous RPC calls towards asynchronous messaging.]

As preparation for my Indigo presentation at the London .NET User Group next Monday, I thought I'd write a couple of posts about Indigo, starting with 'The Road To Indigo'.

Indigo is a new technology that represents on the unification of existing distributed application technology.  It takes Remoting, Enterprise Services, Web Services and Microsoft Message Queue and combines them into a single area of platform-level technology.  In fact, Microsoft has literally taken all of the teams responsible for these technologies and put them in the same building.  Instead of thinking of Indigo as a completely new technology, think of it as the next version of all the existing approaches where all of the best features of each are combined into a single managed-code programming model.  Here's a diagram showing the progression of these technologies:

Indigo also lets you chose the way you want to build applications without tying you in.  Indigo supports and encourages building service-oriented applications using messages based on SOAP.  However, Indigo still supports you and provides features even if you want to create distributed applications using a distributed component approach using remote procedure calls.  Choosing one approach over the other doesn't mean giving up features such as message-level encryption or transactions.  As Ingo Rammer says, Indigo recognises that you shouldn't be locked into 'a specific programming model, a serialization model, a transport model, or even just a serialization format'.

Bruce Williams from the Indigo team has a nice walk through of the the XML elements in WS-Eventing.  When you strip away all of the namespaces and XML noise, it's a nice, understandable piece of work.

I like WS-Eventing because it's another tool in the architecture toolbox.  To me it suggests the end of polling in web service applications.  In some of the applications I've worked on today the client has had to poll the webservice to check for updates.  With eventing the client will be able to wait to be told if something has changed, rather than pestering the server like a kid in the back of a car on a long road trip asking 'are we there yet?' all of the time.   It's much more efficient to wait and be notified than constantly having to poll to check if something has changed.

I'm not 100% sure about how large corporates will handle security will work across the Internet and corporate firewalls.  For the client to receive the notification they have to have an open port that the notification service can send to.  Perhaps a solution is to use an approach like the MSDN SoapMail sample, where messages were returned to some central location, like a mailbox, which is polled by another service that picks the message up and forwards it to the client.  Perhaps the answer lies in the Security Considerations area of the spec which suggests the notification should be signed with security tokens.

Well, he may not be blogging (yet) but there is an excellent interview with Pat Helland on TheServerSide.NET.  Pat proposes a solution to the debates about what SOA is and how it is defined.   Pat says we should focus instead on HST (Hooking Shit Together):

"Hooking Shit Together emphasizes the fact that I've got two things, they're separate, they're independent, and it's about: How the heck do I make them work together. That then takes me to messaging, that then takes me to Schema, that then takes me to the contracts and the flows between them while allowing them to be independent. I think there's far more in common in doing that inside an IT shop, and doing it across IT shops, than there are differences. So I actually like HST because it emphasizes what's in common about tying things together across machines, without regard to whether they're in the same company or not."

Pat will always have a special place in my heart because he taught me about Microsoft Transaction Server using an analogy about banana splits on an AVI movie that was part of a Visual Basic 5(?) multimedia (remember when that meant something?) CD-ROM.  The man's a genius.

When I was clarifying the Microsoft Messaging Message I said I'd watch the PDC DAT 420 presentation on BizTalk and Indigo.  As my PDC DVDs arrived this week I didn't have to download the presentation I finally got around to watching it.  Here are the key points I took from the presentation:

• From a BizTalk perspective, Indigo is the vehicle that will provide secure, reliable transacted services (over more than just http).
• Indigo is an API, BizTalk is a set of tools.  Indigo will natively integrated into the next version of BizTalk after BizTalk 2004.  The demos showed a prototype Indigo adapter that worked with BizTalk 2004.
• BizTalk has been achieving reliable services through the BizTalk Framework 2.  In future this will be deprecated and replaced with WS-ReliableMessaging, which will be provided by Indigo.
• BizTalk will use the security model built into Indigo to enable WS-Security support for Username tokens and X509 certificates that can be easily configured through attributes, configuration files and policy.

The presentation helped me understand that Indigo is the plumbing.  A lot of business functionality will require the use of tools that take advantage of that plumbing, such as BizTalk.  So I was interested a few days later to see an announcement that the BizTalk Orchestration will ship as part of Indigo in Longhorn.  As the article points out, BizTalk seems to debut features that eventually become part of the plumbing. 

I really enjoyed the presentation, and not just because Scott Woodgate has such a pleasing accent to listen to (New Zealand, so similar to Australian with stranger vowels).   The BizTalk 2004 health and activity monitor combined with the Orchestration debugger were great features.  I can imagine that these would reduce a lot of the pain with administrating and developing complex service-oriented systems.  I'm hoping that I'll get to benefit from these tools on future projects.

Suresh Kumar has posted a newsgroup exchange from Don Box and Scott Woodgate that cover most of the points from the talk on how BizTalk will take advantage of the features provided by Indigo.  Suresh also has some other good posts on general BizTalk 2004 resources, BizTalk and Publish/Subscribe and XML Web Services in BizTalk 2004 including talk of a WSE adapter for BizTalk later this year.   If I could find his RSS feed I'd subscribe.

Steve Maine writes about his eureka moment with the WS-Addressing spec.  The basic ideas behind it is that it's a good idea to write the address (e.g. put the address in the SOAP headers) on the front of an envelope rather than only writing the name on the envelope and telling the postman the address (e.g. leaving some address information in the SOAP envelope and some on the transport protocol headers).

The other useful aspect of the WS-Addressing spec is the idea of the <wsa:EndpointReference>.  This is an XML node that can contain any information a client or server require.  It's like a cookie mechanism for SOAP messages.  Omri talks more about it and shows how it's useful in WS-Eventing.  Omri refers to the 'resource model' that WS-Addressing provides:

The idea is that the organization of resources behind a Web service is opaque to the caller.  There can be a single resource that "fronts" for a bunch of other resources - which in turn can be organized hierarchically, scaled out, or employ whatever internal organization is appropriate.  The caller does not (in fact should not) need to know about this internal organization. 

I think this use of a front-controller style pattern is another of the ideas that excited Steve.  Don Box also talks this when he says that WS-Addressing is a solution to URI manipulation.  Instead of having to use querystrings to indicate what should be done at the receiving end, this information can be sent through with the EndpointReference.

Someone who's coming along to my .NET Exchange User Group presentation in Ipswich tomorrow night contacted me about finding a .NET implementation that handles SOAP with Attachments.  Here's some background and a possible solution (let me know if there's a better one).

The Problem
He needs to write a client to consume a web service provided by a finance group that are using Java to publish their messages in the SOAP Messages with Attachments (SwA) specification.   The frustration for his company is that even though Microsoft was a co-author on this specification they provide no implementation that supports SwA.  Omri Gazitts, an Indigo Program Manager has the definitive explanation about why Microsoft has co-authored four SOAP attachments specifications.  Dave Bettin has also been asking good questions about this issue.   Basically the requirements have changed as time progressed.  The main problems with SwA are that it uses MIME which provides no message header length so that the whole message stream must be parsed to find a text boundary between attachments, which is slow, and it doesn't work well with SOAP Headers which makes it hard to integrate with other WS-* specs such as WS-Security. 

Omri has another post that explains the future, which is MTOM that will be backward compatible with SwA.  Unfortunately as Omri says 'we have no plans to support "vanilla" SwA at this time, given that it neither imposes an infoset data model nor is as efficient as DIME - in other words it's the worst of both worlds'.  Where does this leave people like my colleague who need to consume a web service that uses SwA in the wild?

Possible Solution
Well, there is an MSDN magazine article from two years ago that has a lightweight SoapExtension implementation to convert SwA messages into a single SOAP message with the attachments written out as elements inside the SOAP Body (thanks to fellow RD, Christian Weyer for pointing me to this).  It's not too much trouble to convert this SoapExtension to work on the client.  The other part of the problem is an improved MIME parser.  I found the cpShere Email component project on GotDotNet  provides this.   With these two parts I think it's possible to kludge together something that works while we wait for a .NET implementation of MTOM in future.

If anyone has implemented or knows of a better .NET solution please let me know.

Tonight was my first bloggers dinner and quite a night.  I was the first one to sign up and the last one to arrive as Chris Anderson worked out.  What was most surprising was that other than Chris, Don and Ian 'I have no blog comments because I hand-coded my blog myself' Griffiths, I only met a few other bloggers (Chris has a better list).  A lot of blow-in blog readers!  But all are welcome and everyone's company was appreciated.

By the time I rolled up the only chair left was next to Don himself.  Luckily I'd boned up on things to talk about (John 'Policy' Bristowe also primed with a couple of last questions via instant messenge) and managed to garner the following:

• I started off with asking how do you communicate semantic intent with policy?  Don's response was: you don't.  You do it out of band with the telephone.  It's what humans are for.  There are a lot of people trying to apply formal methods to determine the structural correctness of the messages, but the intent is something that humans will work out.
• WS-Addressing.  I raised my beef about it being all well an good to have an opaque string as the address, but where was the protocol binding.  Apparently Indigo is going to use the simple URL style scheme, so a tcp address will look like tcp://domainname etc.
• The important of the Endpoint references is to provide a SOAP cookie style mechanism without having to get into the URL hackery such as query strings.
• Someone said "it just felt unusual to send text-based messages around - isn't that slow?".  Don's response was that the first time he had sex it felt weird, the next time better and now he couldn't imagine it any other way.
• What's different between the M4 and M5 releases of Indigo (Clemens seems to know a lot about this)?  Apparently the programming model has been refactored toto make it more unified and simpler (great news here).  The System.MessageBus.Servicess library has changed significantly because what was there was just not as good as it could have been.  I hope this means better terminology than DatagramPortTypeAttribute and DialogPortTypeAttribute, which always seemed a bit complex to me.
• What's the best way to approach Indigo today?  If you want to learn about indigo today in a way that will pay of in the future, don't worry too much about the high level details.  If you work from the wire level back up it will be most useful (this is least likely to change). 
• The focus with V1 of Indigo is proving that they can get great performance with messaging.
• We discussed Clemens' idea that because Indigo uses a streaming reader over the message body, it may be possible that on intermediary/endpoint may start sending onto another intermediary/endpoint before the full message is received.  Don mentioned that there's no reason to think that the message may ever finish.  Queue zen-like moment of poignant reflection and silence.  'Sort of like one giant Congo-line of a message throughout the Internet' I said.  Don didn't say it but I felt him think 'Yes, Grashopper'.
• We spoke about the fact that my pregnant wife who has always been worried about the size of my head, thought that Don was the only person I have on my PDC photowall that had a larger head than me.  Don said his head was larger than mine in more ways than one.  We agreed that his head was impressive, but he was leaning forward in the photo and clearly his head was not awesomely large (at least physically).
• I suggested Don try the Cinnamon Club for dinner.  We had the Axxiant Xmas dinner there and it was very good (though it was late and I had been drinking ... ).

Afterwards some of us moved off to the pub ...

In preparation for the London Bloggers Dinner with Don Box and Chris Anderson on Monday (see the report of this dinner, and what went on at the pub afterwards), I thought I’d get up to speed and make some notes of the key points made in Don’s interview on the ServerSide.NET.  To my mind this is the best interview I've seen or read about Indigo.

I'm posting these notes mainly so that I can have them handy in SharpReader when I'm offline and in case others might find them useful.  You can see the interview and download the full transcript on the ServerSide.NET site.

What is Indigo? 

• A connective tissue between programs.  It makes it simple for developers to put a message oriented façade on the edge of your application to allow easy communication with others.

Positioning Remoting and ASMX web services

• The problem with .NET remoting was that even though it supported interfaces, it really worked best when both ends of the pipe shared code such as assemblies or DLLs to integrate.  Sharing types like this turns out hard to manage because you often don’t have control over the deployment versioning and testing at each end of the pipe.
• ASMX Web Services (including WSE) are more flexible because they use WSDL and schema which is more adaptable to change than the OO style type interface used in remoting.  ASMX is proving more popular than remoting in the field because it is actually a simpler model and simplicity is a feature.
• Developers should go down the path of integration using messaging because it is simpler and more flexible, but support for remoting won’t go away.
• You don’t have to use XML Schemas and WSDL to describe structures and contracts, but that’s the practical way that everybody does it.
• The problem with components was that they tried to do the right thing separating interface and implementation, but it was too easy to ‘leak’ object concepts and require the sharing of types.

Do components have a place in the world?  

• Don argues that the world is simpler if you think about objects, which are great for programming software that gets integrated, tested and deployed, more or less as an atomic unit and services, which are the central abstraction for deployed, autonomous pieces of software.  Given this perspective he suggests there may not be a place for components.

What’s the difference between service-oriented programming and service oriented architecture?

• SOA is the buzz word du jour, system message bus will probably be next, whereas service-oriented programming is more focussed on the reality of the code.
• Service oriented programming is about four ideas:

1. Boundaries are explicit.  In Indigo you have to place attributes on your code that you want publicly exposed
2. Services are autonomous.  Indigo doesn’t assume you’ll deploy a whole system as a single unit.  Each service should be secure and reliable as a stand-alone unit.
3. Share schema and contract, not class.  It’s too easy to mix up interfaces and implementation, so it’s a better idea to separate schema and contract into two distinct ideas.  Schemas are defined in XML Schema language, and contracts, which are message exchange patterns, are defined in WSDL.
4. Compatibility based on Policy.  To know that services are compatible you have to care about other things than just the signature of the messages.  WS-Policy is supported in Indigo and provides a way to share this over the wire in an interoperable way.

Does Policy impose versioning restrictions?

• Policy is similar to marker interfaces in Java or C#.  You have to have a stable name that is immutable that is decoupled from the actual signatures.  Both parties have to agree on the name and it has very strict version requirements.
• WS-Policy provides a way to declare compatibility with any number of versions of a given policy assertion.  The logic engine is able to determine if party A’s assertion lines up with party B’s assertion.
• In the PDC bits you can see the first message sent on the wire is a ‘get policy’ message.

Can policy be used to create versioning schemas?

• There are three policy specs:

• WS-Policy which is the XML form for writing a logic expression
• WS-PolicyAttachment, which provides a way of taking those expressions and applying them to a particular domain
• WS-PolicyAssertion which contains a small number of concrete assertions that plug into the expressions defined in WS-Policy.

• WS-PolicyAssertion contains an assertion called Spec Version that can be used to determine compatibility at the naming level.  It’s looser than strict or nominal type equivalence that is used in C# and Java.  It provides some flexibility, not complete flexibility and Policy is used to do the version negotiation.

Are there situations where the Indigo approach doesn’t work well?

• In future the service-oriented programming model may be a practical way of doing things like cross-process, cross-app-domain or even within a single exe.
• The focus is making sure Indigo has a solid model that can be implemented and scale down well.  Objects took a ‘near’ metaphor and tried to stretch it out, Indigo is trying to take a ‘far’ metaphor and shrink it.  Indigo is striving to have very low performance costs within a single exe.

What are the performance optimizations being worked on now?

• The team is very focused on make sure that the I/O implementation is as well tuned as possible.  They want to make sure we can get bytes in, in XML and other forms, in to and out of an app domain as fast as humanly possible. 
• They want to make sure that it’s not tied to a specific protocol, such as HTTP.
• The idea is ‘I create a message here, I need it to appear in your program as fast as possible’.
• Performance is an important goal for the first release.

WSE and Indigo

• WSE is the vehicle to give developers access to implementations of the WS-* protocols as quickly as possible.
• The goal is still to integrate them into the .NET platform, but sometimes the platform has to ship before standards are set, which is why WSE will have a future post-Indigo.
• Post-Indigo WSE may be used to track protocol evolution on top of the Indigo infrastructure rather than just the ASMX infrastructure.

Will Indigo do anything new at the wire level?

• Indigo is about the software productization of ideas (such as the WS-* specs) rather than interesting new wire protocols.  For example, transactions on the wire will use WS-AtomicTransactions but within the service they are doing a lot to make transactions easier to use from managed code with extremely efficient implementations.

John Bristowe, recent MVP, fellow YAWP'er and lover of Australian Red Wine posts about his conversion of another person to the love of WS-Policy.  I've been a long-term (well for around six months, which qualifies as long-term in WS-* land) lover of Policy. Policy just keeps getting better in Indigo with metadata exchange in Indigo, but the movement needs to start now.  I think we need a slogan or song.  'People for Policy' or something like that.

John also cheekily suggests that I may have a 'showdown' with Klaus Aschenbrenner after his article on WSE 2.0 security.  I'm more that happy to share the SoapContext with anyone else who's into WSE - there's always room for more pipes and filters in the message pipeline!.  Also from John's blog I've discovered Morten Abrahamsen is doing a great job blogging about WSE topics.

Wow.  4 posts in a single evening - I'm feeling a bit Robert Scoble all of a sudden.

If you care about designing web services and believe in the 'share schema and contracts, not class' mantra then you'll be interested in XML versioning.  I mentioned pre-PDC that this was a big issue.  At the PDC Doug Purdy presented one approach that Whidbey Indigo will use to enable versioning with web services.

Just this week Dare Obasanjo has published 'On Versioning XML Vocabularies' and tonight I noticed that Dave Orchid from BEA also attacks the topic with his post 'Providing Compatible Schema Evolution'. 

In fact Dave Orchid's whole site seems excellent (it's my weekend homework since I've only recently subscribed and his feed doesn't contain the full articles so I can't read him on the train).  One good point I noticed was in this post:

'Schema authors tend to make tightly coupled Web services because they can't fully version or evolve them'. 

Avoiding this problem will be key if web services are to realize the promises they are offering today.

MSMQ kind of semantics can be implemented [in Indigo] in two ways:  They can be implemented inside a channel, or out in an application serving as an intermediary.

Indigo comes with a channel called the reliable channel that implements point-to-point buffering, point-to point-queuing - so that you can be on a laptop sending messages to me.  The messages stay in a buffer. Later your app goes away, I connect, the messages get sent to me, I reply, the messages sit in a buffer, later your app comes out and drains them.

Or I can build a queue intermediary.  Indigo V1 will ship with a queue class that you can put in an intermediary sitting at an address.  You can put messages to that and you can pull messages from it.

So those are the two options depending on whether you think of a transport kind of thing - so the semantics of me talking to you - or whether it’s a real thing that sits between your and app and the user of the app, where many people may be writing to the same queue and many apps might be reading from it.

Over time - Indigo is a long plan - in Indigo v1 we will be releasing classes, so that you can build queues.  Over time we will be implementing 'queue services' - full bore services that known about clusters and the whole thing.  Parts of MSMQ wont be in v1.  The programmer part will be, but the big old configurable server side wont be.

So, the programming model is there for V1, but a replacement for the enterprise aspects of MSMQ will have to wait until the future.  This would cover the apps where I've message queue in the past (basically a private queue that one app can post to independent of another that reads the messages off), though its not (yet) the replacement to Tibco I was hoping for.

What's the relationship between Indigo, MSMQ, BizTalk and  SQL Server "Yukon" Service Broker?  When and where is each technology appropriate?
Basically it depends what type of application you are trying to build and what environment it needs to run in.  Here's a chart adapted from the DAT406 session:

Technology	Indigo	MSMQ	"Yukon" Service Broker
Environment	Any WS-* compliant endpoint	Windows NT	Yukon on both ends
Application	Any distributed application	Any Async NT Application	Database applications
Message Store	In memory or DB persistent store (Yukon/SQL Server 2000)	MSMQ Message Store (NT File System)	Built into Yukon
Type of Message	Persistent and non-persistent dialogs	Reliable, Express and Transactional Messaging	Transactional messaging only
Protocol	Various	Various	TCP only

Where does BizTalk come into it?
BizTalk is a product that that builds upon other technologies in the Microsoft platform.  As BWill says in comments on a previous post, choosing BizTalk or Indigo will be a question of how much of the infrastructure you want to build yourself.  BizTalk has connectors for MSMQ currently, in future it may connect to Indigo or possibly to Yukon Service Broker.  BizTalk 2004 is currently in Beta it's on a different release schedule than the other products.

Where does Yukon Service Broker fit?
Here's Roger Wolter's answer from the DAT406: Building Reliable Asychronous Database Applications with Yukon:

The key to keep in mind is that Service Broker is a database application framework, not a messaging system.  Yeah we send messages to other databases, but if you're building a messaging infrastructure there's not enough in Service Broker to satisfy your needs.  If you are building a queued, asynchronous database application and you wanted to use reliable messaging to scale out that application, Service Broker is the answer.

There were some hostile questions in the DAT406 session about why it was necessary to put a messaging layer in the database.  John Cavnar-Jonson (who definitely needs a blog) calls it an 'abomination' in the Developmentor Indigo discussion list.  Personally, I think it's part of a Dr. Evil style plan from the SQL team - if they were to add a spreadsheet to the product then a great majority of the world's applications wouldn't need an OS letting the SQL team achieve world domination.  Seriously though, there seem to be several good reasons:

• Developers that know SQL can now  develop queued, asynchronous database apps.  The ability to have asynchronous queues is a very nice architectural feature.  Being able to achieve this with SQL syntax like BEGIN DIALOG ... FROM SERVICE ... TO SERVICE is pretty cool.

• It's all in the one box.  Everything happens within the database.  Backup, restore, installation, configuration, monitoring and security are all there in the one location.  So deployment of the database is deployment of the messaging system etc. (no need to hassle with MSMQ installs).

• The  message broker is the database.  It's easy to query the status of messages, processing the queues is as simple as writing a stored procedure, the database can efficiently throttle the queue processing resources and it's possible to farm out message processing work to another machine since all that is required to process a queue is a DB connection string.

• It's fast. The Service Broker is fast because there's no need for two-phase commits for transactional messaging, there's no need to cross processes to get to the messaging platform and if the send and receive queues are in the same database then it's very fast.

Neils Berglund from Developmentor has been teaching Yukon for a while to Microsoft employees (such as Tim Sneath) and has an excellent sample chapter on Yukon Service Broker that's available for free download.

How does MSMQ fit into the longer term picture?
BWill says in my comments that the Indigo team has shared the love and embraced the MSMQ team into its building.  John Cavnar-Jonson did some research at the PDC:

I completely disagree with the idea that Indigo offers all of the functionality of MSMQ. I discussed this exact question at the PDC with Steve Swartz, Mike Vernal, and Anand (whose last name I don't recall, but he's an MSMQ program manager).  Indigo will offer reliable messaging (which is a huge improvement over current web service technology), but it will not be a full-fledged message queuing system.

As I reported from the PDC, the message was:

"We are not building the uber queuing system - we are not a replacement yet for MSMQ - we have support for routing, but we aren't replacing CISCO, we support eventing but we aren't a replacement for Tibco."

I think there's more to be said in this space.  It's likely that this is about achieving an Indigo V1 release (primarily about unifying the three different programming models and baking WS-* specification support into the platform) and then targeting more ambitious goals with future releases.

Which parts of Indigo will ship in Whidbey and which bit will ship before or with Longhorn?
Basically, System.Transaction will be in Whidbey, the rest later.  I'm still digesting Don's WSV302 Indigo Part2: Secure, Reliable, Transacted Services and Jim Johnson's Transactional Programming on the Windows Platform presentations to understand this more deeply.

Given that the last version of WSE will be wire-level compatible with Indigo and that a future version of WSE is likely to support WS-ReliableMessaging, what are the benefits of Indigo other than the simplified programming model?
Even though I love WSE I'm following the words of Hervey and accepting that WSE is V.Last++.  This question was me fishing for what features Indigo will provide me with as an architect/developer that I can't get from WSE.  BWill mentions:

Advantages of Indigo over WSE: three off the top of my head are performance, integration, and support [it's part of the platform rather than WSE's 2+1 support policy]. I'm sure there are others. Also, note that there is no guarantee that every feature in Indigo will also be WSE.

So, no bites as to what the extra functionality of Indigo might be, so I'm still fishing (e.g. digging deeper into the Longhorn SDK Indigo Samples).  Of course, Indigo has learnt from WSE, so the Indigo programming model will also be nicer (though the WSE programming model is already small and well refactored).

Indigo is committed to supporting WS-* standards and interoperability, but what extra functionality will be available if the whole environment is made up of Indigo boxes?
I'm still trying to get a feel for what features and functionality might be available in Indigo.  BWill mentions the fact that Indigo will likely run faster in an all-Indigo environment. 

Indigo does offer Peer to Peer functionality
Robert Scoble tells us:

I stopped in on Don Box yesterday and he gave me a demo. Indigo is going to radically change how we think of Internet technologies. Imagine something that looks like a website, but that doesn't require a centralized server. Now you're getting your mind around what Indigo could do. Indigo is designed to take advantage of our always on, always connected computers.

It's not difficult to spot an Evangelist with a marketing strength is it :)? Sounds like a simple programming model on top of existing network stacks opens up opportunities to use the Internet for more than just the web browsers against a central web server.  I think I'll review WSV306 Indigo and Peer to Peer apps on the train this week.

On the 'death of objects' and the SOA 'paradigm shift'
Same started off by saying:

SOA [is] one of those paradigm shifts - it really does mean the death of objects at least as we know them.

Before I could suggest that we fine any blogger who uses worn-out expressions like 'it's the death of [technology X]' and 'paradigm shift', Sam later clarifies that he didn't mean objects but OOA/OOD: 

I really meant OOA/OOD - how do you now design/decompose system design/requirements into architectures and I think it's less now about classic OOA/OOD and tightly coupled object design and more about a loosely coupled collection of components under a service.

I agree that proclaiming the death of objects was a misstatement. It may be more accurate to say that components, which are based around sharing types between parts of the system, are likely to become less of a primary focus with the rise of SOAs.  I agree with what Bryan Noye's says:

I don't think SOA means the death of OOP or Components at all. Just like most people build components using OOP, I think most people will built SOAs using OOP and Components. They are not competing concepts but complementary.

SOAs are important where there is a need to share messages and interoperate with unknown others
For my mind, SOAs provide the most benefits when there is a need to share data/information and  interoperate with other groups, possibly on unknown platforms, that you have no control over. There will still be plenty of applications that are built in the current n-tier component Enterprise Architectures. Talking with Jim Johnson at the PDC he made the point that the benefits of SOA with external partners may also be benefits within an organisation or service boundary. I agree, but I think the technology platform (e.g. Indigo) and management tools (e.g. whatever Microsoft are planning here) have a long way to go before these benefits outweigh those of using existing component-based technologies.

Interoperating with others often means going for a lowest common-denominator approach which is always going to perform slower than when you can go with a binary format and control both ends of the wire (as Sam mentioned, using ASMX just doesn't give the same performance as current 'binary typed' systems, which is why Indigo will do special things if it knows it is working in an all-Indigo environment).

SOAs are currently still complex and difficult to build and manage
SOAs are currently still complex and difficult to build and manage. They are complex because the standards are still being implemented (on a recent project I was on it took a major international bank nearly 3 months to convince a market-leading J2EE vendor to adopt SOAP headers and honour the 'mustUnderstand' attribute).  Newer standards like WS-Addressing are still being worked through and implemented.  It's difficult because of the layers of the technology that must be understood in order to build the systems.  Just read Clemens' description of his latest FABRIQ project to see the level of technology understanding, skill set and experience you need to build an SOA project today. As the tools develop and experience and awareness of SOA's grow they are likely to get easier and simpler to build, in the same way that client server and then n-tier were once considered complex and difficult but are now considered main stream.

Services are about outside, Objects/COM+/Enterprise Services/MSMQ are for inside
It's useful to be aware of boundaries as Don Box demonstrated at the PDC.  There are parts of SOA that are designed to be used on public organisational boundaries and some that are better deployed within an organisational boundary or even behind a service boundary.  Clemens Vasters' makes the distinction between 'near and far'.

When using services outside an organisational boundary there are benefits to using open standards and working with contracts and schema rather than type, since it's difficult to control what is at the other endpoint.  Enterprise Services and MSMQ provide useful functionality that isn't yet covered in WS-* standards, but the problems with these two approaches is that they often share binary type information or require a Microsoft box or adapter at the other end.  This doesn't mean they shouldn't be used in SOA, just that they are better used inside the organisation where it's possible to have more control over the communication and the endpoints.  Within the service boundary there's still a service to provide, and it's here that technologies like components/COM+/ES/MSMQ are likely to be just as useful as they are today.

Gregor Hohpe's Enterprise Integration Patterns provides useful SOA guidance
I think that Sam Gentile is right, there are some architectural changes that need to move towards SOA and message-oriented systems.  Luckily Gregor Hohpe has written Enterprise Integration Patterns - Designing, Building, and Deploying Messaging Solutions a great book distilling his experience of these systems.  Hervey Wilson's reading it and its on Ingo Rammer's list of recommended books as well!.

A month after the Indigo 'Kimono opening' at the PDC there's still a lack of clarity about what Indigo is, how it relates to other messaging technologies and what's the best way to start developing applications today.  While a lot of this was covered at the PDC my perception is that some of the message hasn't been ack'd successfully from the audience.  [Update: See my more recent post 'More on the Microsoft Messaging message' for some answers to these questions]

The Longhorn DevelopMentor mailing list had an excellent exchange yesterday on Indigo, which has lead me to highlight some areas where I'd like a clearer message:

• What's the relationship between Indigo, MSMQ, BizTalk and SQL Server Service Broker?  When and where is each technology appropriate? (Thanks to John Cavnar-Johnson for highlighting this)
• How does MSMQ fit into the Indigo longer term picture? Is MSMQ for within the Enterprise or will it focus more on outside the Enterprise messaging?
• In terms of EAI technologies, where should we look for this functionality - Indigo or BizTalk?
• Which parts of Indigo are going to ship in Whidbey and which bits will come in Longhorn?
• Given that the last version of WSE will be wire-level compatible with Indigo and that a future version of WSE is likely to support WS-ReliableMessaging, what are the benefits of Indigo other than the simplified programming model?
• Indigo is committed to supporting WS-* standards and interoperability, but what extra functionality will be available if the whole environment is made up of Indigo boxes?

I'm  know some of these were addressed at the PDC.  I'm still digesting some of it (for example, the PDC DAT406 presentation on the "Yukon" Service Broker shows how MSMQ, Indigo and Service Broker are positioned, though not BizTalk).  Other questions I'm still researching.

Web services are being adopted in the wild
There certainly is a lot of material to cover in web services from the specifications to the implementation, so we only briefly touched on Indigo.  There were 30 or so developers there. Starting with the mandatory 'polling' questions I was surprised that around 60% of them had used web services already.  Of that group about a third had added some form of security (HTTPS, VPN or internal network).  There were some good questions in the group ('Can I use WS-Security in a PHP application?  Will Microsoft compete with Tibco and the EAI vendors with Indigo and/or BizTalk?  Has Don Box stopped using the term GXA?)

Blogs are a great way to learn about Web Services
I mentioned that blogging is a great way of tracking what's happening web services. I showed how:

• Omri Gazitts a PM on the Indigo team, talks about web services attachements and that DIME is still the best choice for today, but MTOM is Microsoft's preferred approach for the future.
• Hervey Wilson, WSE lead developer, has said that WS-ReliableMessaging is coming in a future release of WSE and writes great content about lots of other topics.
• Rebecca Dias, Product Manager for WSE, has mentioned as a result of feedback at the PDC, "Attachments will be abstracted away [in WSE 2.0] so that migration to MTOM will be more seemless(sic)"

How to get started reading blogs
First download a news aggregator like SharpReader or RSSBandit then download my modified OPML file.  It contains many Microsoft bloggers as well as other web service related blogs (including those on Matt Powell's list).

I also promised a link to the code that downloads all of the PDC powerpoint presentations.

Don Box is a great speaker, but as Scott Hanselman and Steve Maine and this post demonstrate, sometimes understanding and explaining the reasons behind his tenets can be difficult.  As an intellectual entertainment and learning exercise, I'm throwing my hat into the ring, trying to answer a commentor on Scott's website who asked what is meant by Don's tenet of 'Share Schema, Not Type' (actually it was 'share schema, not class'), or as I like to ask 'Why should I use web services (with schema and WSDL) over .NET remoting or DCOM?'.

Steve's Answer
Steve's answer goes like this:

• Types came from the land of COM and were all about binary implementations of interfaces.  Interfaces described the methods that could be called on an object, so they were all about behaviour.
• There were two problems with Types.  Firstly, since they just described behaviour, it was hard to make sure that all implementations of the interface produced the same behaviour (that they were semantically consistent).  Secondly, that once published the interface was immutable and hard to extend.
• Schema are from the land of Web Services and are all about describing the layout of a message. Schema communicate state and nothing about behaviour.  Contracts are the way that schema describe behaviour by specifying the format of input and output of messages that are sent to a service.
• Schema solve the problems of Types.  Schema can extend types by using the open content model.  Schema can ensure semantic consistency across implementations by specifying the schema of the input and output messages.

I like Steve's story, but I'm not sure it's the answer.  The open content model is one approach for extending schemas, though it still works like COM and requires that the newest version provide all of the content required for all previous versions, so I'm not sure how much of a victory this is for the schema approach.  Also, I'm not sure of the differences in terms of semantic consistency, between a COM interface that defines the types of input parameter and output parameters from a method, and contracts based on schema that defines the format of input and output messages of a service. 

Scott's Answer
Scott's answer was that types are unique and immutable and require the sharing of an assembly, whereas schema is a description of the XML content that acts like a contract between parties that can be used without the need for an assembly.  I think this emphasis on avoiding a binary assembly is closer to what Don was talking about.

What was Don thinking?
In Don's PDC Session “Indigo": Services and the Future of Distributed Applications, (slides here) session, Don's tenet was 'Share schema, not class: Integration based on message formats and exchange patterns, not classes and objects' (see Tenet 3 of  my transcript of Don's talk at the PDC, which now seems less comprehensible than I'd hoped).  In his MSDN article Don restates it as "Services share schema and contract, not class". 

"Classes are convenient abstractions as they share both structure and behavior in a single named unit. Service-oriented development has no such construct. Rather, services interact based solely on schemas (for structures) and contracts (for behaviors). Every service advertises a contract that describes the structure of messages it can send and/or receive as well as some degree of ordering constraints over those messages. This strict separation between structure and behavior vastly simplifies deployment"

So, I think the advice behind Don's tenet is to build services using the service-oriented approach (using schema and WSDL to bind the input and output message) rather than an object oriented approach (DCOM or .NET remoting).  This is because it is easier to evolve the structure of the input and output messages using schema rather than having to redeploying an assembly or a binary interface description (*.tlb).

Don's argument is that service-oriented approaches can evolve more flexibly because they can use "features such as XML element wildcards (like xsd:any) and optional SOAP header blocks to evolve a service in ways that do not break already deployed code."  I'm not entirely convinced that COM didn't provide a similar level of flexibility as the xsd:any wildcard (e.g. using As Object in VB).  The optional SOAP headers seems feasible.

What do I think?
I think that interoperability across binary platforms is the key reason to favour schema and contracts over classes.  Object-orientation and types are all about binary representations that are just too difficult to get all of the binary platform vendors to agree on.  Schema provides a machine verifiable mechanism that currently uses text representations which are the lowest common denominator and can be implemented across all platforms.

Within a platform, I think using schema and contracts (web services) over classes (DCOM/remoting) wins on pragmatic grounds because schema and contracts seem easier to deploy (no binary object copying required), debug (it's easy to sniff the packets on the wire to work out what's going on) and evolve (using schema and wildcards seem easier to work with than interfaces in COM, especially when with the XML versioning improvments in Whidbey).

In the panel on Indigo a question was asked about how far Microsoft were going to go in making Indigo support pub sub messaging and competing with Tibco.  The answer was that Microsoft wanted to go all the way, but they couldn't do it all in V1.  The core infrastructure will be there in V1, but it will get easier to access in future versions.

It's 8:30am and there's a room full of geeks wanting to get deep down and dirty into the Web Service protocols.  Omri is the Product Unit Manager for the Advanced Web Services group, so he's the man responsible for the WSE team.  He demonstrated the secure, reliable and transacted demo that Bill Gates gave with IBM on 17 September.  The slides for this session are available.

Positioning Indigo's Protocols
The wire protocols that Indigo uses are key to interoperability story.  Omri is positioning Indigo as the Internet's L7 protocol, comparing Web Services as the top of an Internet stack above XML on top of HTTP on top of TCP on top of IP and onwards.  Microsoft see Indigo as supporting applications between mobile devices, P2P, B2B and EAI style applications.

The Secure, Reliable and Transacted Demo
The demo between Microsoft and IBM showed a commitment to interoperability, as well as a commitment to making sure the protocols involved in web services were royalty free.  The demo was a Supply Chain Management application involving a supplier managing inventory and ordering parts from an auto manufacturer.  It demonstrated security, reliability and transactions in an interoperable way.

MindReef  SoapScope - a better SoapTrace packet sniffer 
Omri showed the messages flying on the wire using MindReef SoapScope, the same tool that Scott Hanselman used in his Learning To Love WSDL presentation at TechEd 2003 (the funniest presentation I've listened to).  Werner also recommends it.  It looks like this is a better approach than MSSoapTrace, which is so buggy that it makes development a horror.  Are Microsoft going to fill this hole?

Web Service Design Principles

General-purpose, composable protocol framework.  Web services allow a building block approach, rather than a monolithic stack like DCOM.  The use of SOAP headers allows the various specifications to be improved and versioned in isolation, without having to redo everything.  Omri mentioned that making the specifications composable was a challenge.  He pointed out that WS-Attachement which was based on the idea of using MIME attachments, based on email, was a mistake that couldn't be combined with other SOAP specifications, like security that relied on SOAP headers.  It became clear that to add security to attachments they would have to reinvent many of the same ideas used to secure SOAP messages: a better model would be to put the attachment within the SOAP envelope so that WS-Security using SOAP headers would provide security.  Omri mentioned that they had learnt from their mistake and were proposing MTOM (message transmission optimization mechanism) as a SOAP-based attachment specification.  As Martin Gudgin says, DIME is dead.

Architecture is meta-data driven, policy based.   WSDL provides some meta-data about a server such as the basic information about messages and ports.  However, this is not enough to describe what the service needs.  This is where policy comes in.  Policy provides another way of expressing the extra metadata that a service needs ( Basically schema is not enough to describe what the service requires).  You can say things like I am a service that requires strong 128 key encryption.  Also, I support HTTP but I am inside a trust domain with you so I can go faster by UDP or TCP.  Deployment information and info about a service that can't be captured in WSDL is in policy.

Protocols are factored from both transports and applications semantics.  Basically web services place information from the transport into the soap message, so rather than using HTTP request response model based on HTTP headers, the information is placed inside the SOAP message to provide abstraction from the transport. 

Architecture allows intermediaries.  Having intermediaries is useful for routing and other reasons.

Uniform data model for protocols/content.  The same infoset is used for both the header and the body.  It allows for header and body-based content-based routing.

Broad industry partnership.  Nothing was going to get off the ground without interop.  Interop was necessary for a transformational change. Omri says that Microsoft knows interop is essential if Web Services are going to be a revolutionary technology.

WS-Addressing
Addressing is a core part of the architecture, the work horse of the stack.  It provides a mechanism to encapsulate identity information in an XML end point reference. There is a basic <Address> element that is the network address of the resource.  There's also the <ReferenceProperties> element that can be used for extra information.

The idea is to allow people to host more than one service at a particular network address.  So if there is a service facade it can host multiple services, without having to do ugly bizarre things with the URLs.  The addressing mechanisms are transport neutral and the internal resource organisation is opaque.

The header has to have a to and action.  It can also have a From, ReplyTo, FaultTo, MessageId (a unique id) and RelatesTo (that provides a way to correlate the message).

Attachments: WS-Policy, WS-PolicyAssertions, WS-PolicyAttachments
Policy provides a framework for making statements about resources.  The policy can say that the service requires (such as transports and security).  Policy can also be used at runtime to make sure that the messages meet these requirements. 

WS-PolicyAssertions provide some basic requirements that can be expressed such as TextEncoding, Language, SpecVersion (Uri asserting compliance with a namespace spec) and MessagePredicate (XPath statements about the message)..  WS-PolicyAttachments solves to the problem of how to attach policy to an endpoint (e.g. matching an address with a policy).

WS-MetadataExchange
This addresses how to get metadata from another endpoint.  There isn't yet a specification to cover this although the demo used a simple get to retrieve policy, WSDL and schema.  It was implemented in the bits using a simple GET mechanism.  Kevin Hammond mentions some more details of how Indigo is doing it in the current bits.

WS-Trust
This defines how to broker existing trust relationships.  It defines how to exchange security tokens using a security token service.  The specification details a RequestSecurityToken message that is sent to a trust service, that replies with with a RequestSecurityTokenResponse.  This is currently supported in WSE and Hervey Wilson, the lead developer has posted a code sample that demonstrates it.  WS-Trust is one of the building blocks of WS-SecureConversation.  

WS-SecureConversation
WS-SecureConversation defines a mechanism to allow clients to make multiple calls to a web service, without incurring the overhead of asymmetric encryption.  It involves using a SecurityContext token based on symmetric encryption.  Think of it as a security 'session' object.  The token can be received using WS-Trust to request one from a security token service, or from having one party create one, or through negotiation.

WS-SecurityPolicy
This specification allows for policy statements about security tokens.  It can be used to specify the types of tokens, the confidentiality or encryption requirements, the algorithms to use and the which parts of the message require signing or encryption.

WS-ReliableMessaging: better QoS, less code
Reliable messaging allows developers not to worry about an inconsistent network.  It provides ExactlyOnce and in order messaging that saves developers from having to write code to worry about lost, duplicated or delayed messages or endpoint failure.  Sent messages are stored in a persistent local store and are only removed when an ACK is received from the server, otherwise they are retransmitted.  Early and Adopter have more discussion. Rebecca Dias has a Starbucks metaphor for the specification-challenged.

Transactions
Transactions are described by two specifications, WS-AtomicTransactions and WS-BusinessActivity, based on the WS-Coordination.

WS-Coordination.  The common parts of the two-following specifications.  It deals with how to set up a context for the transaction, based on WS-Addressing.

WS-AtomicTransactions.  This is the standard two-phase commit (prepare, rollback or commit) using web services.  It uses locks to prevent others accessing resources.  It works best inside an organisation where you can trust the locks to be used correctly.  Don Box calls this the 'anti-availability protocol'

WS-BusinessActivity.  This is a more flexible approach that doesn't use rollbacks or locks.  Instead transactions can be reversed using compensating actions.  In this way the system continously moves forward, there is no need to roll something back (if money was credited into one account, but something else in the transaction failed, post a credit to correct it rather than using a transaction to lock and roll back the initial credit).  It also removes the need for complete agreement between all resources before committing.  Omri's example was a transaction that books a flight, hotel and car for the PDC.  If the car isn't available, he still wants to make the trip and use a bus or a taxi, but there's no need to cancel the booking.  This approach is best for loosely coupled (think external web services) situations, but isn't in the PDC Indigo bits (they think it's important, they just couldn't do it in time).

Omri has announced he will repeat his session on "Indigo: the Web Services Protocols and Architecture" which is great.  Martin Gudgin says that they are demonstrating the sample from the Gates/IBM demo (I want these bits in WSE!).  I saw the comments on the WS-Transaction specs that I'll blog on tomorrow (or I'll beg Drew, who does a very, very thorough job of session notes, to go).  I'd like to sponsor Drew to go to conference I can't make it to in future).

I was so busy catching up with people over lunch I got there late and was sitting in the corridor watching the TV again.  Even though I spoke with Omri last night at Meet the Experts, I didn't recognise his voice and the TV only shows the PowerPoint from the smaller rooms.  I thought that it was Don Box with a shot voice from singing with Band On The Runtime.

Steve Millet is talking about the improvements in the Indigo model for security tokens.  The good news is that the madness has stopped: when a UsernameToken is validated you only need to return a bool rather than the password.  WSE 1.0 and 2.0 require the password to be returned allowing WSE to work out whether they match.   This was uncomfortable for several reasons, such as the fact the password might have been hashed, or just the fact that sharing the password back with the framework feels like a 'boundary violation'.  I'm glad that we're seeing the end of this bizarre API practice.

Other interesting tidbits were that SAML tokens will be available in Indigo.  Now, if they were only in WSE ...

Existing WSE/ASMX applications are likely to have a good upgrade path to Indigo, with similar security attributes.  There are also extensibility hooks in Indigo to do custom security token handling, so there's an upgrade path for WSE (though this is almost certainly having to write code).

The message from Joe Long a PUM with XML Enterprise Services is that there's an easy migration path from existing applications to Indigo.  The changes are mostly using different namespaces and attributes.  Most of the code inside methods stays the same, except for .NET remoting where instead of using 'new' inside a method the developer has to explicitly state that the object is created remotely

Key Questions

• I have an app - how do I make binary work with Indigo?
• Now how do I move the code to Indigo
• If I start writing a new app today, what should I do to best set myself up to move to Indigo?

There are two approaches to integrating:

• Teach the infrastructure new protocols. Put WS-Security into the existing infrastructure like DCOM and MSMQ
• Teach new infrastructure with existing protocols.  The problem is that these protocols don't interop or work over internet.

Various points

• ASMX code migration is basically just a matter of changing the namespaces and attributes, the code within the methods just work.
• .NET remoting requires similar basic changes.  Is .NET Remoting dead?  No, it's used inside the same process, creating objects and treating them like local objects.  The only difference is the new - you have to declare whether you're doing it locally or remote.  Channels or sinks are not supported, but most of the time these were used to overcome limitations in the platform.  Now things are better.
• There will be a new version of WSE (WSE 3.0?) before Indigo is releases.  The latest version of WSE will also be supported.

WSE code migration

•  Migrating WSE code to Indigo may require a non-trivial development investment.
• There will be whitepapers saying how to migrate.

The key PDC message, or welcome to the Service Oriented World

• Objects are appropriate within boundaries
• Services are appropriate across and within boundaries

What to do today

Build services using ASMX

• Enhance your ASMX service with WSE if you need the WSE feature set and you can accept the support policy

Use object technology in a service’s implementation

• Use Enterprise Services if you need ES rich feature set, you are communicating between components on the local machine and have performance issues with ASMX or WSE
• Use .NET Remoting if you need to integrate with an existing proprietary protocol, you are communicating in-process, cross app domain
  Use System.Messaging if you need the reliable messaging and queuing features in MSMQ

Things to watch out for today

ASMX

• Avoid or abstract using low-level extensibility such as the HTTP Context object

.NET Remoting

• Avoid or abstract using low-level extensibility such as .NET Remoting sinks and channels

Enterprise Services

• Avoid passing object references inside of ES
• Do not use COM+ APIs – use System.EnterpriseServices
• Do not use MSMQ APIs – use System.Messaging

After exhausting every question I could think of with James Newkirk, I walked over the black shag pile carpet to the Indigo area of the Microsoft pavilion.  Jim Johnson saw me hovering and asked me if I had any questions.  I said I was a WSE afficianado and wanted to join the wave and learn more about Indigo.

Jim is an architect working on the transactions within Indigo.  Jim is a really nice guy and took me through the WS-specs (Jim was on the working groups that wrote them) that relate to transactions:

• WS-Coordination - the foundation of the transaction 'stack'.  It's concerned with associating an activity identifier with a set of messages.
• WS-AtomicTransactions - This defines a protocol that allows existing transaction systems (think MS and IBM) to wrap their proprietary protocols in an interoperable way.  We talked about how exposing transaction locks outside service boundaries was not something that you want to do very often.  Jim pointed out that within an organisation's boundaries it is useful to be able to achieve two-phase commit using web services protocols.
• WS-BusinessActivity - this is the long running compensating transactions.

We spoke about how these specifications were going to be made available to developers.  It turns out the object model to manipulate these transactions will ship around Whidbey and some of the low level implementation would ship with Indigo.  Don mentioned these topics in his talk this afternoon.

Jim was an incredibly interesting and open guy.  He spoke with a lot of passion about what he's currently interested in - making transactions available automatically to all managed objects.  He had a great, almost Socratic way of talking about topics, for example "Imagine a world where transactions took a millionth of a second", "Imagine a world without catch statements, or where there was no clean up necessary within the catch statement".  Each time he let me try and think through what things meant or what would be possible.  It was so fascinating that I completely forgot about lunch.

We also spoke about more general career topics.  He believed it was important to find a question that you could think deeply about, then find a group of people who are doing it and finally have a passion for shipping software.  I really enjoyed the opportunity to talk with Jim.

The fatigue is showing: I got the session code and the room number mixed up so went off looking for a non-existent room and have ended up sitting in the corridor watching the television.  I met Doug at the Microsoft pavillion before the session and he's in fine form (so excited that he's losing his voice).

Some key points:

• Serialization is not the same as XML programming.  Xml programming is using XSD to get CLR type.  Serialization is about saving a CRL type to XSD or to another CLR type.
• Indigo uses a new attribute [DataContract] instead of [Serializable].  This is part of the general shift from the framework doing things invisibly, or by magic, to the new approach where the developer must opt-in.  Developers must mark up the class with [DataContract] and the members with [DataMember].  Like other attributes in Indigo, these attributes don't care about the CLR's visibility because serializing into XML is not the same as controlling access to objects.
• It's possible to deserialize one CLR type into a different CLR type provided that the serialization class and member names match.
• [DataMember(VersionAdded = 2)] allows you to support versioning in schema.

How to version schemas
A group is added to the schema called 'UnknownData'.  The previous approach that Doug mentioned at TechEd was the open content model. The downside of the open content model is that you lose the checking in future versions.  UnknownData uses the open content model, but it includes a version number which means you can continue to extend the schema.  This works in an interoperable way.

I took extensive notes from Don Box's WSV201 Indigo overview here.  I put them on a separate page because of their length.  Between these notes and Tim Sneath notes you've basically got a transcript.  Here are some questions I've still got

What shall we do about WS-ReliableMessaging and WS-Transactions and WS-Federation before Indigo ships?
Indigo is meant to ship with Longhorn, which Don admitted was still 'a few weeks away' yesterday.  So what are we going to do in the Interim?  Since we know Hervey and Keith from the WSE team have been working all summer on WS-Federation, WS-Reliable Messaging and WS-Transactions I'm assuming these bits exist today within Microsoft.  Will we be able to see them or will we have to wait for Indigo?

How does Indigo fit with existing EAI technologies?
We got the message yesterday that Microsoft are building the service bits that if they didn't write, we'd all write again ourselves (or have already written).  I'm still interested to see how for Indigo will go.  As Don said yesterday:

"We are not building the uber queuing system - we are not a replacement yet for MSMQ - we have support for routing, but we a