In the keynote yesterday, Don Box and Chris Anderson showed how Indigo can be used in a REST style by sending Plain Old XML (POX) over HTTP.  It’s a great testament to the quality of the Indigo extensibility architecture that they could do this.  By supporting both the REST and SOAP styles it takes the heat out of the debate that one is better than the other.

One of the Hands On Labs goes through the code they used in detail.  The example uses a HTTP GET request to return a plain XML payload, in this case a RSS feed.  An extra level of detail was the fact that the RSS added a custom element indicating an 'adult content rating' system, which was signed.  There were several interesting points from the Hands On Lab.

One complexity around supporting a REST style with Indigo comes from the fact that the HTTP GET request does not have any WS-Address-style action.  To get around this you can use the mapAddresssHeadersToHttpHeaders attribute of the HttpBindingElelement to make the URI of the HTTP GET request the To value on the Indigo Message object.

Since no action is specified on the HTTP GET request the ServiceContract needs to trap any unmatched action using the OperationContract's Action named parameter:

public interface IRestStyleRequestResponse
{
   [OperationContract(Action="*")]
   Message GetRequest(Message request);
}

Since the HTTP GET request on the HTTP transport in Indigo usually returns a description of the service, this needs to be turned off using the Description property of an instance of the ServiceHost type:

// service is an instance of ServiceHost
// stop the HTTP GET returning the site description.
service.Description.Behaviors.Remove(typeof(ServiceMetadataBehavior));

To hook in this binding a custom BindingElement extension was written.  The lab walked through creating a IChannelFactory and IListenerFactory and the rest of the necessary objects as part of the WCF stack.   Just before the transport binding element in the Binding stack there's a custom encoder which does the job of writing out the Message to the wire as plain XML, rather than a SOAP message (as Don Box said yesterday, 'it lathers the SOAP off the message on the way out and SOAPs up the message on the way back in').

I did a presentation on Indigo to the VBUG London group last night.  The turn out was pretty good, although it was obvious that many English football fans were missing (though they probably would have enjoyed the User Group more than the humiliating defeat).

Off the top of my head, here are the things that keep standing out for me about Indigo:

• The power of separating choices about the binding from the code that a developer writes.  Being able to switch from the tcp with a binary encoding, to reliable messaging over HTTP, to MSMQ just by changing the configuration file is great.
• The simplicity of the programming model.  The majority of the functionality that required Enterprise Services, such as Instancing and Sessions, are available using attributes on either an interface/class or a method such as ServiceBehavior or OperationBehavior.
• The fact that the details of the protocols are hidden under the covers.  Instead of focussing on the WS-* protocols, the focus is on the functionality required.  This has taken me a while to get my head around after having used WSE, which is powerful, but involved having to understand a lot of the details of the underlying protocols.

I'm looking forwad to some of the Level 400 sessions at the PDC, such as Kenny Woolf's presentation on Channels.

As Mehran noticed, I couldn't help but show off the Refactor! tool that's shipping for free in Visual Basic 2005.  This was in response to an attendee who believed that Visual Studio was 'a multi-megabyte bloated version of notepad'. 

 

Kirk Allen Evans has an interesting post about proposing a distributed architecture for a client where he finds, after doing performance testing, that the speed difference between different transports reduces as the amount of work done within each call increases.  Based primarily on this finding he's recommending ASMX web services as a better strategy than using COM+/Enterprise Services for his client.

I remember coming across this arguement at TechEd 2004 in Amsterdam where Christian Weyer spoke about some Thinktecture research which showed that the difference in speed between Web Services and other distributed technologies (remoting, enterprise services/COM+) was significant if the work done in the distributed call was minimal, but once a reasonable amount of work was being done the performance difference became less and less important.  Worrying about the speed of the transport without thinking about the amount of work done in each call is a little bit like worrying how fast a car can rev if you have it in neutral.

Of course, this is all keeping with the long-known guidance from the Indigo team, which is to start with ASMX and only go to other options if you need specific features that they provide.  Richard Turner's recent white paper provides more details.

I've just landed back in the UK after a three-week round the world holiday to Australia and back via Redmond for the Indigo Software Design Review (SDR) last week.  The SDR gave me a chance to play with a slightly updated Indigo build than the publicly available Indigo Community Technical Preview.  I was going to list the highlights of my week but then I saw that Omri Gazitts, from the Indigo Team, posted a list of his favourite Indigo features that maps pretty closely to what I was going to mention.  My key take-away was that the team have done a great job designing Indigo so that it is feature-rich and easy to use with good extensibility points throughout the system.

While I've been away Mike Taulty (Layers of Indigo, IInputChannel and IRequestChannel, Channel encoding and filtering messages) and William Tay (Message Tracing, Logging and Activity Management and the details of the default Indigo bindings) have both been doing some excellent job of kicking the Indigo tyres and exploring the CTP builds.

I'll write more when I get a fresh VPC image installed with the CTP bits.

As Rebecca Dias mentions, my article ("Why WSE") covering the high-level reasons to use WS-Security has been published on MSDN.  It covers the provides benefits WSE provides, such as end-to-end message-level security, content-based routing, and policy through leveraging the WS-Security, WS-Addressing, and WS-Policy specifications. 

Let Rebecca know what you think of it.

If you want a longer article on the same material I'd recommend fellow-RD and web services enthusiast, William Tay's piece on Solving Real World Business Problems with Web Services Enhancements in .NET

Microsoft Research in Cambridge have released the WSE Policy Advisor for Microsoft Web Services Enhancements (WSE) 2.0.  The Policy Advisor is an an unsupported tool that acts as a security diagnosis tool for WSE2 policy files (think of it as an FxCop for web service security policy files).  It analyses the policy file for common security vulnerabilities, provides a description on the risk and remedial advice.  It can be launched as a stand alone application or from the policy tab of the WSE Settings Visual Studio add in.  If you are intersted in WSE 2.0 and Policy then  download the Policy Advisor and run it against the sample files that ship with WSE 2.0 and send the research team feedback.

I've been a fan of using policy files to secure web services with WSE for a long time.  As Clemens says, authoring a policy file by hand is pushing things too far.  In combination with the WSE Settings add-in the Policy Advisor provides a great service for anyone wanting to understand and apply policy files, without having to get too focused on the XML angle brackets. The help file contains a list of all the problems the Policy Advisor can detect and is an excellent learning resource if you want to learn about the purpose of many of the policy elements.  For example:

This policy accepts messages with unauthenticated or elements. (Alarm)
Risk: The message is authenticated, but authentication does not cover and . Those elements are often used to implement replay protection, and should thus be authenticated. Otherwise, an attacker may intercept a message and generate a series of slightly different messages that will be accepted as distinct, genuine messages from the original sender. (The risk may be mitigated if the transport provides integrity protection, or if the recipient implements replay protection using other authenticated elements.)
Advice: Insert wse:Timestamp() and wsp:Header(wsa:MessageID) in the element in the assertion.

It also has warnings about the evil that using unencrypted UsernameTokens, though I'd highlight Keith Brown's excellent MSDN article on Securing the Username Tokens with WSE 2.0 as the best source of guidance in this area.

Here's how the tool integrates with the WSE Settings Visual Studio addin:

Below is a screenshot of the report that the Policy Advisor produces, in this case it is reporting against the secure conversation sample that ships with WSE 2.0.  The top part of the window describes the report, the bottom tree view highlights all of the issues found and the relevant policy for each problem.

I'm going to be presenting 'Using Web Services Enhancements 2.0 for Messaging' at the London MSDN 'Understanding Web Services' event next Tues 30 Nov. David Gristwood has more details about the event that will include some content on Indigo. I found out this morning that there are only two places left, so if you want to come along, register now.

In a recent Opinari email (I can't find it on his website), David Chappell defends his belief reuse is one the fundamental benefits of Service-Oriented Architecture and that SOA will be more successful than previous approaches such as object-oriented or component-oriented development.  His rationale is that services operate at a higher level of reuse than binary classes, and they solve many of the human problems that prevent reuse, such as having to have one single view of business entities and forcing reuse through 'draconian' human policies.

Here's my summary of his three major arguments:

1. Service orientation will lead to better reuse of business-oriented software because it is about sharing application functionality at a higher level than binary classes:
   • Evidence for this comes from the fact that the most successful examples of business reuse comes from packaged applications such as SAP and PeopleSoft
   • Attempts to share business classes, such as San Francisco, haven't been successful.
2. Previous approaches to re-use have failed due to human reasons rather than technology:
   • It is hard to agree on a definition of a business entity, such as customer. 
   • Even if there is agreement about the 'one right view' then it is likely to change over time.
   • It is difficult to force developers to reuse software - given a choice (or allowed to get away with it) most developers will reinvent rather than re-use.  Where reuse has been achieved with previous technologies, it was mostly through 'coercive management structures'
3. Service-orientation overcomes some of the human issues involved with reuse because:
   • Services don't require a single definition of business entity, just conformance with a particular schema. A service-consuming application may have a different view of a business entity as long as it can map to the service's view of that business entity.
   • Services can better handle changes over time.  Service-boundaries are a better way of insulating consuming applications from changes to the internal implementation than base classes and binary inheritance as there is a cleaner separation of external interface from internal implementation.
   • Since services are more explicit with their boundaries, and often these boundaries are physical, it is harder for developers to find ways around them.  This is similar to Clemens' point that well designed services do not share databases.  David's argument is that these design decisions make it easier to enforce reuse.

I like his focus on the human problems that plague reuse.  I also agree that sharing schema is a better approach than sharing class (part of the four tenets of service-orientation), and that using interfaces rather than inheritance leads to more flexibility to cope with change over time (a well-known principle).   I find the argument that service-orientation will make re-use easier to enforce less persuasive, since this seems more to do with deployment than the design philosophy (a component-oriented application could also reuse components that encapsulated a private database).

In an interview with Mary Joe Foley, Joel Spolsky says that Indigo will make building applications easier, but it won't lead to a whole new class of applications.

If you look at the different aspects [of Longhorn], like Indigo, I'm not really even interested. It's just a big communications architecture that makes it easier for programmers to build communications things. But there's no application you can't build right now because you don't have a good communications architecture. It might be harder (without it), but it's not going to enable you to build a whole new class of applications.

I think there is some truth in this statement.  As Steve Swartz has said, Indigo is a replacement of the infrastructure and an evolution of the programming model, through simplification and a unification.   As Clemens has demonstrated that you can use existing Microsoft technologies (ASMX, MSMQ, Enterprise Services, and Remoting) to do many interesting things today.  If you need to build a service oriented application it’s worth starting today and not holding off until Indigo arrives.  However, I think Joel underplays the benefits of making the programming model for service oriented applications simpler. 

The problem with building service oriented applications today is that it takes talented developers, like Clemens to do them.  As Don has mentioned, one of Indigo's goals is to make things so easy that he will have to join MS to survive. 

The applications that are developed on Indigo may not be ‘a whole new class of applications’ (though I think it’s highly likely that the killer apps of the future will be based on top of communication technologies), but the fact they are easier to develop them should lead to more of these types of applications being developed.  This is in the same way that VB enabled an increased number of COM applications that could have been built with C++.  It’s an interesting argument as to whether VB lead to ‘a whole new class of applications’ being developed.

There are several questions that I still have about the impact of Indigo:

• Where will the business imperative to build applications that communicate more come from?  Is it that there is a demand today but businesses aren’t developing the applications because of the technical cost? 
• Will the increased ease of development encourage businesses to build applications that they haven’t considered building today?
• Will the pool of distributed application developers increase, or will Indigo just be adopted by the developers who are already building distributed applications?
• Will the increased simplicity of Indigo mean that the architectural concepts behind service orientation will be more widely used?  For example, I’m puzzled as to why MSMQ doesn’t get more love – message queuing is an important architectural design and MSMQ works very well with a very simple programming API – but I don’t see it used as widely as I think it could be.

Thoughts? Comments?

Dave Bettin writes about his experiences doing a user group presentation on Indigo.  His main disappointment was that few attendees seemed to know about Indigo or the existing ASMX web services in .NET.  This got me thinking about how to get the message out about distributed apps, service-orientation and Indigo.  Is it the case that most developers need to be educated about the benefits of scalable, distributed, service-oriented apps or is it that they don't need these approaches to solve their business problems?

Dave's feedback is similar to Clemens' experience earlier in the year, where he spoke of the difficulties selling the relevance of Indigo today before he settled on explaining the 'why' over the how.  Both Dave and Clemens' experienced the same difficulty making the connection between Indigo and developers' jobs today.  Clemens' says that one of the barriers is that people think Indigo is only going to be relevant for 'Big Apps', when in fact it will be important to anyone who builds apps that expose functionality to other apps. 

One argument is that in order to get more interest in Indigo we need to educating developers about the benefits of services and distributed apps and get them to stop writing monolithic apps and start separating the logic across tiers and services.  We need to teach them more about the benefits of the services and distributed application approach, such as scalability.  Sam Gentile touched on this sentiment when he wrote that most .NET developers don't grok distributed computing. 

The other position regarding evangelising Indigo is mentioned by Alex Lowe in a comment on Clemens' post, is that:

Of course, we know that in the real world there are lots of applications that don't expose functionality to other applications. There are plenty of line of business applications for small and medium businesses that don't require ASMX, Remoting, and in the future Indigo.

Alex also has a summary post on his position where he says that the number of applications that need (and by extension, developers) that actually need to know about how to architect scalable applications is low.  He argues that bloggers are a distorted sample because they represent the group that need to build scalable distributed applications and they are generally well up on things.

I'm interested what other people think.  Is the problem that most developers don't understand the benefits of service-oriented distributed applications or is it that most developers are writing apps that wouldn't benefit from service-oriented distributed applications?  What business application scenarios are most likely to benefit from adopting service-orientation and Indigo?

I hosted a lively session on what Service Orientated Architecture really means at TechEd Amsterdam.  While it was a Birds of a Feather session, I decided to run it as a park bench format in order to take advantage of having David Chappell, Michele Leroux Bustamante and John Hooper (blogless MCS UK Architect) come to the session.  Here were some of the interesting discussion points that came up:

• There was some agreement that SOA is a pragmatic marketing term that unites many existing architectural principles around SOAP. 
• The closest agreement about a definition for SOA was that it was based around common architectural principles of encapsulation, loose coupling and messaging.
• There was some discussion about whether asynchronous messaging was a necessary part of service orientation. My feeling is that since you can achieve synchronous patterns over asynchronous communications that having asynchronous messaging capabilities is extremely useful.
• Although it's possible that these principles could be applied without SOAP, it's the fact that Microsoft, IBM, BEA and others have agreed that SOAP will be the lowest common denominator that is the pragmatic reason behind the current push for SOA.
• David made a point that service orientation will be whatever Indigo supports when it ships.  Shipping software always wins.  I think there's a lot of merit in this argument, but to the extent that SOA is based on generic architectural principles it is worth considering using these principles in systems that are design today (as Clemens demonstrated all conference).  If SOA principles help solve your business problems today then it's definitely worth starting today rather than waiting for Indigo.
• Some delegates were suspicious that using Indigo would enable them to interoperate with other systems that didn't use Indigo.  There was some confusion about the idea that Indigo will provide an object model that can be used to develop a system that has the capability to send messages between the systems that are based on interoperable WS-* specifications.
• The four tenets of service orientation are necessary but not sufficient for a system to be considered a service oriented architecture.  Some people thought they were too technologically focussed because they were tied too closely to XML technologies.
• David mentioned that the best SOA installation he'd seen was using CORBA several years ago - it had support for finding services, common schemas etc.  Michele backed up the need for shared industry schemas based on some of her experienced.
• John Hooper was interested in Pat Helland's assertion that services should not share transactions, which led into the difference between WS-Transactions (classic two-phase commit) and WS-BusinessActivity (compensating transactions).
• A delegate who was learning about the topic came up to the chairs and spoke about what he'd learnt at TechEd.  This was great to hear and generated a lot of discussion.  Clemens' presentations clearly had some impact with many in the audience.

Here's me with David Chappell after the session.  I've been a fan of his since reading Understanding ActiveX and OLE when I started Microsoft programming.

 

 

 

After writing about how WSE 2.0 can use policy and config files to secure web services with no lines of code, I was thinking about how 'magic' it seemed and had an aha! moment when I realised that this demonstrated the power of the Pipes and Filters pattern or aspect-oriented style approaches.  I believe that these approaches will play an important role in service-oriented applications in future.  Here are some recent quotes that back this up.

In my post about using Policy with WSE to create secure web services with no lines of code, I mentioned that this seemed like a good practical demonstration of an aspect-like approach, similar to the Pipes and Filters patterns from Gregor's book.  The policy filters are hooked into the incoming and outgoing messages pipelines are able to ensure that messages to and from the service conform to a particular policy, including retrieving tokens and signing and encrypting.  This means that the security of the service can be configured outside the service code, making for cleaner implementations.

Harry Pierson mentions that WS-Policy is aspect-like in his interview on TheServerSide.NET:

We do a lot of thinking around aspects in Web services, we just don’t call them aspects Web services, we call the Policy. All of the work around Policy is very aspect-like. If you’re spinning up a Web services with Web services enhancements, the new 2.0 stuff, there’s a Policy that defines, okay, if you want to talk to me, you have to be encrypted and digitally signed ... now I can communicate that to whomever is sending me a message. ... the WS-I engine can actually now say, okay, I’m expecting messages that [are signed and ecnrypted]  so I can actually enforce to make sure that those things have actually occurred before it ever reaches the business logic. So, that’s very aspect-like, and that’s going to be very critical going forward around service oriented architecture

Ted Neward writes about a recent presentation on Shadowfax and mentions it uses the concept of a pipeline of interceptors. He mentions that Shadowfax uses this approach to deliver functionality such as tracing, authorization, duplicate detection, instrumentation, authentication, authorizations and transactions.  Chris Garty started an interesting thread about this on the Shadowfax message board, where it was revealed that the Shadowfax team spent some time Gregor Kiczales (one of the creators of AspectJ).

This same pipeline/interception approach has been used in WSE, ASP.NET, Remoting and COM+.  Indigo will implement the same kind of approach using Channels and ChannelProviders.  I'm going to keep reading around this area to understand more about this approach and where it how it can be used successfully.

I'll be on .NET Rocks! tonight talking about WSE 2.0 and building web services with Microsoft technology today.  I'll be joined by fellow Regional Director and Commonwealth Citizen, John Bristowe.  If you can't listen live, it should be available streaming on Monday.

Update: As Carl has mentioned, the streaming/download version of the show has been posted.

Don Box and Doug Purdy did a 'keynote' for the Connected Systems Track.  They started out by asking what questions the audience wanted to see.  A great set of questions were proposed and the answers contained some of the most valuable content in the session.  Here are my notes on their answers, and some they didn't get time to do.

How does WSE 2.0 fit in with the Indigo direction?
It lets you use the protocols we have today.  WSE takes your ASMX investment and keeps you in the game as we do this protocol work.  If you don't track the protocols it may not be so important. 

Indigo will be the primary technology for using the WS-* specifications in future.  WSE takes your ASMX investment and lets you add support for those specifications today.

What's the future of DIME?
MTOM.  DIME was an experiment - we were on the wrong track that didn't support security.  Microsoft got together and did PASWA that became MTOM.  It will be in Indigo and other MS technologies.

WS-Security vs. SAML?
There are many different kinds of tokens that may be used, such as Username, X509 certificates and Kerberos tokens.  Don said it was unlikely that a token type, like SAML will become the 'single token format to rule them all'.  No definite answer on where the SAML support was.  As I learnt on Saturday, trying to implement SAML support is a non-trivial exercise - it would be nice if there was a clear statement from Microsoft on when it will be supported in the platform (so that you don't have to share my dll in order for us to use it when we talk).  I think it will be part of the identity management work in future.

How successful is WSE at interop?
Microsoft do bake-offs with WSE where they get all the vendors in a room and try and make the specifications work.  There wasn't a definite answer other than this.

How do you talk SOAP from a Windows Service?
Don's answer was that you do the hard work to host ASMX inside a service then put an ASMX façade and call into the service with ES or Remoting.

I thought this missed the point that the recently released WSE 2.0 supports Soap Messaging, which allows you to implement SOAP messaging over TCP.  I think this would be a far easier way of hosting SOAP within a windows services.

What is the technology to replace COM+ in the long term?
ES investment will keep working.  Deployment, interception and synchronization are being brought forward into Indigo.  Many of the ES semantics are a direct correlation with the Indigo model.  Doug mentioned that ES programmers will be the most prepared to work with Indigo when it ships.

Is there an issue with the verbosity of web services payload?
Don's answer was that Indigo will 'negotiate up' and switch to a faster way of communicating if the other endpoint uses Indigo.  How they do this is to be seen (there were comments at the PDC that the first Indigo call will be a policy request to see if the other end is an Indigo endpoint).  Don mentioned that the industry is having a hard time defining binary protocols that allow user definition are difficult.  Binary protocols that support user defined structures are hard.

How do we discover services and determine policy at run time?
Don mentioned that UDDI was a solution you could use today.  In a show of hands only 4 of 200 attendees were using UDDI (2% adoption?).  According to Don it makes some hapy, but some customers want more.  They want a more flexible model for describing things without having to use the tModel (which is hard to grok).  There also other groups want to discover services on devices, so WS-Discovery is where Microsoft are headed.  It is a small spec that is easy to understand that can be easily implemented.

Will we need to continue to be plumbers to do web services security?
I thought this was a great answer:

For a while.  WSE makes it easier, but if things go wrong you'll need a plumber.  We have not done our job in Indigo if people have to understand the protocols.  Your common developer needs to solve business problems, not the protocol problems.  Indigo is adding value without focussing on the protocols. 

No matter how good WSE does, since we are ironing out the interop you'll still need to read WS-Sec.  Now at least we only need plumbers when things go wrong.

What's the migration path to SO?
This was really the content of Richard Turner's talk in the track, he's also written a great post on detailing prescriptive guidance on preparing to upgrade to Indigo.  The basic message is don't do tricky things.  If you are doing something that was hard to figure out, maybe that was for a reason.  So things like SoapExtensions or custom message sinks in Remoting are not going to upgrade well.  There's lots of material out there that shows that Microsoft have a 'good story' on upgrading from various technologies.

Unanswered questions:

• What is the MSMQ equivalent for COM+?  Is MSMQ going away?
• How does EIF fit into SOA?
• Strategies for native to managed interop?
• Will Indigo support mailslots?
• Are there any application blocks for SOA?
• How do we handle events across appdomains?
• Security - you have authentication and authorization - what about any attacks through the channel - filtering content? Content-awareness in firewalls?

Joshua Block a senior staff engineer at Sun and author of Effective Java gave a great session on API design.  Joshua highlighted that good API can be great company assets because good APIs capture and retain customers.  The talk was particularly enjoyable because Joshua illustrated his points with reference to good and bad Java APIs.  You can read the full session write up by James Robertson.

 

I love the challenge of API design.  As Joshua mentioned, it's incredibly hard, and when working on large public projects like .NET or Java you only really get one shot at it.  Once it's out there it's extremely difficult to make changes to it.

 

This is where I think the Indigo team is on a great thing.  Steve Swartz describes his role as making sure that the Indigo programming model is easy to use.  Don Box mentions today that he sees his role as removing the need to have to absorb complex detail in order to create working programs.  This team knows how important it is to get the API correct.  As Don said in January "the APIs are the real lock in".

 

I was surprised that Joshua didn't say more about the need to test the APIs with a developer audience.  He did mention that when designing and interface for others to provide implementations for, it's worth using the 'rule of threes' and writing three implementations yourself before shipping the API to ensure that it's possible.  But he didn't really say anything about API usability testing, the use of personas or the need to get APIs out into the community for feedback.  These are all things that I believe Microsoft is doing incredibly well with Longhorn (I'm not really in a position to say regarding Java, but would be interested to know what goes on there).

 

Joshua did mentioned the importance of documentation on an API.  Without documentation it means that developers have to guess at what a method does.  This is where I really like Visual Studio .NET's support for code comments in C#.  Together with the examples in MSDN documentation this makes writing C# code much more enjoyable.

 

An audience member asked Joshua whether he was bitter about Microsoft copying C# and learning from all his mistakes.  His answer was that he wasn't a bitter man but he had wished that Microsoft had learnt from all of his mistakes rather than repeating some of the same ones in the .NET Framework.

I've been blog-lite while I prepared and then gave my Indigo presentation to the London .NET User Group.  The audience seemed to enjoy it and I had a good time presenting it.  Some quick points:

• The selling point of the Indigo Service model is that it is simple and easy to use.  The benefits for developers is that they get all the benefits of enterprise-quality features such as transactions, reliable messaging and security without having to write any code.  Letting Microsoft do the plumbing allows developers more time to write code that focuses on solving complex and interesting business problems.
• I took on board Clemens' experience and started talking about the 'why' of distributed technologies before delving into how Indigo works.  Luckily this group was very switched on.  I got some good questions about the performance of Indigo across AppDomains/Processes/Machines.
• Someone asked me to provide some background on why Enterprise Services are a better choice than Remoting for communicating between components on the local machine.  Clemens' has some more information on it here and here. 

Doing a presentation where you don't have the current version of the software to demo is a challenge (the PDC Indigo bits are from an older, M4 build, which has been substantially refactored in the now-being-coded M5 build, so there's not a lot of value in showing demos with the PDC bit).  In order to get some code samples I had to transcribe screen shots from the PDC videos (difficult since Steve Swartz didn't use word-wrap in Visual Studio).  I'm filled with (even more) respect for Don Box and Steve Swartz's presentation skills after realizing they managed to do four Indigo presentations at the PDC without even compiling, let alone running any of the applications.

Given the PowerPoint dependence and Clemens' reports on the complexity of the topic I decide I'd have to resort to audience bribery.  I drew on my experience in TheatreSports and brought several bags of Minties from the Australia Shop at Covent Garden (to my horror I discovered today that they are made in New Zealand!) and threw them to the audience whenever I detected the signs of PowerPoint-induced lethargy.

John Udell has written an article on WS Alphabet Soup based on his discussions with John Schewuck, an architect on the Indigo team.  Here are the interesting points:

• Indigo may ship before Longhorn to enable people to provide developers with access to the functionality provided by the newer WS-* specs.  I know the message from the PDC was Whidbey < Indigo <= Longhorn, but I guess haven't really considered it will come before Longhorn (I thought this was what WSE was for since Indigo is a 'baked in' part of the .NET Framework).  Not that I'd complain if it came earlier.
• There are a lot of Web Service (WS-*) specifications, but this is a good thing because they are composable.  You only need to implement the specifications you need.  The alternative would be a single specification that tries to do everything.
• The WS-* specs are inventing a lot of what has been provided before in DCOM and CORBA.  As Don Box says, 'The entire motivation for defining SOAP was to get the industry out of the "you need my runtime to talk to me" business that DCOM and Java RMI so happily got us into in the 1990's.'
• The advantage of using SOAP as a message description format  is that it provides extensible SOAP headers ("an accordion-like construct").  Previous network communications protocols was their dependence on fixed-offset dependencies that broke if you had to extend or modify the packets.
• Microsoft is wise to spend time worrying about the usability of its APIs.  This was my initial reaction at the PDC - 'Wow, they have done a lot to make this really easy for developers'.
• The key to reducing the cognitive overload caused by the specifications is an easy programming model (this acknowledges the reality that not everyone gets as excited as John Bristowe about a new specification).  Indigo's use of attributes in the Service Model is a major plus here.  How awesome is it to be able to write code like this to add WS-Security (signing, encryption), WS-ReliableMessaging and WS-AtomicTransaction support:

[Service]
[Security(Disabled=false, 
 SecurityProfileName="MyDynamicProfile",        
 ClientAuthentication=UsageMode.Required,
 IntegritySupport=UsageMode.Required,
 ConfidentialitySupport=UsageMode.Required)]
[ContextPropagation(typeof(ITransaction)]
[Connection(DeliveryAssurances.Full)]
public class MySecureReliableTransactedService
{
    [ServiceMethod]
    [Authorization(Role="Clerk")] 
    public void CreditAccount(string acc, decimal amount)
    {
        …
    }
}

The Loosely Coupled blog also has more commentary on the interview with John.

Matthew Adams left a comment on my last post disagree with my characterisation about services being more to do with messages as opposed to remote procedure calls (RPC).   It was a good question that I thought was worthy of a lengthier response.

What I was suggesting in the original post is that overtime there will continue to be a move towards services that use messages, and in particular SOAP messages based on the XML Infoset, rather than RPC style calls over time.  SOAP because it gives a broader reach for interoperability because of it's lowest common denominator approach and associated standards.  Messaging because it gives more flexibility in terms of what is sent and the message exchange patterns that are supported. 

The Great SOA Definition Debate
I've been trying to get away for getting into a debate about what services are, mainly because I haven't found a useful definition of service that says more than 'it's a great idea to have blocks of code that do something' and when designing these blocks of code, encapsulation and loose coupling are good practices.

By Matthew's definition a service is something that can do something, at a particular addressable location on a particular protocol, with a particular message format, in a such a way as the consumer doesn't have to care about how that something is done.

Using this definition of a 'service' it is true that the service doesn't have to be implemented as a 'message' end-point.  It is possible to implement these 'services' in ASMX web services, in-process .NET classes and remotable objects.  The problem is that these choices can impact the amount that the consumer has to know about the way the service is called, if not how it is implemented.

Don's Definition of Services
I like Don Box's definition of a service as a 'program that you communicate with using messages'.  These messages are SOAP messages based on the XML Infoset (a logical view of an XML document - angle brackets and text representations may or may not be involved).

The beauty of using SOAP messaging is that it is a lowest-common denominator of solving the problems involved with providing a service:

• How do we address the service, which protocol should we use to talk to it? (WS-Addressing)
• How do we describe the format of the messages that are sent and received? (SOAP)
• How do we describe the format of the contents of these messages (WSDL, XML Schema)
• How do we describe the pattern of the messages that are exchanged? (WSDL)
• How do we describe the other characteristics we need to successfully talk (WS-Policy)

What's wrong with RPC end points?
My main issue with RPC end points is interoperability and flexibily.  RPC-style calls to RPC-end points in Matthew's terminology calls can do all of these things, but they are more likely to do it in a less-interoperable way that requires greater knowledge of the underlying platform, protocol or programming model (for example you may need to agree on the binary representation of the parameters used in the procedure call which may mean sharing DLL's or worse, an entire development platform).

Of course any remote procedure call can be thought of as just a particular kind of message.  So it's possible to create services that are based around RPC-style calls made within a message (which is how ASMX web services work today).  Within web services there are two types of SOAP message format - RPC and document.  As Yasser Shohoud points out in his article RPC/Literal and Freedom of Choice, RPC formatted SOAP messages are just a subset of the document formatted (I'd argue message-based) SOAP messages .  As he says, ASMX web services show it's possible to use a document style message format but with an RPC programming model.

[Update: See Shawn Smith's post on this topic.  In the last section of this post I wasn't very clear about the distinction between SOAP message formats and message exchange patterns.  My point was that it is a better idea to implement more flexible approaches to both the message format and the message exchange patterns because the more flexible approach can accommodate the more rigid approach.  In this sense the SOAP Document format can accommodate the SOAP RPC message format and the asynchronous message exchange patterns can accommodate RPC message exchange patterns (synchronous request/reply).

Ultimately, it's not worth getting too fussed up about this as both the ASMX and Indigo programming model will support you whichever way you'd like to work.  You can write services based around asynchronous messages or using synchronous request/response (stop-and-wait) RPC-style calls.  As I said at the start of this post, over time I believe there will be a movement away from synchronous RPC calls towards asynchronous messaging.]

As preparation for my Indigo presentation at the London .NET User Group next Monday, I thought I'd write a couple of posts about Indigo, starting with 'The Road To Indigo'.

Indigo is a new technology that represents on the unification of existing distributed application technology.  It takes Remoting, Enterprise Services, Web Services and Microsoft Message Queue and combines them into a single area of platform-level technology.  In fact, Microsoft has literally taken all of the teams responsible for these technologies and put them in the same building.  Instead of thinking of Indigo as a completely new technology, think of it as the next version of all the existing approaches where all of the best features of each are combined into a single managed-code programming model.  Here's a diagram showing the progression of these technologies:

 

Indigo takes the best features of each approach and makes them available in one location.  If you're developing ASMX and Web Services today you can't  take advantage of reliable messaging (as in MSMQ) or transactions and object pooling (as in Enterprise Services) across web services calls.  With Indigo it is possible.  If you're an Enterprise Services developer today it's difficult to take advantage of flexible versioning provided by schema or being able to describe requirements using policy files (as you can in web services with WSE).  Indigo makes it possible to take advantage of all of these features.

Indigo also lets you chose the way you want to build applications without tying you in.  Indigo supports and encourages building service-oriented applications using messages based on SOAP.  However, Indigo still supports you and provides features even if you want to create distributed applications using a distributed component approach using remote procedure calls.  Choosing one approach over the other doesn't mean giving up features such as message-level encryption or transactions.  As Ingo Rammer says, Indigo recognises that you shouldn't be locked into 'a specific programming model, a serialization model, a transport model, or even just a serialization format'.

Combining the features of these four different technology areas into a single programming model is great news for developers.  We get the best features from each previous approach while only having to learn one programming model!  There's no more complexity about which technology or programming model to use for a particular distributed application scenario. It's the same programming model whether it's a peer-to-peer application on a hand-held device, right up to a secure, reliable and transacted message queue behind a massive online retailers.  

For those times when the platform needs to be extended, Indigo also provides an excellent extensibility model.  Even in this case there's only one extensibility model to learn.

Indigo brings us closer to Microsoft's long-term aim of helping developers spend less time on the plumbing and more time on the application code.

Bruce Williams from the Indigo team has a nice walk through of the the XML elements in WS-Eventing.  When you strip away all of the namespaces and XML noise, it's a nice, understandable piece of work.

I like WS-Eventing because it's another tool in the architecture toolbox.  To me it suggests the end of polling in web service applications.  In some of the applications I've worked on today the client has had to poll the webservice to check for updates.  With eventing the client will be able to wait to be told if something has changed, rather than pestering the server like a kid in the back of a car on a long road trip asking 'are we there yet?' all of the time.   It's much more efficient to wait and be notified than constantly having to poll to check if something has changed.

I'm not 100% sure about how large corporates will handle security will work across the Internet and corporate firewalls.  For the client to receive the notification they have to have an open port that the notification service can send to.  Perhaps a solution is to use an approach like the MSDN SoapMail sample, where messages were returned to some central location, like a mailbox, which is polled by another service that picks the message up and forwards it to the client.  Perhaps the answer lies in the Security Considerations area of the spec which suggests the notification should be signed with security tokens.

When I was clarifying the Microsoft Messaging Message I said I'd watch the PDC DAT 420 presentation on BizTalk and Indigo.  As my PDC DVDs arrived this week I didn't have to download the presentation I finally got around to watching it.  Here are the key points I took from the presentation:

• From a BizTalk perspective, Indigo is the vehicle that will provide secure, reliable transacted services (over more than just http).
• Indigo is an API, BizTalk is a set of tools.  Indigo will natively integrated into the next version of BizTalk after BizTalk 2004.  The demos showed a prototype Indigo adapter that worked with BizTalk 2004.
• BizTalk has been achieving reliable services through the BizTalk Framework 2.  In future this will be deprecated and replaced with WS-ReliableMessaging, which will be provided by Indigo.
• BizTalk will use the security model built into Indigo to enable WS-Security support for Username tokens and X509 certificates that can be easily configured through attributes, configuration files and policy.

The presentation helped me understand that Indigo is the plumbing.  A lot of business functionality will require the use of tools that take advantage of that plumbing, such as BizTalk.  So I was interested a few days later to see an announcement that the BizTalk Orchestration will ship as part of Indigo in Longhorn.  As the article points out, BizTalk seems to debut features that eventually become part of the plumbing. 

I really enjoyed the presentation, and not just because Scott Woodgate has such a pleasing accent to listen to (New Zealand, so similar to Australian with stranger vowels).   The BizTalk 2004 health and activity monitor combined with the Orchestration debugger were great features.  I can imagine that these would reduce a lot of the pain with administrating and developing complex service-oriented systems.  I'm hoping that I'll get to benefit from these tools on future projects.

Suresh Kumar has posted a newsgroup exchange from Don Box and Scott Woodgate that cover most of the points from the talk on how BizTalk will take advantage of the features provided by Indigo.  Suresh also has some other good posts on general BizTalk 2004 resources, BizTalk and Publish/Subscribe and XML Web Services in BizTalk 2004 including talk of a WSE adapter for BizTalk later this year.   If I could find his RSS feed I'd subscribe.

Tonight was my first bloggers dinner and quite a night.  I was the first one to sign up and the last one to arrive as Chris Anderson worked out.  What was most surprising was that other than Chris, Don and Ian 'I have no blog comments because I hand-coded my blog myself' Griffiths, I only met a few other bloggers (Chris has a better list).  A lot of blow-in blog readers!  But all are welcome and everyone's company was appreciated.

By the time I rolled up the only chair left was next to Don himself.  Luckily I'd boned up on things to talk about (John 'Policy' Bristowe also primed with a couple of last questions via instant messenge) and managed to garner the following:

• I started off with asking how do you communicate semantic intent with policy?  Don's response was: you don't.  You do it out of band with the telephone.  It's what humans are for.  There are a lot of people trying to apply formal methods to determine the structural correctness of the messages, but the intent is something that humans will work out.
• WS-Addressing.  I raised my beef about it being all well an good to have an opaque string as the address, but where was the protocol binding.  Apparently Indigo is going to use the simple URL style scheme, so a tcp address will look like tcp://domainname etc.
• The important of the Endpoint references is to provide a SOAP cookie style mechanism without having to get into the URL hackery such as query strings.
• Someone said "it just felt unusual to send text-based messages around - isn't that slow?".  Don's response was that the first time he had sex it felt weird, the next time better and now he couldn't imagine it any other way.
• What's different between the M4 and M5 releases of Indigo (Clemens seems to know a lot about this)?  Apparently the programming model has been refactored toto make it more unified and simpler (great news here).  The System.MessageBus.Servicess library has changed significantly because what was there was just not as good as it could have been.  I hope this means better terminology than DatagramPortTypeAttribute and DialogPortTypeAttribute, which always seemed a bit complex to me.
• What's the best way to approach Indigo today?  If you want to learn about indigo today in a way that will pay of in the future, don't worry too much about the high level details.  If you work from the wire level back up it will be most useful (this is least likely to change). 
• The focus with V1 of Indigo is proving that they can get great performance with messaging.
• We discussed Clemens' idea that because Indigo uses a streaming reader over the message body, it may be possible that on intermediary/endpoint may start sending onto another intermediary/endpoint before the full message is received.  Don mentioned that there's no reason to think that the message may ever finish.  Queue zen-like moment of poignant reflection and silence.  'Sort of like one giant Congo-line of a message throughout the Internet' I said.  Don didn't say it but I felt him think 'Yes, Grashopper'.
• We spoke about the fact that my pregnant wife who has always been worried about the size of my head, thought that Don was the only person I have on my PDC photowall that had a larger head than me.  Don said his head was larger than mine in more ways than one.  We agreed that his head was impressive, but he was leaning forward in the photo and clearly his head was not awesomely large (at least physically).
• I suggested Don try the Cinnamon Club for dinner.  We had the Axxiant Xmas dinner there and it was very good (though it was late and I had been drinking ... ).

Afterwards some of us moved off to the pub ...

In preparation for the London Bloggers Dinner with Don Box and Chris Anderson on Monday (see the report of this dinner, and what went on at the pub afterwards), I thought I’d get up to speed and make some notes of the key points made in Don’s interview on the ServerSide.NET.  To my mind this is the best interview I've seen or read about Indigo.

I'm posting these notes mainly so that I can have them handy in SharpReader when I'm offline and in case others might find them useful.  You can see the interview and download the full transcript on the ServerSide.NET site.

What is Indigo? 

• A connective tissue between programs.  It makes it simple for developers to put a message oriented façade on the edge of your application to allow easy communication with others.

Positioning Remoting and ASMX web services

• The problem with .NET remoting was that even though it supported interfaces, it really worked best when both ends of the pipe shared code such as assemblies or DLLs to integrate.  Sharing types like this turns out hard to manage because you often don’t have control over the deployment versioning and testing at each end of the pipe.
• ASMX Web Services (including WSE) are more flexible because they use WSDL and schema which is more adaptable to change than the OO style type interface used in remoting.  ASMX is proving more popular than remoting in the field because it is actually a simpler model and simplicity is a feature.
• Developers should go down the path of integration using messaging because it is simpler and more flexible, but support for remoting won’t go away.
• You don’t have to use XML Schemas and WSDL to describe structures and contracts, but that’s the practical way that everybody does it.
• The problem with components was that they tried to do the right thing separating interface and implementation, but it was too easy to ‘leak’ object concepts and require the sharing of types.

Do components have a place in the world?  

• Don argues that the world is simpler if you think about objects, which are great for programming software that gets integrated, tested and deployed, more or less as an atomic unit and services, which are the central abstraction for deployed, autonomous pieces of software.  Given this perspective he suggests there may not be a place for components.

What’s the difference between service-oriented programming and service oriented architecture?

• SOA is the buzz word du jour, system message bus will probably be next, whereas service-oriented programming is more focussed on the reality of the code.
• Service oriented programming is about four ideas:

1. Boundaries are explicit.  In Indigo you have to place attributes on your code that you want publicly exposed
2. Services are autonomous.  Indigo doesn’t assume you’ll deploy a whole system as a single unit.  Each service should be secure and reliable as a stand-alone unit.
3. Share schema and contract, not class.  It’s too easy to mix up interfaces and implementation, so it’s a better idea to separate schema and contract into two distinct ideas.  Schemas are defined in XML Schema language, and contracts, which are message exchange patterns, are defined in WSDL.
4. Compatibility based on Policy.  To know that services are compatible you have to care about other things than just the signature of the messages.  WS-Policy is supported in Indigo and provides a way to share this over the wire in an interoperable way.

Does Policy impose versioning restrictions?

• Policy is similar to marker interfaces in Java or C#.  You have to have a stable name that is immutable that is decoupled from the actual signatures.  Both parties have to agree on the name and it has very strict version requirements.
• WS-Policy provides a way to declare compatibility with any number of versions of a given policy assertion.  The logic engine is able to determine if party A’s assertion lines up with party B’s assertion.
• In the PDC bits you can see the first message sent on the wire is a ‘get policy’ message.

Can policy be used to create versioning schemas?

• There are three policy specs:

• WS-Policy which is the XML form for writing a logic expression
• WS-PolicyAttachment, which provides a way of taking those expressions and applying them to a particular domain
• WS-PolicyAssertion which contains a small number of concrete assertions that plug into the expressions defined in WS-Policy.

• WS-PolicyAssertion contains an assertion called Spec Version that can be used to determine compatibility at the naming level.  It’s looser than strict or nominal type equivalence that is used in C# and Java.  It provides some flexibility, not complete flexibility and Policy is used to do the version negotiation.

Are there situations where the Indigo approach doesn’t work well?

• In future the service-oriented programming model may be a practical way of doing things like cross-process, cross-app-domain or even within a single exe.
• The focus is making sure Indigo has a solid model that can be implemented and scale down well.  Objects took a ‘near’ metaphor and tried to stretch it out, Indigo is trying to take a ‘far’ metaphor and shrink it.  Indigo is striving to have very low performance costs within a single exe.

What are the performance optimizations being worked on now?

• The team is very focused on make sure that the I/O implementation is as well tuned as possible.  They want to make sure we can get bytes in, in XML and other forms, in to and out of an app domain as fast as humanly possible. 
• They want to make sure that it’s not tied to a specific protocol, such as HTTP.
• The idea is ‘I create a message here, I need it to appear in your program as fast as possible’.
• Performance is an important goal for the first release.

WSE and Indigo

• WSE is the vehicle to give developers access to implementations of the WS-* protocols as quickly as possible.
• The goal is still to integrate them into the .NET platform, but sometimes the platform has to ship before standards are set, which is why WSE will have a future post-Indigo.
• Post-Indigo WSE may be used to track protocol evolution on top of the Indigo infrastructure rather than just the ASMX infrastructure.

Will Indigo do anything new at the wire level?

• Indigo is about the software productization of ideas (such as the WS-* specs) rather than interesting new wire protocols.  For example, transactions on the wire will use WS-AtomicTransactions but within the service they are doing a lot to make transactions easier to use from managed code with extremely efficient implementations.