Web services are being adopted in the wild
There certainly is a lot of material to cover in web services from the specifications to the implementation, so we only briefly touched on Indigo.  There were 30 or so developers there. Starting with the mandatory 'polling' questions I was surprised that around 60% of them had used web services already.  Of that group about a third had added some form of security (HTTPS, VPN or internal network).  There were some good questions in the group ('Can I use WS-Security in a PHP application?  Will Microsoft compete with Tibco and the EAI vendors with Indigo and/or BizTalk?  Has Don Box stopped using the term GXA?)

Blogs are a great way to learn about Web Services
I mentioned that blogging is a great way of tracking what's happening web services. I showed how:

• Omri Gazitts a PM on the Indigo team, talks about web services attachements and that DIME is still the best choice for today, but MTOM is Microsoft's preferred approach for the future.
• Hervey Wilson, WSE lead developer, has said that WS-ReliableMessaging is coming in a future release of WSE and writes great content about lots of other topics.
• Rebecca Dias, Product Manager for WSE, has mentioned as a result of feedback at the PDC, "Attachments will be abstracted away [in WSE 2.0] so that migration to MTOM will be more seemless(sic)"

How to get started reading blogs
First download a news aggregator like SharpReader or RSSBandit then download my modified OPML file.  It contains many Microsoft bloggers as well as other web service related blogs (including those on Matt Powell's list).

I also promised a link to the code that downloads all of the PDC powerpoint presentations.

As part of a new consulting engagement I started today, I found out that the latest version of the .NET Solution Build & Deployment Process & Tools has a visual designer for MSBuild Files.  The Tools are writtien by Andy Reeves, from Microsoft UK's Solution Development Centre and were the basis for the extensions included with MSBuild.  The download page also has more about why MSBuild was chosen over NAnt.  John Lam has also posted about the NAnt community's reaction to MSBuild.

The new engagement involves 4 hours train travel each day, so I'm going to have to kurb my blogging to get to bed.

Some ThoughtWorks experience with TDD and Refactoring
Dan North, a ThoughtWorker, has an article in the Java Developers Journal called Test-Driven Development Is Not About Testing,  his major point is that 'it's not about the tests; it's about seeing how little you actually need to do and how cleanly you can do it!'  Jeremy Stell-Smith, also ThoughtWorker, has a reply post TddIsOnlyOnePieceOfThePuzzle where he makes the following statements about TDD:

• TDD is a tactical thing.  It helps developers from adding unnecessary complexity and allows them to think about how the code will be used before other developers use it.
• It's important to refactor while using TDD to reduce duplication to keep the system clean.
• If a system has good automated testing and loose relationships between classes, then tests can give good feedback about the extent of damage that a change to the system might bring.
• Tests work best when they test small isolated areas of functionality that shouldn't break if anything else changes.  For this reason, unit tests are easier to manage that end-to-end integration tests

There are also some points that Jeremy makes that seem a bit 'religious' such as 'changing your mind later is about as expensive as changing it later'.  This just doesn't seem rational, but perhaps I'm just arguing for a shades of grey rather than this black and white style statement.  I know that TDD makes it easier to make a change, or to measure the impact of a change, but it's still better to make changes as early as possible because it is more expensive to make changes later.  However, I think that his assertion that 'tests must be as fine grained as possible so that most changes no matter how radical don't affect many tests' seems a fine goal, but is hard to achieve.

Early Microsoft experience with TDD
Chris Anderson points to Randy, a friend of his within Microsoft who's reflecting on his experience with TDD.  Randy says that it takes a lot of discipline to stop writing code before a test case that uses it and that Visual Studio is a hindrance when working test first because the intellisense stops being helpful when the writing test code for methods that don't yet exist (Roy wrote a good post showing how IntelliJ handles this situation in a much cleverer way, hopefull Microsoft will pick this up and copy extend it).  Randy also talks highlights how TDD can be difficult:

 ... it's a completely different ball game when you're changing the behavior of code as opposed to writing entirely new code. Say you've written a sizable body of code with inter-related classes and non-trivial implementations of whatever. Now suppose you want to make a fundamental change to one of the more central class's behaviors ...  substantial build breaks where methods no longer exist, because the functionality they provided has been made unnecessary ... doing test first in this case is tricky. You can start by changing the test that exercises the fundamental class in the new way, and then changing the fundamental class to pass the test. Unfortunately, then you've got build breaks across your entire project, and fixing those involve work that should also be done in test-first fashion. The way I did this was to exclude all the other files from the project, get the fundamental tests working again, and slowly re-introduce the other functionality, tests first of course, fixing/updating as I go. Maybe it's the same amount of work in the end, but it feels more cumbersome

From my experience and the developers I've talked to at the Extreme Tuesday Club, if the code is difficult to test then it can highlight weaknesses in the design of the code (such as it being too tightly coupled).  Put another way, designing code with testing in mind often leads to code that is better designed.  The pain that Randy highlights when working with inter-related classes encourages a designs that are more loosely-related so that they can be more easily tested. 

Mock Objects is an approach to making testing easier in large complex systems.  As the authors of the definitive paper on Mock Objects say testing can improve code "by preserving encapsulation, reducing global dependencies, and clarifying the interactions between classes."  More on that in a future post.

Browsing the Longhorn newsgroups, I found a post by David Solomon saying that the Inside Windows 2000/XP/Server 2003 Interactive Video Tutorials CD-ROM was included in the PDC bags.  I had seen this CD in its innocuous blue cover, but I didn't realise that has 12 hours of video presentations, as well as an e-book version of Inside Windows 2000 version 3, and retails for US$950.

The content is really good.  This is great stuff for me as I need to improve my knowledge of Windows internals.  I would have liked to have attended Dave's pre-conference but I felt I had to hear the Web Services story.  Now with the videos on my laptop I can go through the material at my own pace. I'll never be bored on a train again.

Chris Sells and Larry O'Brien are both having a crisis of confidence over their role as authors or information providers in the new world of blogging, Google and RSS.  I think its a good time to look at what value they add and what's important in helping developers learn. Says Chris in Time For An Exciting Career In Electronics:

what I'm best at in the whole world has either become completely unnecessary or incredibly easy. The skill that I spent most of the last 10 years learning was to look at a group of types, APIs, reference docs, headers, source code, etc., distill it down to a set of architectural intensions and then to weave a story through the whole thing to make sense of it for developers that weren't able to glean the intensions themselves (most often because they had real jobs). I call this exhaustive search for intention officially at an end. I'm not longer needed except as a gatherer for those not yet facile in RSS aggregators (which, luckily for me, is still a large number : ).

And Larry in 'Being run over by the cluetrain':

With the whole gamut of Internet-based communication (Websites, newsgroups and mailing lists, Google, email, and blogs), the typical path between technical question and answer has become much more direct. ... Independent authors and teachers have traditionally exploited the very inefficiencies that are being paved over by these technologies. The community no longer has the same incentive to pay money for magazines, books, seminars, and mentoring / consulting: they get the same substance faster and cheaper, if perhaps not with the same style, context, and specificity.

The mistake being made here is to think that the value of authors is based around converying factual knowledge.  The advent of encylcopedias didn't reduce the need for teachers.  There's a big difference between transferring information and helping people learn.   I think it's a great thing that technical authors can now move up the tree away from technical answers and onto higher order tasks like helping people learn how to adapt to the new technologies (that seem to be changing every three years).

I also think both of these guys sell themselves short.  Even though Chris knows about the intentions behind the software, there's always a market for someone who can tell a story and transfer this information to others.  At the PDC, most of what Don had to say has been written somewhere else previously, but I'll still pay for the experience of hearing him tell the story.  Heck, I've read most of Chris' online articles, but I still paid for his latest book (I wanted the convenience of the book and I wanted to have him tell me the whole story about windows forms, not just part of it)

I'd  also say the valuable skill that Chris has is not his own deep knowledge of the intent behind APIs, but his ability to transfer this knowledge of the intent to me.  Larry's book has the right idea - 'Thinking in .NET', not 101 code examples.  I want these guys to help me think, and learn how to think about the problem, not just to find the answer.

What's clear to me, after 19 Microsoft Certified Professional exams (including sitting the core MCSD exam for the third time), is that knowledge expires fairly quickly, whereas knowing how to learn continues to pay back.  This is exactly what Chris has written about in Learning How To Learn.  I can't believe he thinks he'll be replaced just because some of the MS tech team are blogging themselves.   Maybe he just needs a hug.

When I met the Longhorn Shell Team at Meet The Experts at the PDC I told them I had read Adam Bar's 'Proudly Serving My Corporate Masters', a great book on his 10 years as a developer at Microsoft (that's available to read online).  He left Microsoft frustrated with the fact that Windows never had a proper shell environment.  Well, it turns out that Adam's going back to Microsoft to join the Longhorn Shell Team.

Others have also been impressed with the new shell capabilities, code named MONAD, specifically the ability to deal with objects rather than text streams.  Larry O'Brien says it was the best demo he saw at the PDC, even though the demo was on a bus to venue. As he says 'sysadmin scripting: priceless'

Don Box is a great speaker, but as Scott Hanselman and Steve Maine and this post demonstrate, sometimes understanding and explaining the reasons behind his tenets can be difficult.  As an intellectual entertainment and learning exercise, I'm throwing my hat into the ring, trying to answer a commentor on Scott's website who asked what is meant by Don's tenet of 'Share Schema, Not Type' (actually it was 'share schema, not class'), or as I like to ask 'Why should I use web services (with schema and WSDL) over .NET remoting or DCOM?'.

Steve's Answer
Steve's answer goes like this:

• Types came from the land of COM and were all about binary implementations of interfaces.  Interfaces described the methods that could be called on an object, so they were all about behaviour.
• There were two problems with Types.  Firstly, since they just described behaviour, it was hard to make sure that all implementations of the interface produced the same behaviour (that they were semantically consistent).  Secondly, that once published the interface was immutable and hard to extend.
• Schema are from the land of Web Services and are all about describing the layout of a message. Schema communicate state and nothing about behaviour.  Contracts are the way that schema describe behaviour by specifying the format of input and output of messages that are sent to a service.
• Schema solve the problems of Types.  Schema can extend types by using the open content model.  Schema can ensure semantic consistency across implementations by specifying the schema of the input and output messages.

I like Steve's story, but I'm not sure it's the answer.  The open content model is one approach for extending schemas, though it still works like COM and requires that the newest version provide all of the content required for all previous versions, so I'm not sure how much of a victory this is for the schema approach.  Also, I'm not sure of the differences in terms of semantic consistency, between a COM interface that defines the types of input parameter and output parameters from a method, and contracts based on schema that defines the format of input and output messages of a service. 

Scott's Answer
Scott's answer was that types are unique and immutable and require the sharing of an assembly, whereas schema is a description of the XML content that acts like a contract between parties that can be used without the need for an assembly.  I think this emphasis on avoiding a binary assembly is closer to what Don was talking about.

What was Don thinking?
In Don's PDC Session “Indigo": Services and the Future of Distributed Applications, (slides here) session, Don's tenet was 'Share schema, not class: Integration based on message formats and exchange patterns, not classes and objects' (see Tenet 3 of  my transcript of Don's talk at the PDC, which now seems less comprehensible than I'd hoped).  In his MSDN article Don restates it as "Services share schema and contract, not class". 

"Classes are convenient abstractions as they share both structure and behavior in a single named unit. Service-oriented development has no such construct. Rather, services interact based solely on schemas (for structures) and contracts (for behaviors). Every service advertises a contract that describes the structure of messages it can send and/or receive as well as some degree of ordering constraints over those messages. This strict separation between structure and behavior vastly simplifies deployment"

So, I think the advice behind Don's tenet is to build services using the service-oriented approach (using schema and WSDL to bind the input and output message) rather than an object oriented approach (DCOM or .NET remoting).  This is because it is easier to evolve the structure of the input and output messages using schema rather than having to redeploying an assembly or a binary interface description (*.tlb).

Don's argument is that service-oriented approaches can evolve more flexibly because they can use "features such as XML element wildcards (like xsd:any) and optional SOAP header blocks to evolve a service in ways that do not break already deployed code."  I'm not entirely convinced that COM didn't provide a similar level of flexibility as the xsd:any wildcard (e.g. using As Object in VB).  The optional SOAP headers seems feasible.

What do I think?
I think that interoperability across binary platforms is the key reason to favour schema and contracts over classes.  Object-orientation and types are all about binary representations that are just too difficult to get all of the binary platform vendors to agree on.  Schema provides a machine verifiable mechanism that currently uses text representations which are the lowest common denominator and can be implemented across all platforms.

Within a platform, I think using schema and contracts (web services) over classes (DCOM/remoting) wins on pragmatic grounds because schema and contracts seem easier to deploy (no binary object copying required), debug (it's easy to sniff the packets on the wire to work out what's going on) and evolve (using schema and wildcards seem easier to work with than interfaces in COM, especially when with the XML versioning improvments in Whidbey).

Well, it's been an amazing PDC.  Before I came I thought it would be mostly about the sessions and the technology, but it's also about catching up with people, making new relationships and socialising.  So many times I've thought that I'll have to wait for the DVD to review the material and let it all settle in my head.  The time here has mostly been about understanding the general directions and seeing the motivations and directions for the technology.  As one friend said, "why go to the sessions to watch the PowerPoint when I can go to the Microsoft Product Pavillion and get a Product Manager or Lead Developer to take me through it?"

Some outstanding points;

• Meeting Chris Sells and Steve Maine in person.
• Catching up with all the bloggers shown on my photo blog roll.
• Chating with the Microsoft employees at the Pavillion, the sessions and in the speaker lounge.
• Seeing Jamie Cansdale get the opportunity to demo his NUnit Addin and other tools to Anders Hejlsberg (inventor of C#) and James Newkirk one of the creators of NUnit.
• Seeing my blog stats go through the roof.
• Spending an evening at Universal Studios and managing to ride only th escalators as I was so busy catching up with other bloggers that the rides had shut by the time I got there.
• The end of having to guess what Indigo is.

I'm really excited to see the new technology and to get on board the Indigo wave.

In the panel on Indigo a question was asked about how far Microsoft were going to go in making Indigo support pub sub messaging and competing with Tibco.  The answer was that Microsoft wanted to go all the way, but they couldn't do it all in V1.  The core infrastructure will be there in V1, but it will get easier to access in future versions.

It's 8:30am and there's a room full of geeks wanting to get deep down and dirty into the Web Service protocols.  Omri is the Product Unit Manager for the Advanced Web Services group, so he's the man responsible for the WSE team.  He demonstrated the secure, reliable and transacted demo that Bill Gates gave with IBM on 17 September.  The slides for this session are available.

Positioning Indigo's Protocols
The wire protocols that Indigo uses are key to interoperability story.  Omri is positioning Indigo as the Internet's L7 protocol, comparing Web Services as the top of an Internet stack above XML on top of HTTP on top of TCP on top of IP and onwards.  Microsoft see Indigo as supporting applications between mobile devices, P2P, B2B and EAI style applications.

The Secure, Reliable and Transacted Demo
The demo between Microsoft and IBM showed a commitment to interoperability, as well as a commitment to making sure the protocols involved in web services were royalty free.  The demo was a Supply Chain Management application involving a supplier managing inventory and ordering parts from an auto manufacturer.  It demonstrated security, reliability and transactions in an interoperable way.

MindReef  SoapScope - a better SoapTrace packet sniffer 
Omri showed the messages flying on the wire using MindReef SoapScope, the same tool that Scott Hanselman used in his Learning To Love WSDL presentation at TechEd 2003 (the funniest presentation I've listened to).  Werner also recommends it.  It looks like this is a better approach than MSSoapTrace, which is so buggy that it makes development a horror.  Are Microsoft going to fill this hole?

Web Service Design Principles

General-purpose, composable protocol framework.  Web services allow a building block approach, rather than a monolithic stack like DCOM.  The use of SOAP headers allows the various specifications to be improved and versioned in isolation, without having to redo everything.  Omri mentioned that making the specifications composable was a challenge.  He pointed out that WS-Attachement which was based on the idea of using MIME attachments, based on email, was a mistake that couldn't be combined with other SOAP specifications, like security that relied on SOAP headers.  It became clear that to add security to attachments they would have to reinvent many of the same ideas used to secure SOAP messages: a better model would be to put the attachment within the SOAP envelope so that WS-Security using SOAP headers would provide security.  Omri mentioned that they had learnt from their mistake and were proposing MTOM (message transmission optimization mechanism) as a SOAP-based attachment specification.  As Martin Gudgin says, DIME is dead.

Architecture is meta-data driven, policy based.   WSDL provides some meta-data about a server such as the basic information about messages and ports.  However, this is not enough to describe what the service needs.  This is where policy comes in.  Policy provides another way of expressing the extra metadata that a service needs ( Basically schema is not enough to describe what the service requires).  You can say things like I am a service that requires strong 128 key encryption.  Also, I support HTTP but I am inside a trust domain with you so I can go faster by UDP or TCP.  Deployment information and info about a service that can't be captured in WSDL is in policy.

Protocols are factored from both transports and applications semantics.  Basically web services place information from the transport into the soap message, so rather than using HTTP request response model based on HTTP headers, the information is placed inside the SOAP message to provide abstraction from the transport. 

Architecture allows intermediaries.  Having intermediaries is useful for routing and other reasons.

Uniform data model for protocols/content.  The same infoset is used for both the header and the body.  It allows for header and body-based content-based routing.

Broad industry partnership.  Nothing was going to get off the ground without interop.  Interop was necessary for a transformational change. Omri says that Microsoft knows interop is essential if Web Services are going to be a revolutionary technology.

WS-Addressing
Addressing is a core part of the architecture, the work horse of the stack.  It provides a mechanism to encapsulate identity information in an XML end point reference. There is a basic <Address> element that is the network address of the resource.  There's also the <ReferenceProperties> element that can be used for extra information.

The idea is to allow people to host more than one service at a particular network address.  So if there is a service facade it can host multiple services, without having to do ugly bizarre things with the URLs.  The addressing mechanisms are transport neutral and the internal resource organisation is opaque.

The header has to have a to and action.  It can also have a From, ReplyTo, FaultTo, MessageId (a unique id) and RelatesTo (that provides a way to correlate the message).

Attachments: WS-Policy, WS-PolicyAssertions, WS-PolicyAttachments
Policy provides a framework for making statements about resources.  The policy can say that the service requires (such as transports and security).  Policy can also be used at runtime to make sure that the messages meet these requirements. 

WS-PolicyAssertions provide some basic requirements that can be expressed such as TextEncoding, Language, SpecVersion (Uri asserting compliance with a namespace spec) and MessagePredicate (XPath statements about the message)..  WS-PolicyAttachments solves to the problem of how to attach policy to an endpoint (e.g. matching an address with a policy).

WS-MetadataExchange
This addresses how to get metadata from another endpoint.  There isn't yet a specification to cover this although the demo used a simple get to retrieve policy, WSDL and schema.  It was implemented in the bits using a simple GET mechanism.  Kevin Hammond mentions some more details of how Indigo is doing it in the current bits.

WS-Trust
This defines how to broker existing trust relationships.  It defines how to exchange security tokens using a security token service.  The specification details a RequestSecurityToken message that is sent to a trust service, that replies with with a RequestSecurityTokenResponse.  This is currently supported in WSE and Hervey Wilson, the lead developer has posted a code sample that demonstrates it.  WS-Trust is one of the building blocks of WS-SecureConversation.  

WS-SecureConversation
WS-SecureConversation defines a mechanism to allow clients to make multiple calls to a web service, without incurring the overhead of asymmetric encryption.  It involves using a SecurityContext token based on symmetric encryption.  Think of it as a security 'session' object.  The token can be received using WS-Trust to request one from a security token service, or from having one party create one, or through negotiation.

WS-SecurityPolicy
This specification allows for policy statements about security tokens.  It can be used to specify the types of tokens, the confidentiality or encryption requirements, the algorithms to use and the which parts of the message require signing or encryption.

WS-ReliableMessaging: better QoS, less code
Reliable messaging allows developers not to worry about an inconsistent network.  It provides ExactlyOnce and in order messaging that saves developers from having to write code to worry about lost, duplicated or delayed messages or endpoint failure.  Sent messages are stored in a persistent local store and are only removed when an ACK is received from the server, otherwise they are retransmitted.  Early and Adopter have more discussion. Rebecca Dias has a Starbucks metaphor for the specification-challenged.

Transactions
Transactions are described by two specifications, WS-AtomicTransactions and WS-BusinessActivity, based on the WS-Coordination.

WS-Coordination.  The common parts of the two-following specifications.  It deals with how to set up a context for the transaction, based on WS-Addressing.

WS-AtomicTransactions.  This is the standard two-phase commit (prepare, rollback or commit) using web services.  It uses locks to prevent others accessing resources.  It works best inside an organisation where you can trust the locks to be used correctly.  Don Box calls this the 'anti-availability protocol'

WS-BusinessActivity.  This is a more flexible approach that doesn't use rollbacks or locks.  Instead transactions can be reversed using compensating actions.  In this way the system continously moves forward, there is no need to roll something back (if money was credited into one account, but something else in the transaction failed, post a credit to correct it rather than using a transaction to lock and roll back the initial credit).  It also removes the need for complete agreement between all resources before committing.  Omri's example was a transaction that books a flight, hotel and car for the PDC.  If the car isn't available, he still wants to make the trip and use a bus or a taxi, but there's no need to cancel the booking.  This approach is best for loosely coupled (think external web services) situations, but isn't in the PDC Indigo bits (they think it's important, they just couldn't do it in time).

Omri has announced he will repeat his session on "Indigo: the Web Services Protocols and Architecture" which is great.  Martin Gudgin says that they are demonstrating the sample from the Gates/IBM demo (I want these bits in WSE!).  I saw the comments on the WS-Transaction specs that I'll blog on tomorrow (or I'll beg Drew, who does a very, very thorough job of session notes, to go).  I'd like to sponsor Drew to go to conference I can't make it to in future).

I was so busy catching up with people over lunch I got there late and was sitting in the corridor watching the TV again.  Even though I spoke with Omri last night at Meet the Experts, I didn't recognise his voice and the TV only shows the PowerPoint from the smaller rooms.  I thought that it was Don Box with a shot voice from singing with Band On The Runtime.

ObjectSpaces introduces a mapping layer that separates the business logic from the data access logic to reduce the amount of code to maintain.  It's a declarative mapping between objects and relational tables.  It makes sense when you have a strong or large business logic layer.  It will be available in Whidbey.  They will also have a nice User Interface in Visual Studio to help with the mapping.  I've heard from some ThoughtWorks friends that it's not as good as the Java versions or Neo, an open source product from ThoughtWorks (this guy may have been biased ;)

It works with mapping files to tranlsate from an object query/update to a SQL query/update.  The idea is that objects are responsible for saving their own data and can just call a .Persist method to save the data to the database.  One of the examples is showing how to use GetObjectSet to execute query strings against the objects.  It's a new query language called OPath that lets you write select statements such as this to return data:

It's being given by Luca Bolognese who has promised that his goal is to avoid PowerPoint hypnosis and aim to make sure that no one falls asleep.  He's Italian (yes, really) and a very funny speaker.  However, his accent can sound funny.  He's reading out SQL statements and adding an 'a' to every word, so when he's reading SQL statement it sounds like 'selecta ... froma ... exista'

Classic quote:  Someone asked why they ar eusing OPath rather than XPath.  He said 'we used XPath in the prototype and gave it to the group of programmers, and they came back and said it was too hard.  So we did a search and replace to switch the slashes with dots and they loved it'.  Actually it appears that the two are different problem domains.

I'm sitting with Peter Provost (by chance, the photographer of many of the photos in my photo blog roll), soaking up the blogging energy that he's emitting.  He's actually writing the source code as we see it (a true touch typing programming god!) so perhaps he'll post it later (Peter's notes are now available).

Update: Paul Wilson left a comment pointing to more examples in his article on ObjectSpaces. 

Steve Millet is talking about the improvements in the Indigo model for security tokens.  The good news is that the madness has stopped: when a UsernameToken is validated you only need to return a bool rather than the password.  WSE 1.0 and 2.0 require the password to be returned allowing WSE to work out whether they match.   This was uncomfortable for several reasons, such as the fact the password might have been hashed, or just the fact that sharing the password back with the framework feels like a 'boundary violation'.  I'm glad that we're seeing the end of this bizarre API practice.

Other interesting tidbits were that SAML tokens will be available in Indigo.  Now, if they were only in WSE ...

Existing WSE/ASMX applications are likely to have a good upgrade path to Indigo, with similar security attributes.  There are also extensibility hooks in Indigo to do custom security token handling, so there's an upgrade path for WSE (though this is almost certainly having to write code).