Recently I've been doing a "tour of the trenches", helping a major client with their web-based applications for the Insurance industry. It's taught me that there's a lot of development work that goes on in companies that doesn't relate to the overall aim of delivering business value through software, specifically around the building of "platforms".  Brad Abrams noted something similar when he went on his road trip, there are a lot of companies building custom platforms.

From my experience a lot of these platforms turn out badly since most companies can't afford the time or money to build and create successful platforms.  Rather than writing platforms, I'm looking for ways to use Other Peoples' Code to build solutions that deliver better business value.  At TechEd this week I'm interested in evaluating Office 2007, and particularly the server platform, as a platform to build on.

Looking at the domain of Insurance applications it's easy to see why lots of developers want to write a platform.  The basic flow is similar in structure to many other types of financial trading:

• An application form, with one or more wizard-like pages, containing controls with single- and cross-field validation.
• The application form is sent to a rating engine (usually authored by Underwriters) which determines whether the business will generate a quote and if so, what the value of that quote is.
• After a quote is generated there are various sequential and state-based workflows around that can occur (e.g. accepting the quote, revising the quote, "binding" the quote to make it a policy).

With that in mind I'm currently looking at Office Server 2007 as a possible platform to build on.  Here's a sketch of what I'm looking:

• Create the application form in InfoPath (giving me the single- and cross-field validation), then publish it as a HTML form using Office Forms Server (there was a session at PDC as well).
• Use Excel Services to host the spreadsheet that contains all of the logic for calculating the Quote.  Once we agree on a set of named cells/ranges in the spreadsheet, the underwriters can keep control of it.
• Use SharePoint's support for Workflow to implement the Workflow.

Obviously I'm just sketching at this stage (I still have questions about licensing, performance and flexibility are key) and I need to do some prototyping, but overall I'm interested in the possibility of being able to build basic Insurance trading applications on top of Office 2007 as a platform, dramatically reducing the amount of code I will need to write.

TechEd Boston started Sunday night with Ray Ozzie admonishing us that there's a 'Services Disruption' coming and proposing the slightly tongue-twisting 'Client Server Service Synergy' (and the lesser-used companion phrase 'Client Server Service Symmetry') as the way forward, with Microsoft offering more services in future. 

Ray with the usual dramatic structure of a presentation (much like what is recommended in the "beyond bullets" book), setting up that we're heading for a new disruption in the way that we work.  He did the standard mainframe -> mini computers -> micro computers -> client/server -> the internet ("the mother of all disruptions") -> peer-peer set of disruptions, personalised through his own life experiences with Lotus Notes and Groove. There was the standard argument that we need to be on the lookout for new disruptions.

The drivers that are changing things for today are the promise of multiprocessors (32 to 1000 CPUs inside a single machine - up to "oodles" in Ray's words) the reduction in storage space and the growing ubiquity of bandwidth.  The interesting point that Ray highlighted was the fact that in the past it was research and corporate environments that drove datacentres, whereas today these were being driven by experience with consumer market through search, advertising and consumer shopping sites.  Ray's point was that the benefits of investments in these environments will be felt more widely.

He wasn't terribly specific about how this would happen, though he was trying to position Microsoft's approach as a mix between the client services approach and external services in a "client server service synergy".  He showed windows desktop search searching over the internet, SharePoint sites and the local machine as example of this.  He also showed Microsoft Dynamics using 'business mashups' with Windows Live virtual earth.  He also mentioned the online/offline architecture of Groove.  There was much mention of the Windows Live set of products.  There was a Windows Live Identity that I hadn't heard of before - I'm assuming it's a rename of Passport? It might have been the jetlag but it felt like another moment of marketing fatigue (apparently MOM will become System Server Operations Manager as well).

The presentation hall felt very much like an aircraft hangar and the regular sound of planes overhead reinforced this.  Ray was the first keynote speaker I've seen wearing a suit jacket on one of these talks.  Obviously the chip-implant wasn't powerful enough to convert him to the standard casual 'uniform' of a blue shirt and a pair of chinos.

Kirk Allen Evans has an interesting post about proposing a distributed architecture for a client where he finds, after doing performance testing, that the speed difference between different transports reduces as the amount of work done within each call increases.  Based primarily on this finding he's recommending ASMX web services as a better strategy than using COM+/Enterprise Services for his client.

I remember coming across this arguement at TechEd 2004 in Amsterdam where Christian Weyer spoke about some Thinktecture research which showed that the difference in speed between Web Services and other distributed technologies (remoting, enterprise services/COM+) was significant if the work done in the distributed call was minimal, but once a reasonable amount of work was being done the performance difference became less and less important.  Worrying about the speed of the transport without thinking about the amount of work done in each call is a little bit like worrying how fast a car can rev if you have it in neutral.

Of course, this is all keeping with the long-known guidance from the Indigo team, which is to start with ASMX and only go to other options if you need specific features that they provide.  Richard Turner's recent white paper provides more details.

I'm on the Ask The Experts stand here at TechEd (back tomorrow and Wednesday at 2pm) and the first person I spoke to today asked me how to solve a socket access permission when using a when using a tcp service with WSE 3.0 Tech Preview.  Working through this also answered a second common question people have, which is 'what's the default port number that WSE uses when "soap.tcp://localhost/" is specified as the service address?'.

The exception in questions was:

An attempt was made to access a socket in a way forbidden by its access permissions.

This occurred when I was trying to run the TCPStockService sample application and here are the key lines:

Uri address = new Uri("soap.tcp://localhost/tcpstockservice");
// This starts a TCP-based listener if there isn't one already started.
SoapReceivers.Add(new EndpointReference(address), typeof(StockService));

Digging through reflector it turns out the the constructor of the Microsoft.Web.Services3.Messaging.SoapTcpTransportOptions options sets the defaultPort:

So, 0x1f91 in Hex turns out to be 8081 in decimal, so by default the WSE tcp transport listens on port 8081 if no port is specified.  Now the exception message made more sense, since I had another application listening on port 8081.  Changing the port number, or stopping the process that was listening on port 8081 solved the problem.

Thanks to the MSDN Product Feedback centre I can send a suggestion to improve this error reporting straight through to the product team.

Here are some photos and conference highlights from TechEd Amsterdam (completing my backlog of blog posts).  Aside from what I've blogged already, the highlights of the conference for me were:

• Being there when Don Box spoke for the first time about BOA.  See here for a summary of BOA postings. [update: The concept of Business Oriented Agents (BOA) was a joke designed to send up the hype about the 'next new thing' and the lack of clarity in press reporting about concepts such as SOA.  Unfortunately the joke was not clearly understood and some people are understandably upset about it.  This post was my only reference to it and I apologise if any readers felt mislead.]
• Hanging out with the other Indigo guys and others who I'd met at PDC and TechEd US.
• Meeting many people from the UK community at the BoF and Chalk and Talk sessions.  Thanks to everyone that came along and those who had the courage to spend some time on the park bench answering questions and making statements.
• Increasing my list of UK Microsoft bloggers - welcome Johnny Hall (XP afficionado) and Peter Foot (Compact Framework MVP)

Here are some photos from TechEd Amsterdam, completing my backlog of posts.

Some of the 6,000 drums at the start of the keynote.	Delegates enjoying the 3D presentation that finished the keynote.
Pat Helland trying out his Wizard costume in the speaker lounge.	Heidi, one of the amazing organising team, showing a way to deal with the size of the RAI conference centre.
A "family photo" of all of the Microsoft Regional Directors at the Staff and Speaker Dinner on Friday.	The Boom Chicago Team making fun of the competition for Best Speaker evals between Kimberly L Tripp, Rafal Lukawiecki and Steve Riley

I hosted a lively session on what Service Orientated Architecture really means at TechEd Amsterdam.  While it was a Birds of a Feather session, I decided to run it as a park bench format in order to take advantage of having David Chappell, Michele Leroux Bustamante and John Hooper (blogless MCS UK Architect) come to the session.  Here were some of the interesting discussion points that came up:

• There was some agreement that SOA is a pragmatic marketing term that unites many existing architectural principles around SOAP. 
• The closest agreement about a definition for SOA was that it was based around common architectural principles of encapsulation, loose coupling and messaging.
• There was some discussion about whether asynchronous messaging was a necessary part of service orientation. My feeling is that since you can achieve synchronous patterns over asynchronous communications that having asynchronous messaging capabilities is extremely useful.
• Although it's possible that these principles could be applied without SOAP, it's the fact that Microsoft, IBM, BEA and others have agreed that SOAP will be the lowest common denominator that is the pragmatic reason behind the current push for SOA.
• David made a point that service orientation will be whatever Indigo supports when it ships.  Shipping software always wins.  I think there's a lot of merit in this argument, but to the extent that SOA is based on generic architectural principles it is worth considering using these principles in systems that are design today (as Clemens demonstrated all conference).  If SOA principles help solve your business problems today then it's definitely worth starting today rather than waiting for Indigo.
• Some delegates were suspicious that using Indigo would enable them to interoperate with other systems that didn't use Indigo.  There was some confusion about the idea that Indigo will provide an object model that can be used to develop a system that has the capability to send messages between the systems that are based on interoperable WS-* specifications.
• The four tenets of service orientation are necessary but not sufficient for a system to be considered a service oriented architecture.  Some people thought they were too technologically focussed because they were tied too closely to XML technologies.
• David mentioned that the best SOA installation he'd seen was using CORBA several years ago - it had support for finding services, common schemas etc.  Michele backed up the need for shared industry schemas based on some of her experienced.
• John Hooper was interested in Pat Helland's assertion that services should not share transactions, which led into the difference between WS-Transactions (classic two-phase commit) and WS-BusinessActivity (compensating transactions).
• A delegate who was learning about the topic came up to the chairs and spoke about what he'd learnt at TechEd.  This was great to hear and generated a lot of discussion.  Clemens' presentations clearly had some impact with many in the audience.

Here's me with David Chappell after the session.  I've been a fan of his since reading Understanding ActiveX and OLE when I started Microsoft programming.

Clemens' session on his ProseWare application at TechEd Amsterdam last week was one of the best conference sessions I've seen.  Proseware is "an industrial-strength, robust, service-oriented example application that newtelligence has designed and implemented for Microsoft".  The application clearly demonstrates how to go about building services today with currently shipping technology, reinforcing that there's no need to wait for Indigo to start building service oriented apps!  I'm hoping that we see a public release of these bits soon on MSDN.

Points that grabbed me:

• Guidance on where to use messaging patterns: Use the OneWay pattern where there is no intelligent immediate reply or no reply is needed, use Request Response pattern where a message asks a question that can be answered immediately (in under a second) and use the Duplex pattern when a message asks a question that can be answered later as the service has capacity (e.g. anything that takes over a second).
• Clemens showed how to achieving 'near enough' reliable web services with HTTP and services that use MSMQ transactional queues behind the service interface.  If there were any problems placing the message onto the queue then an exception would be returned inside a SOAP fault response.  He optimised this further by using a void response type on the web service method, even though it was not marked as OneWay, so that if there were no problems placing the message on the queue then the web service response message would be small.
• ProseWare is based around a repository of XML schema files which he uses to dynamically generated the 'message' classes in the application using pre-build steps.
• All of the service projects shared these schemas rather than having any project references linking to the binaries (services are autonomous).
• The pre-build step inserts ISerializable attributes onto the message classes so that they can work with Remoting.
• These message classes are used as the only input parameter to all of the public web service methods.  These classes leverage XML Schema's support for allowing any element or any attributes to come through, which is a powerful way of allowing for future extension to the message.  Dare goes through this in one of his previous posts.  The XML Serialization attributes look similar to this:

[System.Xml.Serialization.XmlAnyElementAttribute()]
public System.Xml.XmlElement[] Any;

[System.Xml.Serialization.XmlAnyAttributeAttribute()]
public System.Xml.XmlAttribute[] AnyAttr;

• Clemens showed how to use the properties in COM+ 1.5 in XP/2003 to set the home directory for an application, meaning it is possible to use a .NET config file to store the config. He's blogged about this previously here and here.
• He created his own object pool to create something similar to the ADO connection pooling but for COM+ objects.  He simply pops a component out of the pool and pushes it back when he's done, avoiding the excessive overhead when calling new in a COM+ environment (which has to create the pipeline connections).  He has also blogged about this JIT activation pooling here and here.
• He mentioned how LRPC, which is used under the covers in ES, is the fastest way to go cross-process on a single machine.  In order to get this benefit you need to use the JIT activation pooling in order to avoid having the performance gains wiped out by the cost of creating ServicedComponents (again, something Clemens' previously blogged).
• The nice part was that Clemens had done the hard yards and built an application installer that handled created the SQL Server and Windows user and group accounts.  He even showed where he'd found bugs in the OS and how he'd had to work around them.  Information like this is priceless (well, worth a lot of contracting dollars) if you're ever working in a situation where you need to achieve these outcomes.

Should we write test code against interfaces or something more abstract than the implementation?
James mentioned that MBUnit is a tool that allows you to test against an interface.  The question was whether you should create interfaces that enable tests to be written against them in case further implementations were created in future.  James' attitude was that this might result in wasted work ('you aint gonna need it') since you may not need it, or may not need it now.  Instead, abstract things out when you need them - don't create an interface just to test it.

James also said that an interface is not a good example of the contract of what is being done - it is the name of the method with input and outputs, but does not reflect how the method reacts to the input. James writes tests that show the real interaction between someone that calls the code and what it produces.

How much of the class should we test? Public methods only or protected and private methods as well?
Around a third of the group thought you should test protected and private methods, about another third thought that we should only test public methods.

The arguments for testing internal and protected methods:

• To ensure that the internals work. One delegate mentioned writing a test before each internal method. Someone else said the start with the public method then refactor and hide the method by turning it private.
• Another argument was An argument is that if you write a granular class with lots of private methods, then the tests should be just as granular as the thing being tested.
• The internals are the most important as they contain the biggest areas of logic, so they should be tested directly.

The argument for testing public methods:

• Private method tests inhibit refactoring - you have to refactor your tests with a change, increasing the burden and making it less likely that the tests are updaed.
• You may need to test all of the scenarios from the real world against the public method. If your class is written well then the private methods should be tested.
• If you separate the tests from the production code then you must test the public methods.
• Using public methods gives you a clear limit to the number of unit tests that you need. Public tests will get most of the issues, there's no payback from more investment by testing private and internals.

James says that there is no winning this argument. He favors testing the public interface because it decouples the test from the implementation which will discourage refactoring. However, there are cases where it doesn't make sense to expose something just to test it. He thinks it is 80/20 or 90/10 favouring testing through the public interface.

Whidbey will have friend assemblies that allow developer's to split two assemblies, this other assembly is akin to being inside the same assembly. You can do this today with a multi-module assembly built through the command line, not through Visual Studio.

How do we introduce Unit Tests to a 'legacy system' without tests?
No one in the session had seen legacy code that is easy to test if the testing hasn't been thought of upfront.  One person mentioned that they created tests for each issue logged through the help desk and then used these as a regression suite.  This also demonstrated the value of unit testing to their organisation.

James suggested drawing a line around a piece of the code that you need to change. Test it's external behaviour, make changes and ensure the tests still run. They aren't unit tests, but who cares? Create a boundary, create tests and then change. The key point was to conserve effort and only write tests on things that you are going to change.

James also mentioned a book to be published in September by Michael Feathers called 'Working Effectively With Legacy Code' that describes how to handle this difficult situation.

How to convince people to do test first? Argue against the concept that it will take too long to write the tests first?
One delegate mentioned that this is hard because the best way to convince people is to show results, which requires a practical example, which means you have to know how to do it (what classes, how to unit testing). It takes time and experience but it will work eventually.

Someone else mentioned the green bar of success (my favourite) on the screen is a big part of demonstrating it.

You have to make sure the person has the right mindset. Not everyone has a zero-defect mindset. They want to write tests more than write code.

James says the story is not about the individual developer who wrote the code and knows it works. It is about the team and the ongoing evolution and maintenance is where the tests matter. How can you be confident of your result without them? With unit tests I know they works, before I relied on something else, but now I know.

Why should we write the tests first?
James says that no one will take my word for it. What convinces people is having to test something that hasn't been created with testing as a priority. If a setup method has 20 lines of code and many objects being created and only contains a single assertion - it wasn't written with testing in mind. Someone will say it is really hard to test it.  So the next question is 'What would you do differently?'  The answer to this question is what people should do in Test First.

James said that he believes that testing has to be seen as a primary part of development. We have to incorporate test inside development. There may be QA, but they start at a different level. You have to incorporate testing as part of development.

In the PAG group they have 2,000 unit tests that get run against the library every time someone modifies the code. When James talked to the testing and QA group who do integration testing -they didn't know what to do - the unit tests did the easy part of the testing. They have to start at a higher, more complex level to start thinking about critiquing what is going on rather than I just pass 32,000 characters in a web app and it breaks. These are necessary tests, but it doesn't take a lot of skill to do these types of tests.

Someone made the point that software development lags behind electronic engineering where hardware must be tested first.

James mentioned that studies have show that Test First is 16% longer, but the quality was much higher.

How do you test a function that is dependant on other objects or libraries?
One delegate mentioned that using Dynamic Mock objects relied on finding a sweet spot.  They are useful with objects that have relationships, but the problem is that if you refactor your code then it is highly likely that a whole slew of tests that will break.  The problem is compounded with dynamic mocks since you only see this at run time rather than compile. It works well at mocking the database, but not the more dynamic

James described mock objects as the situation where object A interacts with objects B and you need some way of 'switching out' Object B to create a 'dummy response' from the calls of Object A.

James believed that if you are interacting with something complex, building the simulation of something that is complex is a worthless activity - you spend more time writing the simulator than the test and it doesn't tell you anything? Just because my mocks work, does my real system function?  James uses a rule that if mock objects have an if statement inside them then that's too far - the behavior is too complicated and there's no value (it says there are multiple situations). They do have value, but we need to separate a simulator and a mock object. For example, writing a mock JDBC implementation - that is way too complicated.  Be skeptical of spending a lot of time spending time building MockObjects.

Someone asked - 'Don't you think that it depends on the situation?' James: "I could answer everything that way"

James mentioned the Inversion of Control pattern - this is the a situation where object A depends on object B - the developer wants to be able to switch object B out as a mock or a stub - how can we do that? Inversion of control says you create the dependent object B, or a mock or stub external to object A and then pass it into object A as a interface parameter in a constructor. That's a lot of complexity to add to the initialization,  James was interested in opinions about whether the complexity is worth it?  He mentioned that some people have pushed back on the idea because of the complexity of the construction - if all you are doing this for testability (it also decouples the design) then it is too much work?  He believes that what has to happen is that we need different language structures and we will never get there if we dismiss these ideas now. In the Java world there are lots of work being done on using containers in another way.

How do you we test the database? Where do you get your data for testing? How do you unit test things that involve the database?
James thought that the problem with the database is similar to the MockObjects. It is a lot of effort to Mock out the database. Sometime it is easier to use the database than mock, you need a database to run the unit tests and you have to fill the database, read in a dump, which takes longer and is more effort.

James' experience is that at some point you write tests against the database because he wants to test that the Data Access Layer works. When he write DAL tests he uses the database. But he don't use the DB for anything that uses the DAL - he mocks them out. A technique he use to ensure he's written good unit testing is to deliberately break something and see the results.  If he finds a cascade of errors, it means he hasn't isolated the tests correctly.  He also mentioned that if he comes back to an a feature afterwards and spend a lot of time in QA it means he didn't do well enough with the tests.

With databases,  it is a good idea to  use a transaction and rollback the transaction at the end of it to ensure that.  There was some discussion about how to construct database tests.  The problem with building the database in scripts the unit tests take too long and they wont be run. How long is too long?

How long should the tests take?
There were a range of experiences including:

• Four or five machines run it each product, it takes 6 hours across machines.
• Someone else has developer tests and then smoke and build tests, it can take a long time in the nightly build.
• Another project had two CruiseControl systems that built at different intervals, running a short and long set of tests.

James thinks an hour is too long - the reason is that ideally you would be able to get very quick work around. This is how it works successfully. Waiting an hour for the feedback means that a developer could only make 8 or 10 changes a day. In that hour a number of things are accumulated. James mentioned his experience on an early project where it took 12 hours to recompile that application.  In this project if someone made a change to a header file in C++  they had to recompile, so what they'd do is create global static variables instead of the header to ensure it works before 'doing it correctly' (which was never done). Sorting out problems was very complex because it was effectively an integration nightmare..

Should all developers run all tests? It depends on the architecture - if there are subsystems you could do that. In XP it says all of the tests all of the time. James' group run 2000 tests in 340 seconds against 3 databases.

How do you manage dependence tests with the database? How to restore state with persistent storage.
It is a good idea to stub out the system so that you can test against something that you depend upon. Then when the implementation is delivered you can run the tests and it becomes clear where there is an integration problem.

One delegate spoke of how he stubs out the system, then write tests and implementation, then the stubs are removed and the real implementation of the rest of the tests are done. The platform should support this - James mentioned a project where he used 'linker polymorphism' - relied on the linker to substitute.  Another delegate said they had written something that as part of the the daily build checks out the project file, it changes the XML structure of the C# project to switch references and changes them to the actual assemblies.

How do you test concurrency? Multithreading?
These are just really hard. Testing timing problems with unit testing are hard. One delegate spoke about how he had written unit tests for a multithreaded apps and said it was one of the best examples to convince others of the value of unit tests. It took 2 months to create, but then it only took 1 day to shift it from Java to C#.

The xUnit tools should support this but don't make it any easier.

Another delegate used a timer library and extended the Nunit assertion.

Roadmap for MS
VS Team System will include unit testing support some time next year. It will have a number of integrated testing tools. The keynote yesterday showed a tool 'very much like' and 'subtly' different tool to Nunit. James will be talking about the difference.

There are load testing tools, web testing tools in there. It is a team system that is extensible by many different partners. Many partners, such as CompuWare (did a functional UI testing), doing stuff inside this 'platform' that can be extended.

Someone will write something that will allow Nunit to execute in this environment.

How do you generate tests?
Team system will create a stub test from the code. These are just boiler plate - it doesn't do analysis of the running code and propose a 'good test'

One delegate had a poor experience with a tool that created tests automatically because the tests didn't take into account the intention of the class which resulted in the tests failing. Writing the test manually is more useful as these can act as a specification for the class.

James thinks looking at the implementation to drive the tests is looking at it the wrong way. If you wrote the implementation wrong and then create tests off these, what's the value? The tests should say 'this is what the implementation should do'

People often propose this if they haven't done unit testing before.

How do you test distributed applications?
It relies on subsystems - just write tests that test the local machine, when you have to integrate it all together you might need to run tests, but these are not unit tests in this situation.

How do you unit test GUIs?
One delegate used Rational Robot - costs a fortune. If you are careful and script it then you can get away for a few builds without it breaking. SendKeys was also used.  Another person just avoided the problem by trying to get all of the functionality out, recognising that it is hard to test the UI.  Someone said that it is hard to test drag-and-drop operations.

James menioned that Robot is not good to drive development because they require the UI to be done, but it is hard to do.  NUnit Forms and NUnit ASP were also mentioned.

What frameworks can you recommend?
Nunit, csUnit, mbUnit, CLRUnit
HarnessIt (commercial)
Xunity (Commercial)

How do you do Web unit testing?
HttpUnit works, but it is Java, these can be used successfully to drive HTTP requests and do some testing on the HTML that is returned.
They suffer from many of the brittleness problems that the robot testing does - you have to use id's on the elements - but it requires a lot of thinking about how you output.

David Chappell presented a barn-storming presentation based on the idea that the future is services, that services will be called by business processes and that we need to look for a platform that will manage business processes.  He argues, convincingly that we can't expect App Servers to perform this role.  The answer comes with Business Process Platforms.  Here he positions BizTalk 2004 as the answer and goes so far as to claim that it will be the major product at future TechEd's and that getting close to business processes (through BizTalk) could be a key part of keeping your job as a developer since business processes are much harder to outsource than simple services.  You can also read David's whitepaper on BizTalk 2004.

Application Evolution
There are four waves of applications: Mainframe, client/server, multi-teir and now service oriented.  They share the idea that there is a database at the back end, but the key with services, it that they are designed with the idea they can be consumed by other applications, not just humans.  The reason why we can have services today is that we have web services  - an agreement on soap and other specs - amongst all of the vendors.  It's a huge change that we are at the start of - 4th generation.

Soap is like TCP for applications.  It took years between the start of TCP and its ubiquity: the same will happen with SOAP and web services.  It may be five years away.

To think of SOA as just about soap is folly - the reality is that going forward we will see some apps exposing their services via SOAP, but we will also see other diverse ways.  Not all apps will be SOAP.

Who calls services?
SOA talks a lot about how to build about how to define and build services and miss the point of 'who is going to call these services?'

David proposes three groups: UI (portal, asp, jsp, win forms etc), other services (we will have composite services) and business process (some central business process platform that will manage relationships between services).

The need for a Business Process Platform
Where should we build these business processes?  Is it in an app server such as J2EE container?  No, an app server all by itself is not the right place to build service-oriented service.

We are seeing a new kind of platform designed to support business processes.  In other architecture shifts we've seen new platforms - mainframe to client services produced RAD tools, the shift to tiered apps produced App Servers, and now with services we need to support business processes that drives services.

Requirements of a Business Process Platform
What do we need from a business process platform? Something that manages communications with other applications, business process implementation, scalability, modifiable business rules, process monitoring, tools for working with trading partners, cross-app authenticaiton, human interaction with business processes.

Rules change much faster than processes change - so separate out the rules from the processes.

Business Processes: Your job may depend on them
Business processes are more immune from outsourcing than the services.  So as developers we have to start caring more about business processes.  In five years time we'll need to be closer to the business or move to Bangalore.

BizTalk: A Business Process Platform
David mentioned that he had previously avoided BizTalk because he thought integration was messy, boring and 'on the side'.  However, he thinks it will move to the centre of application development.  If you believe in the move to service orientation then you have to believe that business processes that drive those processes are fundamental, therefore BizTalk will be the centre of it all.  It is about to go mainstream as the service-oriented world becomes a reality.

It's not about B2B and EAI.  These are just subsets of the larger space of business processes. 

BizTalk Engine
This is built fundamentally built around the concept of a message.  It doesn't mean only asynch, you can use RPC, but what is processed here is messages.  The incoming message comes in and is processed by a receive adapter - software that knows how to talk to a service or application (there are lots).  Here is the difference between AppServers.  AppServers just support SOAP, but not the diversity of communication technologies.

The message is processed by a receive pipeline.  It does many things, including converting it to XML. The message then comes into a MessageBox (a SQL Server database) that other engine parts subscribed to (e.g. show me all messages from this organisation).  An orchestration (the BizTalk term for a business process) retrieves the message.  It may publish a response to the MessageBox, then back through a send pipeline and a send adaptor.

BizTalk Adapters
Microsoft provides adaptors for MQSeries, SOAP and SAP.  You can make your own or buy them.

Tools
Platforms need tools.  You can build your own adaptors in the Microsoft.BizTalk.Adapter.Framework namespace (notice, a namespace, demonstrating it is a .NET application).  There's also a pipeline designer, biztalk editor (used to create XSD schemas) and a mapper (mappings and xslt transformations between schemas).

Some customers are simply happy with the mapper.

Process implementation with BizTalk Orchestration
Orchestrations compile into .NET assemblies.  It has simple shapes like if-then-else statements, loop, send, receive and parallel actions.  The process logic is simple and doesn't require a high-powered language (e.g. you don't need C# or VB.NET).  Using a graphical language is a decision that Microsoft and many others have done.

Another advantage of a graphical language is that you can use it to communicate with people that understand the business domain.

This visual language can also be authored in Viso for use by Business Analysts.  This is something that will grow over time.

Orchestrations are another reason for preferring Business Process Platforms over AppServers (which implement processes through lower level code rather than graphical representations).

State
Business processes involve people, so state may need to be maintained for a very long period of time.  So we can't use in-memory.  The Business Process platform needs to manage it.  BizTalk does this by storing state automatically and reloading it if needed.

Scope
Business processes involve transactions over long periods of time.  So BizTalk avoids locks (services shouldn't let others take locks on their data), it uses long running transactions that use compensation.  Biztalk uses scopes to manage transactions that can be atomic or long-running.

Correlation
If I send two purchase orders to a service (ERP app), how can I get the correct invoice response from the service?  You can't use request response synchronous calls because the real world doesn't work that way.  So you put in a GUID in the message, but how would this work if you can't alter the response from the service?  You could match on particular fields.  This is what BizTalk does - you define fields that should be used to match responses. 

Scalability Support
BizTalk host instances enable request to be automatically load balanced across orchestrations and MessageBoxes.

Modifiable Business Rules with the Business Rules Engine
Rules change more rapidly than processes - it makes sense to separate them.  If you are building a business process in BizTalk then you can bake process and rules together in an orchestration.  However, if your process has volatile rules, you can build the process as an orchestration and put the rules in a set of rules defined by the rules engine.  This is worth doing so that you can change and redeploy the rules easier than deploying the orchestration.

BizTalk provides a business rule composer that allows business rules to be expressed in a more natural way.  You define terms (sort of like creating an object model) and then the business process rules (like script glues object models into an app).

Process Monitoring with Health and Activity Tracking
There are two levels of monitoring: technical  and business level.  The Health and Activity monitoring tool (HAT) shows the technical side.  The business level if Business Activity Monitoring (BAM) that shows real-time information about running processes (as separate from Business Intelligence data, which is not real time). BAM is based on views on a tracking application.  Business people can use Excel to do this, developers have interfaces they can use.

The Goal: Business Process Management
There's no agreement yet on what a Business Process platform should have, but we are getting a picture.  It has to communicate with other apps, scale out, support business process implementations, workflow with human beings, modifiable business rules and process monitoring.  BizTalk 2004 is Microsoft's implementation of this.

Why is SOA more well known that BPM (Business Process Management)?  Names are confusing in this case, but there's a change coming to the way we build software.  It implies more than SOAP, it requires service for building business processes.  Biztalk is a founation for building, managing and monitoring business process, in the world today and the service oriented world to come.

Pat Helland's just finished a great presentation on services (a highly polished of the version he gave at the PDC and in other locations available online).  The highlight was him singing 'Mr CIO Guy' - a 'speculative retrospective' based on what the future could look like if the Harvard Business Review's article stating that there was no more competitive advantage in having IT were true - all to the tune of American Pie by Don McLean.  Don Box was on bass and David Chappell was on piano.   It received a standing ovation from the audience (a first for TechEd?).

The video will be available on www.pathelland.com in a couple of days.

They keynote started with 6,000 attendees finding african drums on their seats.  To paraphrase Apocalypse Now, "I love the smell of animal hide in the morning".  A group of drummers from South Africa warmed the audience up with some massage (not successful) and drumming (much more successful).  It was more exercise than some delegates in a long time.  The sound of 6000 drums around the room was an excellent start.  They were used instead of clapping to respond to cool announcements.

Accessibility

An interesting demo followed by a blind computer user, showing how special software drives an external Braille device.  Combined with screen reading software it allows him to use the computer.  Interesting demo that showed how frustrating it is to navigate web pages (even those sites designed with accessibility in mind) for blind users.

What's coming in future

The main keynote showed what was coming in the Yukon timeframe of 2006 including Windows 2003 R2, Visual Studio 2005 and SQL Server 2005.  The Longhorn timeframe was shown as 2008 (or in somewhat difficult to understand maths, "in the next 2 to 3 years").  These include Longhorn Client and Server, Office 12, and Visual Studio Orcas.

The other messages was that the focus is now on platforms, frameworks, interop and EAI tools.  So we're no longer about being a specific language developer, or just a system admin.  Now it's a more holistic view based around the 'lifecycle' (bingo!)

Visual Studio Team System Demo

A quick demo of the code coverage, unit testing and static analysis tools along with the 'whitehorse' designers.  The presenter mentioned that the Team System enables you to develop SOA (bingo!) applications.  This confused me as SOA is mostly about the design of external interfaces rather than the deployment of the application within a trust boundary.

Visual Studio Express Edition

See my previous post.

Deployment

Dynamic Systems initiative.  This allows the system to be modelled, check it against the virtual description of the environment and validate the application.  This will be part of MOM 2005, System Centre 2005 (the new SMS that integrates them all).  In Longhorn there will be one integrated management tools.

Virtual Server 2005.  Host multiple servers on the one box.  This will allow parallel test environment alongside production.  Over time there will be true virtual Windows support from Microsoft.

Voice over IP

The CommNet here has Dell machines with phone handsets that can be used to make free phone calls anywhere in the world.  It was also shown how this can intergrate with outlook to record calls and make notes that were associated with the call.  I love this kind of integration.

Mobile Development

Two guys came dressed as full-size Windows Smartphone and Pocket PC devices (what is it with these guys and costumes?) and they created a smartphone application in Visual Studio that could post a photo to their Windows Moble blog.  They showed how to use the Windows Mobile Platform and Visual Studio to build an app that posts a blog entry from a mobile phone by calling a web service.  They then uploaded the app to a central website so that users can purchase and download it via a mobile.  It downloaded and installed on the handset.  They took a photo, the handset added the location information automatically.  You can see the post here.

Jim Gray

Jim Gray spoke about Skyserver.sdss.org The goal is to get the data from one telescope online.  There is 5 years of data from scanning the sky, with the full map to be completed in 2007 (It's 10 billion records and over 2 TB in size).  It can be used by teachers and students to learn about astronomy and computational data mining.  It uses a web service (displayed in a web page).  Jim showed how these images can be inverted and can call out significant features of the image.  You can see this in action here.   It also allows for custom SQL Select statements to query the data.

They are trying to federate these archives and put it into a query.  It's skyquery that publishes a schema web service and a data query web service.  They all talk to the portal, the portal determines which of the 15 centres should answer it.  Does a query plan across repositories.  Demoed a 'transcontinental query' across Baltimore, Cambridge.  Determines the plan in the first call, executes it in the second. http://www.skyquery.net/

Jim also talked about how the answers needed to be stored in partial form.  They allow people to create databases on the portal server that can store answers queries that take a couple of days.

A second project was about CERN.  It is building an enormous accelarator in 2007.  It produces a Gig of data a second (this is the result of screening out the Terrabyte per second of original data!).    They are looking at using 64-bit processors and 10GB internet.  http://ultralight.caltech.edu.edu/lsr-winhec/  They are doing a CD per second (7.1 Gbps) at this stage with Windows 2003 64 bit version.  Disk to disk is up to 450MBps.

The big announcement from TechEd Europe is that there will be 'nominally' priced Express Editions of Visual Studio for hobbyists, students and enthusiasts, including the replacement for SQL MSDE, available in beta from the end of this week. 

Microsoft have realised that they need to make Visual Studio more available to those who want to learn to write application (perhaps not professionals).  All of the major languages will have Express Editions with Visual Studio (C#, VB.NET, C++, J#) as well as SQL Server 2005.

They will be available at a nominal low price that will not be a barrier to getting into programming windows.  As Joel said last week - Microsoft will basically give the development tools for free.

SQL 2005 Express is the replacement for MSDE.  This will be free for download from the end of this week (some more audience drumming drumming).

The focus is learning how to program, evaluate .NET, interact with students and build cool apps.

They all come with a starter kit that contains samples.  The VB one allows you to catalogue the DVD collection.  It uses Amazon's web service to pull down the artwork and details of a DVD (nice!).  The edition shows all of the standard controls.  It also ships with code snippets.  Edit and continue is also enabled (big drumming!)

The MSDN content is now going to search MSDN, MSDN online and the codewise community sites.

Strangely the demo showed the presenter copying and pasting (cargo cultist anyone?) including all sorts of declare statements to enable an image to work as the desktop background.

The intellisense will now offer to fix problems (e.g. missing trailing ')') and offer to add them for you.  Finally.  The demo also showed VB's My feature to enable quicker access to functionality.

Along with Jan Erik Sandberg (founder of the Norwegian Extreme Programming Forum) I'll be running two Chalk and Talk sessions on Extreme Programming at TechEd Europe.  We're going to use a 'Fishbowl' format, sometimes described as a Park Bench Panel.  Ward Cunningham describes this format as 'very much like a wiki for people who happen to be largish in number and stuck in the same room’

Here's how the format will work: we'll start with a panel of speakers (James Newkirk will be with us for the first session at least) 'in the goldfish bowl' on chairs at the front of the room, along with one empty chair.  The rules are that those in the audience, outside the fishbowl, are only allowed to ask questions.  If you would like to say something then we'll encourage you to walk up and sit in the empty chair.  At this point discussion stops until one of the panelist leaves so there's always an empty chair.

Here's the blurb:

Are you interested in Extreme Programming?  Have you thought about starting an XP project?  Do you want to know what works, what doesn’t, what the risks are?  This Chalk-&-Talk is a great opportunity to meet with other interested and experienced practitioners.  We’ll use a ‘fishbowl’ discussion format – an extreme way of having an active group discussion described by Ward Cunningham as ‘very much like a wiki for people who happen to be largish in number and stuck in the same room’.
Wed 30 June 2004, 08:30 - 09:45 Room S
Fri 2 Jul, 16:15 - 17:30 Room S

Here's the list of all of the Birds of a Feather presentations planned for TechEd Europe.  I'm going to be talking on "Service Orientation - what does it really mean" next Thurs 1 July at 18:15, Room R.  The BoF sessions  are often the best value sessions at TechEd since they are a chance to talk with other practitioners about the practice of software development, rather than simply features of products.  It's worth planning them into your schedule as the demand has been high and the room being used is likely to fill up.

I'm going to be hosting a BoF on "Service Orientation, What does it really mean?".   It will be a chance to go over the great definition debate, to look at what problems SO is trying to solve and how to do it today.  Here's the blurb:

Service Orientation receives much hype, but what does it really mean? Is it always the best approach? does it mean message orientation? is it necessarily tied to web services and XML? how do we architect SOA solutions? How de we partition?

Here's the full BoF schedule (the conference site list is a little out of date):

Celebrating the launch of the Web Service Enhancements (WSE) 2.0 at Tech·Ed 2004, Benjamin Mitchell and John Bristowe talk about the advanced Web services specifications that it supports, focusing on WS-Security.

My TechEd conference-buddy John Bristowe has a blow-by-blow account of my CTS302 Securing Web Services with WSE 2.0 session at Teched.  Michael Earls has some notes and a couple of photos from the repeat session (which was a little fast because it turned out to be 45 minutes rather than an hour).  Aaron Skonnard mentions my first session in his TechEd trip report on his new PluralSight blog:

Benjamin Mitchell's session on Web services security using WSE was excellent. His was the clearest presentation I've seen on general security concepts along with concrete code examples.

That's going straight to the pool room. 

After covering so many other peoples' talks it feels strange to read coverage of my own talk.

As Rebecca Dias notes, I'm repeating my CTS302: Security Web Services with WSE 2.0 talk tomorrow at 12:15 in room 33ABC.  

Yesterday's talk was so crowded that firemarshals shut the door (and many attendees had to touch elbows with the person next to them).  Even Keith Ballinger, the WSE Program Manager, was left out in the corridor!

If you were at the talk yesterday I'd be grateful if you complete the session evaluation form on the conference CommNet.  These evaluations are extremely valuable and all of the feedback is noted.

I'd like to extend the 'being at the conference through blogs' experience and invite any readers to leave questions on this posts that I'll answer in the session and a subsequent post.  What would you like to know about securing web services with WSE 2.0?

Jim Newkirk got his 'day in the sun' to speak about Test Driven Development and the tool out in public promoting Test Driven Development and the tools support he's been involved with using Microsoft Visual Studio Team System.

He started out with a quick audience poll of how many people had heard of Test Driven Development (around 80%) and how many were actually using it (about 30%). So a clear victory for marketing over behaviour change there!

The Two Tenets of Test Driven Development:

• Never write a single line of code unless you have a failing unit test.  The goal is to take requirements and express them as test
• Eliminate duplication

How to do TDD
Jim starts by blocking out 4 - 8 hour sessions of development. He spends 15 - 20 minutes at the start of each session thinking about what he is going to do and brainstorming a list of unit tests.

A key part is not to get hung up on completeness, you can always add more later. The purpose of the tests is to describe completion requirements.

The flow of a TDD session: Red, Green, Refactor
The process is:

• Start by writing a test for a new capability
• Compile
• Fix any compile errors
• Run the test and see it fail
• Write the code to make the test pass
• Refactor as needed (clean up any duplication)

The purpose is about how to use the functionality, not how to implement it! The process allows you to build confidence through having a set of tests that pass.

The most successful way to do test is to do it before the development. If you start it first then you need to think about how to test.

Features in Visual Studio Team Systems
Jim used a stack example to demonstrate the process of TDD as well as the support in Visual Stuido Team systems. The first test looked as follows:

[TestClass]
Public class StackFixture
{
   [TestMethod]
   Public void IsEmpty()
   {
      Stack stack = new Stack();
      Assert.IsTrue(stack.IsEmpty);
   }
}

So, the same approach as NUnit, just with new names!

One cool feature was writing a class name followed by a method name that didn't exist yet. After compiling, Jim used a 'smart tag' to choose to create the method stub inside the target class. It wrote this stub and had a 'NotImplementedException' inside it. This is functionality similar to Eclipse and is good to see.

Clemens' talk was about managing state across multiple layers within a .NET application. His message was that there are many types or state and many approaches to dealing with it. It's not just about the ASP.NET session object! He covered a definitions of services, state and its types as well as how to manage state including transactions.

Statelessness doesn't really exist
Stateless doesn't really exist. Everything is stateful when it runs. Just because a component doesn't remember anything across calls doesn't mean there isn't a state penalty. Keeping information on the stack is a way of maintaining state.

Definition of services
A service is autonomous -lives and can be deployed by itself.

A service has its own store. It might be as system with 20 modules having 20 databases.

A service is not XML and SOAP. This is just one way of talking to services.

Services shouldn't share databases
One of the gems I picked up from the talk was that we shouldn't necessarily tightly couple everything at the database layer by putting it all in one place. Sometimes this is done for speed, but the benefit may disappear if you put it in a cluster.

Martin Fowler posted about this today:

The recent rise of Service Oriented Architecture seems to mean very different things to different people, but one plausible thread is a rise of autonomous applications with their own ApplicationDatabase that communicate through service interfaces - effectively replacing shared database integration with rpc or messaging based integration. I'm very sympathetic to this view, particularly favoring integration through messaging - which is why I encouraged the development of EIP. In this view of the world the integration database is no longer the default assumption.

What is state?
All the data an application needs to remember. It can be:

• volatile (the stack manages volatile state)
• transient (a stock ticker)
• permanent.

Transient state may contain useful data
Clemens mentioned that transient data may contain useful data that is worth storing. An example is the contents of a shopping cart at an online store. Keeping this data can provide useful information about the behaviour of people on a site (how many don't complete an order?).

Don Box and Doug Purdy did a 'keynote' for the Connected Systems Track.  They started out by asking what questions the audience wanted to see.  A great set of questions were proposed and the answers contained some of the most valuable content in the session.  Here are my notes on their answers, and some they didn't get time to do.

How does WSE 2.0 fit in with the Indigo direction?
It lets you use the protocols we have today.  WSE takes your ASMX investment and keeps you in the game as we do this protocol work.  If you don't track the protocols it may not be so important. 

Indigo will be the primary technology for using the WS-* specifications in future.  WSE takes your ASMX investment and lets you add support for those specifications today.

What's the future of DIME?
MTOM.  DIME was an experiment - we were on the wrong track that didn't support security.  Microsoft got together and did PASWA that became MTOM.  It will be in Indigo and other MS technologies.

WS-Security vs. SAML?
There are many different kinds of tokens that may be used, such as Username, X509 certificates and Kerberos tokens.  Don said it was unlikely that a token type, like SAML will become the 'single token format to rule them all'.  No definite answer on where the SAML support was.  As I learnt on Saturday, trying to implement SAML support is a non-trivial exercise - it would be nice if there was a clear statement from Microsoft on when it will be supported in the platform (so that you don't have to share my dll in order for us to use it when we talk).  I think it will be part of the identity management work in future.

How successful is WSE at interop?
Microsoft do bake-offs with WSE where they get all the vendors in a room and try and make the specifications work.  There wasn't a definite answer other than this.

How do you talk SOAP from a Windows Service?
Don's answer was that you do the hard work to host ASMX inside a service then put an ASMX façade and call into the service with ES or Remoting.

I thought this missed the point that the recently released WSE 2.0 supports Soap Messaging, which allows you to implement SOAP messaging over TCP.  I think this would be a far easier way of hosting SOAP within a windows services.

What is the technology to replace COM+ in the long term?
ES investment will keep working.  Deployment, interception and synchronization are being brought forward into Indigo.  Many of the ES semantics are a direct correlation with the Indigo model.  Doug mentioned that ES programmers will be the most prepared to work with Indigo when it ships.

Is there an issue with the verbosity of web services payload?
Don's answer was that Indigo will 'negotiate up' and switch to a faster way of communicating if the other endpoint uses Indigo.  How they do this is to be seen (there were comments at the PDC that the first Indigo call will be a policy request to see if the other end is an Indigo endpoint).  Don mentioned that the industry is having a hard time defining binary protocols that allow user definition are difficult.  Binary protocols that support user defined structures are hard.

How do we discover services and determine policy at run time?
Don mentioned that UDDI was a solution you could use today.  In a show of hands only 4 of 200 attendees were using UDDI (2% adoption?).  According to Don it makes some hapy, but some customers want more.  They want a more flexible model for describing things without having to use the tModel (which is hard to grok).  There also other groups want to discover services on devices, so WS-Discovery is where Microsoft are headed.  It is a small spec that is easy to understand that can be easily implemented.

Will we need to continue to be plumbers to do web services security?
I thought this was a great answer:

For a while.  WSE makes it easier, but if things go wrong you'll need a plumber.  We have not done our job in Indigo if people have to understand the protocols.  Your common developer needs to solve business problems, not the protocol problems.  Indigo is adding value without focussing on the protocols. 

No matter how good WSE does, since we are ironing out the interop you'll still need to read WS-Sec.  Now at least we only need plumbers when things go wrong.

What's the migration path to SO?
This was really the content of Richard Turner's talk in the track, he's also written a great post on detailing prescriptive guidance on preparing to upgrade to Indigo.  The basic message is don't do tricky things.  If you are doing something that was hard to figure out, maybe that was for a reason.  So things like SoapExtensions or custom message sinks in Remoting are not going to upgrade well.  There's lots of material out there that shows that Microsoft have a 'good story' on upgrading from various technologies.

Unanswered questions:

• What is the MSMQ equivalent for COM+?  Is MSMQ going away?
• How does EIF fit into SOA?
• Strategies for native to managed interop?
• Will Indigo support mailslots?
• Are there any application blocks for SOA?
• How do we handle events across appdomains?
• Security - you have authentication and authorization - what about any attacks through the channel - filtering content? Content-awareness in firewalls?

The Cabanas are a new idea at this year's TechEd.  They are informal areas where attendees can get closer to the presenters and interact more.  Here's a shot I saw of Rocky Lhotka and Ted Neward presenting a session, with Peter Provost in the background.  It looked like a Roman Forum.

I'm with all of the 'Blue Shirts', speakers and the Microsoft staff, in the keynote overflow room, sharing the experience of watching Steve Ballmer on a video screen.  Here are some key points:

• He's looking trimmer. A gasp of 'Atkins!' went around the room.
• Key messages - do more with less.
• The next 10 years are going to be even greater than the last.
• Only Pfizer spends more than Microsoft on R&D.
• Remember 10 years ago TCP/IP was a separate business to the OS.
• Integration is the key. How many data access layers does Microsoft need.
• How can we narrow down the skillset required to know how to use the products. Integrate to reduce the overhead required to use the platform.
• Windows XP SP2 has taken priority over Longhorn recently.
• It used to be 'features, features, features' now it's 'listen, listen, listen'.
• Watson is one of the biggest advances in computing. Being able to send crash reports to Microsoft means there is a statistical way of rating the issues that users are having.
• Integrated innovation and customer responsiveness to do more with less.
• Security is key focus.
• Spam is too cheap to send - we need to add cost and burden. Using techniques like making the sender prove who they are.
• Interoperability has been a key focus. Microsoft has done more than most people have ever given them credit for in integration. Microsoft is absolutely behind the XML stack as an open standard. The 'best and most important thing to happen to our industry'. It's an 'architected' way of doing interoperability. The old way was writing XML to connect each system.
• Microsoft Office beta web services - allow Office to be a smart end client to web services.
• Becky Dias gets on stage.
• WSE 2.0 is released! Also the Microsoft Office Information Bridge are entering Beta.  Basically web services integrated with Microsoft Office task pane.
• She's clicked on someone's name in Outlook. A task pane has come up with a form that lets her do a stock trade, calling a webservice and gets an ID back again, all without leaving Outlook.
• Demoing policy. Not sure if the audience are getting this. But it's very cool. We don't have to right code anymore. Definitely should have spent more time polishing the WSE Settings Tool wizard screens.
• .NET has more than 50% of the US market. Customers think it is 66% more reliable, 70% think it is faster, 2.7x people think it is more secure.
• The VSIP program has been increased. Oracle and SAP and TibCo will use Visual Studio for their platforms.
• Visual Studio 'Team System' - now trying to do more as part of the software development life cycle. Group development, modelling, testing and deployment.
• It looks like we now get bug tracking within Visual Studio.
• Showing a Whitehorse style screen that diagrams the deployment of the application and can check to ensure that it will work in that environment.  It produces 'build errors' when you compile it.
• Can specify that the system passes build rules, static analysis and unit tests. 
• Finally we have unit tests that are part of the build system (this got a clap!).  Now we know what James Newkirk has been doing at Microsoft!  Rewriting NUnit!
• Also includes code coverage tools as well (another clap)
• There's also a security version of FxCop that is built into 'Visual Studio Team System' based on Microsoft Research's work on Secure Computing Initiative.
• But wait, there's more .... load testing as well (more claps!)
• Back to the Information worker.  Steve has the feeling that SharePoint Team Server, Portal Server, Office and Live Meeting haven't been as well adopted as they should have.
• There will be advances in searching as a result of 'strong competition' (see Google)
• Why choose Microsoft over Linux or Java? More integreated innovation, better responsiveness and trustworthiness, partnerships, choice (more applications, better interoperability).

Overall I was a little disappointed that 'Crazy Steve' didn't make an appearance.  There was no sweat, no ranting, no cheering with the crowd.

Scott's a very funny man and hosted a very entertaining session on code generation.  My jet lag really kicked in at the start of this session, so you might like to see Peter Provost's blog for more coverage as well as Jon Galloway:

• Jon Lam talked about the 'usability tax' from using XML. XSLT is a programming language that is hard to maintain. He prefers using PERL for writing code. XML is hard for humans to maintain.
• Scott started by talking about the CodeDOM as being 'the opposite of terse'.
• Discussion about the line between creating a generic engine versus just solving the problem with code. How do we deal with the trade-off between producing a solution to the current problem versus creating a generic non-specific approach.
• Computer languages are for people to work with, so we write in C#. It is a code generator that produces IL that produces assembly code to run. We want a higher level language to build software.
• 3 kinds of code-generation: Wizard skeleton, Compiler - template and generics, Modelling - result is not a model but an assembly
• Some discussion about whether generators were useful for producing quick and dirty one-off situations.
• Discussion about what the output of code generation should be. Is it the code files, or is it the compiled DLL?
• How do you manage changes - should you do it at compile time or run time? What about it you need to modify things after they have been generated? Should you make them plugins, use interfaces or work with the config files.
• Sometimes code-generation make it hard for others to maintain.
• Don't do the rules engine that solve the 'verbs' problem. Think about the nouns.

How Corillian do code generation:

• Scott talked about how Corillian do it. They model the nouns in a visual tool using an XML schema underneath that can be extended and allows you to create your own vocabulary. The elements on the schema come from another namespace that includes domain specific attributes.
• They then we use a free-ware tool called code-smith which lets you write code-generation syntax in an ASP.NET syntax (<% for each … %> to output the data). They use that to create the code (rather than HTML in the ASP example).
• They created an XML schema adapter that looks at an XML schema and gives you a collection of top-level types and subtypes. Then all the different places where they have domain-specific knowledge they use 'aspects' by placing that logic inside the setters and getters.
• The schema describes the contract between asp.net and the host and the asp.net and the front end like the device that it displays on.
• The adapter reads the schema and presents it in a friendlier way. The CodeSmith studio is an IDE for doing this. It has adapters that takes anything that presents a collection (e.g. a database), then for each table in tables - generate the code.
• The easiest way to jump into code generation it is to use the strongly-typed collection classes that comes from CodeSmith - accounts in ArrayLists should become an accounts object - it allows you to simulate the behaviour of generics now.
• Any time in the schema with max-occurs unbounded - we know it is an array and autogenerate a strongly-typed object it.
• The business people edit the XML in XML spy. Corillian separate domain objects from the message. They have a WSDL explorer. They use WSDL and a custom binding to generate the whole banking services. The proxy is generated from the WSDL - binds domain objects, messages and verbs.
• In future they are looking at using schematron that describe restrictions (e.g. saying something is an integer is not the same as saying it is a social security number). The intention is that the attribute on an element in the schema maps through an attribute in .NET.
• Scott's belief was that anything in the schema should be carried forward because the metadata should not be lost.

If you're at TechEd and would like to catch up with me I'm on MSN Instant Messenger at benjamin AT benjamin DOT net.   I'm especially interested in anyone who's doing work or got thoughts on web services, Indigo or extreme programming. 

I'm also on Scott Hansleman's Microsoft Regional Director Bingo card (available from booths 49-50 in the Pavillion) so come and say hello and I'll help you win a prize. 

Roy was concerned that this represented the 'cult of the worshipping masses' and that we 'should not be handing out autographs, but software, tips, tricks and code'.  Well, happilly I can report that the goal of the Bingo game is to encourage attendees to talk to the Regional Directors.  It's sending the signal that we're here to connect with attendees and share experiences and transfer knowledge.

Meeting the RDS at TechEd is just like a .NET rocks episode but live and in person.

Here are some of my tips on how to make the most of the week at TechEd.

Make a session plan.  Know your entry and exit points.  TechEd is sold out.  Not only that, it is overbooked.  Expect to be sitting in the aisle if you aren't clear about which sessions you are going to and how to get there.  Spend some time learning the floor plan on the first day so you can get between rooms without getting lost. 

Don't stare too long watching the PowerPoint slides.  Every attendee is going to get a DVD with the slides and audio after the show (it will likely be on the web as well), so don't cram your schedule too full with sessions.  Pick the key sessions to attend, you can watch the others later.

Connect with people about the technology.  Instead of going to the breakout sessions, make sure you spend time in the Cabana areas and the Community Lounge.  The Cabana areas are small presentation areas where you can 'heckle' (well, ask questions and interact) the presenter.  The Microsoft stand in the Exhibit Hall has   many key people from the product groups at the event.  They are here to meet you, answer your questions and help you understand the technology.  Take advantage of this chance to have one on one conversations.

Speak to the Presenters, Authors or Regional Directors you see.  Don't be afraid to approach these people if you see them.  They are at the event to answer your questions and find out about your experience.  Use them. 

Focus on questions you'd like to have answered and areas of knowledge you'd like to improve.  Aside from the above, I think there's also a RIO networking area where you can go and find experts who can answer your question.  There are lots of key people from most of the Microsoft teams at the events and they are here to talk with you.   Spend some one on one time with experts rather than just sitting in the audience. 

There was a panel discussion with John Bristowe, Scott Hanselman, Joe Homnick, Joe Lindsay, Terry Mohn, Ted Neward.  the highlight for me was that there still isn't good tool support, or a good story from Microsoft, on how to manage services once they have been deployed.  Other points:

• Scott mentioned that his company provide services to 30% of all online US Banks and 50% of them rely on web services. So they are out there and being used.
• An excellent point was made there isn't good tool support (or a good story from Microsoft) on how to manage services once they are deployed. The tools to ensure that services are up and running and providing guaranteed levels of services are not here yet.
• There was some discussion about whether we should care about angle brackets or not. Scott made the point that we should focus on Infoset. John Bristowe made the point that it can be useful to understand the specs and know what is happening on the wire.
• Ted mentioned that editing WSDL is too hard and no-one at the event other than Scott had actually done it. Scott made the point that most of us had edited HTML files because we did not like the way FrontPage formatted them, so why hadn't anyone done the same with their WSDL?
• Some good points from the more business-focussed members of the panel that they are more interested in tools making developers productive than in having developers that understand the plumbing in web services.
• Discussion about what the definition of SOA is. Answers included 'components with a longer wire', 'objects with explicit boundaries', 'hooking shit together'.  I had a useful discussion aftewards with Joe Homnick that helped me see that SOA is most useful as a concept when talking with businesses about how to architect projects.  As I've mentioned before I'm still not convinced that SOA is definable, but it is based on top of good architectural concepts such as encapsulation, data hiding, loose-coupling, service discovery, messaging patterns and asynchronous processing.  So if we need to call this SOA in order to get all excited about these topics then it may be a necessary evil.
• I followed with the question 'what problems is SOA trying to solve?' answers included high-level re-use and interoperability.
• Discussion about location transparency not being a good idea. Ted suggested the original idea was really about being able to write code without having to know the location of the components, rather than hiding the fact that this was an expensive cross-network call to a component.
• Discussion about the fact that Amazon uses Web Services well but that XML over HTTP is more popular than their SOAP implementation. There was some discussion about alternatives to SOAP such as REST, but they were seen as more limited compared to SOAP. The key for me is that tool vendors need to implement great tools for SOAP so that it is easy to use as these competing approaches.
• The more business focussed members of the panel made the point that the technology is not the main thing, it's supporting business functions. For them the point of the Interoperability work is that it les them go out and buy products that they know can talk together.

Ted Neward gave the keynote at our WS-Interoperability Day even today.  Here are some of the points he made:

• It's important to learn from the past, in terms of previous distributed computing approaches, when looking at Service Oriented architecture and web services.
• One of the key things that made HTTP a ubiquitous approach to solving interoperability problems was that it was simple.  For me this highlighted the fact that it is essential that good developer programming models be developed over the top of the WS-* standards so that it's simple for developers to take advantage of the functionality without necessarily having to be a plumber. 
• It is important that vendors continue to work together and not start developing their own non-standard features.  Ted mentioned that vendors are currently being driven to ensure compatibility in order to make money from the new approaches, but once a market it established they may start fracturing away by adding their own non-standard implementations.  Dare covered this in his post detailing the SQL standards.
• Ted made the point that Objects work better when they are deployed together because they need to talk together.  Good objects are well factored and make a lot of calls.  This means that the same techniques does not extend well to situations where objects are distributed.
• The fastest call that can be made to a network is around 1000 times slower than a local in-process call.  This can raise to 10,000 times slower when the network is involved.  To illustrate what this means, Ted used the analogy that this was similar to his 20 minute commute to work taking 70 days (his math).
• SOA is just a new way to get developers to listen to the worn-out message that  'Distributed Objects Don't Work'
• Interoperable systems work best when the architecture is built from the centre out.  If you start at the .NET and work towards Java for example it's likely that you'll find you've implemented something that is incompatible.
• The problem with starting from the inside out is that WSDL, which provides the definitions, is hard to work with and you really need a tool to author it.

I flew into San Diego last night where Michele Leroux Bustamante had agreed to meet me at the airport.  The plane was an hour and a half late and I was wondering if she'd still be there or even recognise me.  Turns out I needn't have worried she'd be in the group with the three laptops and the wireless router:

That's Anant Kadiyala, Michele and Heinrich Gantenbein getting in all the preparation time they could before our interop demo.

I'm presenting a TechEd session in the Connected Systems track on how to use Web Services Enhancements 2.0 to secure web services.  TechEd looks like being a mini-WSE festival, with Aaron Skonnard doing the pre-conference on Sunday, Keith "My blog has gone" Ballinger talking up a storm on messaging over multiple machines and networks, plus loads of other applied web services talks.

In my session I'm going to cover all of the security features in WSE using as many code demonstrations as I can fit it.  I'll cover the basics such as security tokens, signing and encrypting before moving on to more advanced topics such as token issuing and establishing secure conversations.  I'll show how WSE 2.0 allows all of this to be done with code as well as policy and configuration files.  The best part is that straight afterwards you'll be able to go and do some Hands On Labs authored by Aaron Skonnard covering these topics.  How's that for in-depth educational experience?

Here's the official description

CTS302 Connected Systems: Using Web Services Enhancements v2.0 (WSE) to Secure Web Services
Wednesday, May 26 2:00 PM- 3:15 PM, Room 10
Speaker(s): Benjamin Mitchell
Track(s): Connected Systems, Developer Tools and Technologies, Security
Web Services are being used to cross application boundaries, especially between enterprises. Such interactions need to be secure. See how to use WSE v2.0 and the security protocols that it implements to secure Web service interactions within and beyond the Trust domain

For those interested in doing some background reading before the event, I strongly recommend WS-Security Drilldown in Web Services Enhancements 2.0 by Don Smith.  Rebecca Dias thinks its the best article she's read on WS-Security and I'd agree.  Also Hervey Wilson's blog mentions some truly magical features that will whet your appetite for what you could see.

Before TechEd San Diego I'm going to be presenting a SAML token issuer sample with Michele Leroux Bustamante as part of the Web Service Interoperability Day this Saturday.  The event is a chance to actually see interoperability happening, rather than just watching PowerPoint slides.  We'll be focussing on showing real code demonstrate WS-Security (now an OASIS standard that will be implemented with the release of WSE 2.0) and WS-Policy.

Getting the demos ready has been an international collaboration.  John Bristowe has been waving the WSE/Policy 'Pom Poms'.  Chris Haddad is doing the Java implementation with OpenSAML.

After the demonstration there's a panel discussion on the state of web services standards that should be very lively.

I'm presenting the Connected Applications: Security Basics talk at TechEd San Diego (vote now in the TechEd survey if you're attending). As part of the run up to the event I'm going to blog about some wider security topics, starting with the human aspects of security.
 
Although it's attractive to think that cryptographic techniques can provide perfect security this can never be the case where systems involve humans. The Art of Deception: Controlling the Human Element of Security by Kevin Mitnick illustrates this well. It is a book about Social Engineering, the practice of getting people to do things they wouldn't ordinarily do). It shows how easy it can be to circumvent an organisation's security through manipulating people.
 
The key point of the book is that natural human instincts to be helpful, avoid confrontation and respect authority can be easily used by a Social Engineer to get around an organization's security. Using fictional scenarios the book demonstrates how a Social Engineer can work. Some of the techniques involve posing as a fellow employee or a new employee requesting help. These techniques are often combined with sounding authoritative and being under time pressure ("I'm the new personal assistant to the CIO. I need to get the figures for the last quarter to the CIO for a presentation tonight otherwise I'll lose this job, but I can't open the spreadsheet on the network - can you help?"). The book also shows how easy can be easy it can be to get innocuous information (operating manuals, managers names, department codes, employee numbers etc.) that can be used in later communications to sound trustworthy and reliable.
 
The book demonstrates how the telephone and fax are great Social Engineering tools because they limited built-in authentication. It's easy to appear as someone else over the phone. In a large company with many different offices or a call centre it's possible to talk to someone you don't know personally and few people would think to validate the person's real identity.
 
Education and training are required to avoid falling victim to these techniques. The difficult part is that the attackers can take advantage of basic human instincts while victims have the harder task of acting against these instincts. The book finishes with a sample security policy for an organisation and flow charts to illustrate how to handle requests for information. This is useful but demonstrates how concerns about security need to be balanced against the ease of doing business (e.g. never take a message for a colleague from someone you don't know personally). I believe the threat modeling and risk-based approach are more useful techniques in helping an organisation come up with a security policy that successfully balances their security risks with their business practices.

The book's story approach did become a little tiresome at times, but overall I was impressed to see how humans are often the weakest link in a security system. While some of the stories involved high-tech techniques, such as hacking into the telephone exchange, others were simple cases of using influencing techniques to manipulate people.