I missed this story during the week but apparently the Australian birth rate is so low that we're facing extinction.  To stop the decline the government are offering AU$3000 to procreate!  So my wife and I could have made some money if only we'd flown back to Sydney to have Hannah.

Even though Hannah was born in the UK it turned out she wasn't eligible for a British passport since my wife and I have time-limited visas.  So it was a toss up between Hannah becoming an Australian or a German.  Turns out the Australian High Commission are quicker at issuing citizenship (now I undersatnd why!) than the Germans, and since we needed a passport in a hurry, Hannah's now an Australian.

The most challenging part was taking a photograph that met the passport criteria.  Trying to get a two year week old to open her eyes, look directly at the camera while lying against a light background with no shadows or parent hands visible was incredibly difficult.  After several attempts we were managed to get a photo so Hannah has a passport and can be successfully authenticated at Airports around the world.

I've been quiet in the blogsphere of late because of a number of changes going on in my life.  As well as a new baby I've also switched jobs as well, taking up a contract as a Program Manger with Kalido.  I'm responsible for creating a .NET version of their product APIs as well as managing the Test Team, focussing on automated testing.

Kalido create datawarehouse lifecycle management tools.  The core product, the Dynamic Information Warehouse helps large multinational companies manage complex datawarehouses in a way that can adapt to changes over time.  They were the first company I interviewed with when I arrived in London in October 2001.  I was intending to interview with many more companies, but I was so impressed with their technology that I accepted their offer.  Originally I helped develop a COM version of their product API and now I've been invited back to convert it to .NET. 

I started last week and it is great to be back in a software development company.  I love the challenge of shipping products given limited time and resources.

While writing about Jason Bock's experience with XP I started feeling a bit bored with asking 'Does XP work or not?  Should everyone be doing XP?'.  I read Alistair Cockburn's PhD Thesis that presents a convincing argument that no one methodology will ever be the right way of doing things,  that clever teams will find ways of selecting their methodologies at the start of a project (he gives several key criteria) and that people are really the most important part of it (which is why I'm surprised that so many organisations fail to imitate Microsoft's well-known and documented hiring process).

No methodology is going to be a silver bullet

To me this is useful because it provides a framework from which to view XP and other methodologies.  So XP has some good practices in it, but the notion that it's the only way to do things, or that it will work on all kinds of projects and that all of the practices are required for a successful project is plainly silly.

Questions like 'Is Microsoft doing XP?' aren't really important.  What is important is that any development group thinks about what works, reflects on experience and experiments with new practices to see if they make sense. 

Big methodologies can inhibit reflection

Sometimes I think that the 'big name' methodologies are harmful because they reduce the likelihood that a team or project manager will reflect on their practices.  Methodologies are often attractive to management and project leads because they are packages as an easy solution to the difficult problem of thinking about what practices are best for a particular situation.  I know that RUP says you should customize the methodology to each project, but in reality I've rarely seen this happen.  It's much easier to just use the pre-packaged methodology than it is to think and reflect on experience to develop a custom methodology for each project.

Reflection and customised methodology are harder to market than a brand-name methodology

Saying "We're Agile" or "We use XP" is much sexier and easier to sell than saying "We understand a range of software development practices and use them to create our own methodology at the start of each project, based on the characteristics of the project and reflecting on our previous experiences".

I'm a big fan of the approach that Steve McConnell takes in Rapid Development where he presents a range of software practices that can be used.  For each one he details the amount evaluates things like potential to reduce schedule time, improvement in project visibility, effect on schedule risk, chance or first-time success and chance of long-term success.  The problem is that this isn't as easy to market as other methodologies. 

While I was writing this post I came across the following post from Bob Martin on the Yahoo XP Mailing List who seems to be saying that it's more important to focus on the practices and attitudes (such as the principles behind Agile manifesto) than it is on the brand names:

I'm noticing that the industry is adopting agile practices and attitudes without needing or wanting brand names. In the last year I've worked with a number of teams who started with SCRUM, or XP, or FDD, and who have since been picking and choosing practices from the other methods to suit their needs ... Instead of increasing specialization, the industry seems to be building a body of agile knowledge that it can use as a resource help them build something useful to them. It would not surprise me to find that in ten years the words Agile, XP, Scrum, and FDD have become synonyms.

Recently I've been reading Alistair Cockburn's PhD thesis People and Methodologies in Software Development.  He argues that people are more important than methodology, there's no one methodology that will always be right (and there will always be new ones coming along) and that successful teams create their own methodology based on the needs of the project and the people involved.  Here are the key notes that I took while reading it.

Alistair poses three questions in his thesis:

• Will there always be new software development methodologies, or will there be a convergence at some point?
• If there is a convergence, what are the key characteristics of the methodology?  If there is no convergence, how can project teams deal with the number of methodologies
• What's the relationship between a project's methodology and the people on the project.

Here's the essence of his results:

• Yes, there will always be new development methodologies.
• The best way to deal with a large number of methodologies is to help a team understand what to consider when constructing their own methodology for each project.
• The people on a project team are a 'first order construct' and are more important than the methodology involved.  The characteristics of the team will limit any effect of a methodology.

His thesis is mainly composed of articles that he has written over the years.  Here are the summary points from these articles.

Observations on methodologies and project success
He based a lot of these findings on his own research of teams.

[When interviewing successful projects] what I find striking about these projects is that they show:

• Almost any methodology can be made to work on some project.
• Any methodology can manage to fail on some project.
• Heavy processes can be successful.
• Light processes are more often successful, and more importantly, the people on those projects credit the success to the lightness of the methodology.

And elsewhere:

I ran the seemingly odd hypothesis that if you simply put four people in a room with whiteboards and computers, give them access to the users, and have them deliver running software every month or two, they would have a high likelihood of success in delivering useful, working software. Surprisingly, this hypothesis has not yet been broken.

Methodology is irrelevant in small teams
A key concept is that one methodology can't fit all projects.  Team size is one aspect to consider:

… discussions about methodology are largely irrelevant in small-scale groups:

If the team consists of only two to three people, then the methodology as a distinct topic may be inconsequential. But as the team size grows, coordination issues become significant, and therefore more consideration must be given to the methodology.

Wayne Stevens of the IBM Consulting Group illustrated with the following anecdote:

If only a few drivers cross a large parking lot at night, when there are no other cars, it doesn't matter where they drive. The few drivers can arrange to avoid each other. As the number of cars moving through the parking lot increases, eventually it does matter. With increasing number of drivers, it becomes more important that they establish and follow conventions about parking, driving lanes, and stop signs.

How to select a methodology
When selecting a methodology, some things to keep in mind are:

• Team Size has a big impact on the methodology
  • The larger the team the larger the methodology
  • The more critical the system (the more important the lack of defects) the more important that the methodology is publicly visible.
  • Often there are two sizes that work for a project - small teams using lightweight approaches and larger teams using heavier approaches.
• The quality of the communication impacts the choice of a methodology
  • The richer the communication quality, the less need there is for bureaucracy.

People factors that contribute to a project's success
Things that impact the influence of people on a project's success:

• Communication.  The best way of communicating is face to face with a whiteboard.
• Tending to consistency.  It's important to have the discipline to keep working on a consistent way (when the natural inclination is to get sloppy over time).  This is also why high-discipline methodologies (such as XP) are considered fragile.
• Good citizenship and looking around.  Alistair's quote 'A common answer to asking why a project succeeded at all is: "A few good people stepped in and did whatever was needed to get the job done."'.  This is similar to Martin Fowler's concept of the wandering architect.
• People vary.  Methodologies need to fit the people on the team otherwise they will be rejected.

Barely sufficient methodology
Alistair looks at barely sufficient methodology.  He believe the following factors are sweet spots in these types of projects.

• Two to eight people in a room where they can communicate quickly and easily.
• Onsite customers.  The ability to get questions about the customer's requirements answered easily.
• Short increments.   Deliver a working set of software every 1 to 3 months
• Fully automated regression tests.  That give feedback as soon as possible.
• Experienced developers.  Experienced developers are much faster.

Recommendations
Every project need to develop its own custom methodology based on the characteristics of the people, the project's specific priorities and the technologies being used.  Other important criteria can include team size, geographic separation and project criticality.

It's important to have times of reflection where the team can think about how successful the methodology has been and reconsider and adjust their working conventions.

Jason Bock writes about his experience with Extreme Programming for the last year. I like retrospectives (see Johanna Rothman's blog on Managing Product Development for a good set of questions to use in a retrospective) and people reflecting on experience and thought some of his observations were useful.

Things he likes:

• TDD and refactoring - makes it easy to change the code knowing that tests will pick up any problems.
• Pair programming because it gives quicker feedback on the work as it's being done.

Things he doesn't like:

• Stories. He thinks that the concept is too vague and there isn't enough investigation given to system requirements
• Feeling of it being a religious practice. It's better to treat them as a set of practices rather than religious commandments.
• The coaches often refer to 'the scriptures'. Jason says 'it's weird to hear the XP coaches that the company brought in for these projects to say "Well, the XP process says …".' rather than considering what practices might work.
• He thinks it's harder to get a pulse on the project as a whole. This is interesting as would have thought that the frequent iterations would have generated some

Overall I like the sound of Jason's pragmatism when he says that for him it's a lot more important to 'get it done!'. I've always liked the question 'what do I need to do today to make this product ship on time in future' (I think I read this in Microsoft Secrets years ago).

I'm presenting the Connected Applications: Security Basics talk at TechEd San Diego (vote now in the TechEd survey if you're attending). As part of the run up to the event I'm going to blog about some wider security topics, starting with the human aspects of security.
 
Although it's attractive to think that cryptographic techniques can provide perfect security this can never be the case where systems involve humans. The Art of Deception: Controlling the Human Element of Security by Kevin Mitnick illustrates this well. It is a book about Social Engineering, the practice of getting people to do things they wouldn't ordinarily do). It shows how easy it can be to circumvent an organisation's security through manipulating people.
 
The key point of the book is that natural human instincts to be helpful, avoid confrontation and respect authority can be easily used by a Social Engineer to get around an organization's security. Using fictional scenarios the book demonstrates how a Social Engineer can work. Some of the techniques involve posing as a fellow employee or a new employee requesting help. These techniques are often combined with sounding authoritative and being under time pressure ("I'm the new personal assistant to the CIO. I need to get the figures for the last quarter to the CIO for a presentation tonight otherwise I'll lose this job, but I can't open the spreadsheet on the network - can you help?"). The book also shows how easy can be easy it can be to get innocuous information (operating manuals, managers names, department codes, employee numbers etc.) that can be used in later communications to sound trustworthy and reliable.
 
The book demonstrates how the telephone and fax are great Social Engineering tools because they limited built-in authentication. It's easy to appear as someone else over the phone. In a large company with many different offices or a call centre it's possible to talk to someone you don't know personally and few people would think to validate the person's real identity.
 
Education and training are required to avoid falling victim to these techniques. The difficult part is that the attackers can take advantage of basic human instincts while victims have the harder task of acting against these instincts. The book finishes with a sample security policy for an organisation and flow charts to illustrate how to handle requests for information. This is useful but demonstrates how concerns about security need to be balanced against the ease of doing business (e.g. never take a message for a colleague from someone you don't know personally). I believe the threat modeling and risk-based approach are more useful techniques in helping an organisation come up with a security policy that successfully balances their security risks with their business practices.

The book's story approach did become a little tiresome at times, but overall I was impressed to see how humans are often the weakest link in a security system. While some of the stories involved high-tech techniques, such as hacking into the telephone exchange, others were simple cases of using influencing techniques to manipulate people.

On Monday 5 April, our first wedding anniversary, Hannah Marie Mitchell, our first child, arrived into the world.  You can see more photos and read more about it all here.

My wife did a superb job delivering Hannah without any drugs and in a very short period of time.  I was able to get really involved in the birth, actually 'catching' Hannah as she came into the world as well as the usual umbilical cord business.  An amazing, if incredibly gory, experience.

After the birth my wife's blood pressure dropped to the point where she passed out and started to fit; probably the scariest moment of my life.  Thankfully the medical staff responded quickly, and after lying for a while with her legs higher than her head she fully recovered.  After I'd recovered from the shock, a few hours later, in a sleep-deprived state I was thinking 'wow, thank god they knew how to reboot my wife after a system crash'. 

James Robertson, a Product Manager for Cincom (a Smalltalk vendor) presented this session on blogging.  James has written his own blogging software and news aggregator (BottomFeeder) in Smalltalk.  He appeared to me to be the Robert Scoble of the Smalltalk blogsphere and did a great job of blogging the conference, which was helpful for me as I forgot my laptop power strip.

This was an incredibly frustrating session.   Instead of starting with audience-focussed questions like What is a blog? How do I read one? How can I create one? James started by talking about the XML structure of an RSS feed and the differences between versions before showing or mentioning a news aggregator or web-page view of a blog.  To me this was like explaining the concept of the world wide web by starting with HTML tags and character encoding. 

I think James had previously given this talk to a group of highly skilled developers, whereas at this session it was only me and Martin Fowler who had blogs or even used news aggregators out of the 20 or so people in the room.

Although James was generally critical of Microsoft ("Microsoft Technology is an oxymoron!") he spent a lot of time to talking about the positive things that Microsoft were doing with blogging.  He mentioned Scoble, Chris Brumme and had even found and enjoyed Roy Osherove's posts.  He mentioned that the Microsoft bloggers helped put a human face on the company, built good community relationships and produced outstanding technical content.

The part I enjoyed most was the reactions of several Sun employees in the group.  They were shocked that Microsoft would allow there developers to write whatever content they liked in their blogs.  "How can you be sure that the material is honest and not edited by corporate marekting?" they asked.  "You can find Microsoft employees questioning or criticising Microsoft products" I responded.   They were deeply sceptical and couldn't see how it could possibly work and seemed deeply sceptical. 

I did enjoy the session's focus on how RSS could be used for many different purposes other than simply blogs.  Martin mentioned how he used a hand-crafted RSS feed for his articles (which I hadn't discovered).

Joshua Block a senior staff engineer at Sun and author of Effective Java gave a great session on API design.  Joshua highlighted that good API can be great company assets because good APIs capture and retain customers.  The talk was particularly enjoyable because Joshua illustrated his points with reference to good and bad Java APIs.  You can read the full session write up by James Robertson.

I love the challenge of API design.  As Joshua mentioned, it's incredibly hard, and when working on large public projects like .NET or Java you only really get one shot at it.  Once it's out there it's extremely difficult to make changes to it.

This is where I think the Indigo team is on a great thing.  Steve Swartz describes his role as making sure that the Indigo programming model is easy to use.  Don Box mentions today that he sees his role as removing the need to have to absorb complex detail in order to create working programs.  This team knows how important it is to get the API correct.  As Don said in January "the APIs are the real lock in".

I was surprised that Joshua didn't say more about the need to test the APIs with a developer audience.  He did mention that when designing and interface for others to provide implementations for, it's worth using the 'rule of threes' and writing three implementations yourself before shipping the API to ensure that it's possible.  But he didn't really say anything about API usability testing, the use of personas or the need to get APIs out into the community for feedback.  These are all things that I believe Microsoft is doing incredibly well with Longhorn (I'm not really in a position to say regarding Java, but would be interested to know what goes on there).

This session was led by Steve Freeman and Keith Braithwaite and was focused on what has gone wrong on projects that have done XP.  See James Robertson's post for a blow-by-blow description of the session.  I enjoyed the opportunity to hear the experiences of others who had applied XP.  It also highlighted some of the issues I have with XP.

The difficulty of the Customer role in XP
As I've said before, I think one of the weakest aspects of XP is the way it abstracts away the complex and difficult problem of understanding and designing software for customers behind a simplistic notion of an 'onsite customer'.   The danger from my point of view is that XP projects may ignore the benefits that good business analysts and interaction designers can bring, in lieu of developers just working directly with a customer. 

Even with these reservations, I really do like the way XP focuses on the customer.  One of the problems bought up in the group discussion I was involved with was that often developers are forbidden from talking to customers.  I think developers should definitely have access to customers, or at least 'personas' if the real customers are difficult to get to, and that good projects will also employ other people who have experience dealing with customers (such as BA's, integration designers and even usability experts).

Other points raised included that proxy customers must know their place.  Knowing the problem domain does not mean that someone knows how the solution should work.

Another point was the importance of standing up to the customer and challenging them rather than taking everything they say as correct (they are not always right or always able to say what they want).

The danger of XP sounding like a religion
One of the points was that 'dissenters should be tolerated as long as possible but no longer'.  The point was that one 'professional skeptic' can bring everything to a halt.  Another comment in the session was the need to understand the 'rigour of the disciplines'.  While I understand these points, the language concerns me as it sounds too much like XP is a religion.  This problem was acknowledged later in the session when it was highlighted that it was important not to frighten the customer by being a 'religious fanatic'.

Although XP isn't XP without all of the practices, these still aren't enough
Even though there are a lot of points about XP isn't XP without all the processes, it was also mentioned that the core practices aren't enough on their own.  Mistakes are sometimes made on projects where people have 'read the XP books' but not talked to anyone experience (there were a lot of XP coaches in the session). 

Keith said that mentioned that things like version control are 'assumed' in the core practices but need to be done on professional projects.  Other practices included doing some small design up front but avoiding big design up front (BGUF).  UML diagrams were OK, provided they were just on a whiteboard and not written down.  Also the test/code/refactor cycle wasn't enough to guarantee success, you also required knowledge, skill and experience (to know when to stop for example).

One of the comments I heard at the conference was that a lot of the successful XP projects have very talented people working on them and that this was one of the reasons for success. Any project with successful people is likely to do well, irrespective of the process.  To me this doesn't mean XP isn't useful, just that it may not be a processes that can be applied to all teams.

This session was run by Alan Cameron Willis who works with the Visual Studio team out of Cambridge in the UK. Domain specific languages are meant to express requirements and solutions of a particular business domain. Alan is working on the tools like Whitehorse that are visual designers included in Visual Studio that help development teams design and build applications.

We worked in groups to come up with our own domain specific language that we could use. We were asked to imagine that we had created a Point of Sale system for a restaurant and wanted to sell that solution to different types of restaurants (burger bars, bistros, drive-thru's).  Could we come up with a language that could be used to help model, sell or create the software for these different settings?

Alan suggested a range of different communication from text, to different types of diagrams (e.g. UML), to animations. The key issue from my point of view was who the language needed to communicate with and for what purpose. Is the language for communicating requirements or is it to capture requirements and generate the code?

It's nice to see that Microsoft have highly academic involved in creating their software, and that guys like Alan are also turning this into useful tools such as Whitehorse that we can use as developers.

Alan has posted a summary of the domain specific languages discussion on the OT2004 wiki.