Think smaller

Over at my other blog, I mentioned the fact that the Danish Tax Department is going to go through a major overhaul (Denmark overhauls its tax department).

One of the major triggers for this overhaul, is the problems that the Tax Department have had with creating new IT systems in recent years. Two major IT systems have failed - one which was going to calculate the taxation value of property, and another which was going to be used to collect overdue taxes and other debt to the state.

The failure of these IT projects have cost billions of dollars in both wasted development costs and in lost revenue to the state of Denmark.

As a frontrunner to the overhaul of the Tax Department, the Ministry of Taxation had already decided to not let the Tax Department do the development of the property system but instead do it themselves. This was a major break away from the traditional roles of the Ministry and the Department, where the Ministry was focused on law-making, while the Department had the responsibility of enforcing the laws (e.g. collect the taxes), and make the systems necessary to do so.

As part of the Ministry of Taxation's decision of taking the responsibility of developing the property evaluation system, they also decided to do it in-house, rather than using public tenders to get it developed.

I was recently at a talk given by Steen Larsen, the head of development at the Ministry of Taxation, called "A New Way of Doing Public IT", where he explained the things they had done to create a succesful project, which would deliver a working system on time.

This was very interesting, though I think the title was overselling it a bit, as there are several public agencies and departments which do much of the same as what he described.

Listening to the talk, and seeing the ideas behind the overhaul of the department, I can't help worry if they have identified the right problem.

It seems to me that they still have a tendency to think in large monolitic systems.

As Steen Larsen described it, the IT system they were building was both a system for evaluating the properties, and for the case workers to do their work.

It seems to me, that it should be possible to split these areas of concern up, so they could be considered two different systems, and making the sizes somewhat more manageable. Generally, this allows for a greater chance of success.

Hopefully, the tendency of thinking in monolitic systems is only when they describe the systems, and not when they do the actual development.

Project managers vs Product owners

I frequently come across discussions on the internet about using project managers as product owners in Agile projects.

These discussions often spring out of discussions on what the role of a traditional project manager is in an Agile project, and since project managers often prioritize work in traditional projects, it makes sense to think of using them as product owners, who, after all, should prioritize the work that the developers do.

I disagree.

When Scrum was created, and the roles of Scrum Master and Product Owner came into existence, the people who got those roles, were the traditional project managers - at least according to what Jeff Sutherland said when I got my Scrum Master certification at his course.

That probably made sense back then, but I think it is a bad idea to do so now, for several reasons.

First, and foremost, I think it is a bad idea, because you need the project manager to focus on the project and its priorities. Unless you run an extremely agile project, there is still a need for a project manager taking care of the non-development aspects of the project (e.g. resource planning, dependency management with other projects, and other tasks ignored by Scrum and other Agile methods). In other words, project managers need to focus on the restraints and barriers of the project, and ensure that these doesn't affect the project and its progress.

Product owners, on the other hand, should represent the business/customers, ensuring that they get the most value out of the project as they can, within the boundaries set by the project manager.

Secondly, I think the skillset required to be a good project manager is orthogonal to the skillset required to be a good product owner, and it is best to use people in the role they fits the best.

A good project manager is good at forecasting and evaluating risks for the projects, ensuring that the people inside the project can perform optimally. A good product owner, on the other hand, understands the needs of the business/customers, and can translate this in a way that the developer team (including testers) can understand. This is why good product owners often are business analysts with a developer or tester background.

Thirdly, in most projects, being a product owner is a full time job. The number of developers in a project generally outnumbers the product owner by a fair margin, and having the product owner do project management as well, will slow the project down - or at least limit the number of developers that can be in the project.

All of the above is based on the premise that projects still need project managers, even if the development process (e.g. Scrum) doesn't call for it. If that's not the case, then the only relevant objection is my second point, about the necessary skillset being quite different. But this is a major point! In my experience, the role of the product owner is essential for the success of an agile project, and all too often, the selection of the product owner is taken too lightly.

If you want to find a good place for your project manager, look at the Scrum Master role, which has many similarities, focusing on sheltering the developers and create the framework for the development process.

Social science and systems development

A couple of weeks ago, I wrote a blogpost on Agile and pseudo-science, which led to an interesting discussion on my Facebook wall about the lack of science and data in systems development.

As I wrote in my post, there is an unfortunately tendency of systems developers to rely on pseudo- or popular science, rather than proper science. A lot of decisions are made on assumptions for which there is no real evidence, but which somehow has become embedded in the world of systems development - be it project management, programming, testing, or some other aspect.

We tend to think of systems development as off shot of computer science with some cognitive psychology thrown in. While computer science and cognitive psychology certainly are important for creating systems, they are very much related to the "systems" part of systems development, while they hardly add to the processes, methods etc. that adds up to the "development" part.

A comment in the before-mentioned Facebook discussion suggested that we should look at social science when talking about science and systems development.

This seems like a good idea to me. A lot of what goes on in systems development is about humans interacting with each other, rather than about math, algorithms, cryptography or other aspects of computer science.

I once worked on a project where we had a horrible high turnover, which is definitely a well-known sign of a doomed team. Yet this team was one of the best teams you could imagine - continuously over-performing in the sense of delivering over scope, under budget and without flaws. From everything we know about team dynamics, this team should not have performed like this, but it did, and it could have been good to have had someone qualified there to analyse how this could be the case.

Equally it would be good to have someone look at the opposite type of teams - the ones which in theory should perform well, but which continuously under-perform. The ones where good developers suddenly seem unable to actually finish anything, where simple requirements turn into hideous bloated things, and where the only cure seems to be to kill of the team, never to let them work together again.

In both cases we know the anomalies relate to people, the interaction between them, and the culture of the team and the organization surrounding it. We just don't seem to be able to figure out how to create the first type of team, while avoiding the second type.

Given how many systems development projects fail, maybe it is worth looking at these things? And if we do, maybe we should try to find the proper tools?

Social science would seem like a good fit, and it would seem like a good idea for large organizations with many systems development projects, to take a look at how the research methods of social science could be applied to systems development, in order to become better at it.

Hotfix Hell

I am a firm believer in many agile processes and tools, including iterative development, where you work with many deliveries. This allows for early, frequent feedback, and allows you to find errors early, before the project turns into an error-driven development project.

Unfortunately, this is not always possible.

A type of project I often work at, is the large project which spans several years, where the customer involvement is minimal, except at the start and at the end. This sort of project is pretty much doomed to go over time, go over cost, have many errors etc. They are exactly the reason why agile development has become so popular. Even if the project is developed agile, the lack of customer involvement, will mean that the project could in the wrong direction, without anyone finding out, before at the end.

Typical projects where this happens, are public projects subject to tenders. Here the scope of the functionality etc. is determined before the contractors bid on the contract (though often with a clarification phase at the start), and after the bid has been accepted (plus initial clarification phase has passed), the scope, deadline, and price are fixed. The customer will then often not be involved until the final acceptance test phase, where the solution is often found to be lacking (to put it mildly).

As this approach has obvious flaws, there has been an attempt to fix it by introducing sub-deliveries, which each has to pass the acceptance tests. In my experience, there are typically two sub-deliveries before the final delivery at the end.

This approach might seem somewhat agile, and since it gives earlier feedback, you’d think that it would help. Unfortunately, in my experience, it actually worsens the problem.

The problem is in the acceptance test part.

Picture the typical two year project with three deliveries (two sub deliveries and one final). Given the fact that there is a lot of scaffolding etc. at the start, the first sub delivery will fall after one year, with a sub delivery half a year later, and the final deliver half a year after the second sub delivery (i.e. one year after the first sub delivery).

This is, in itself, not an unreasonable schedule.

Unfortunately, the programmers involved will often have to acquire domain knowledge, while doing the scaffolding and early development of functionality, increasing the likelihood of wrong decisions and/or errors in the implementation. Some of this will become apparent as the first deadline approaches, and might be changed in time - unfortunately some it won’t be possible to change it all, and some misunderstandings will only become clear during the acceptance testing.

Since the project is on a tight schedule, the work on sub delivery two starts - often starting by changing the flaws found before the deadline, which they didn’t have time to fix, but also start on new functionality etc.

Unfortunately, the errors will still be in the code submitted to the acceptance test.

Since the errors are found, the acceptance test fails, and the customer rejects the sub delivery as it stands.

What happens then? Well, this is where the hotfix hell starts.

Given the fact that the code submitted as sub delivery one has failed the acceptance test, the developers have to fix the errors in the submitted code, which is now out of date, compared to the code base. This is done by making a patch or hotfix to the code.

The patched/hotfixed code is then re-submitted to acceptance testing.

If this passes, then all is well. Unfortunately that’s rarely the case. Instead, new errors will be found (perhaps introduced by the fix), which will need to be fixed, re-submitted, tested etc. This will take up considerable amounts of time, calender wise, but also resource wise - meaning that programmers, testers, customer testers etc. will use a lot of time fixing problems in the code, which they could have spent on other things. Other things, such as sub delivery two.

Just because sub delivery one has failed the acceptance test, doesn’t mean that work on sub delivery two has stopped - it is, after all, to be delivered six months down the line.

Unfortunately the plan didn’t take into account the hours need to work on sub delivery one after the delivery and/or date for expected acceptance test. This means that sub delivery two is in problems, since the developers won’t have time to do all the work required for it to pass acceptance test.

Meanwhile, sub delivery one and sub delivery two move more and more apart, resulting in developers having to fix problems in obsolete code - this is frustrating for the programmers, and introduces the risk of errors only being fixed in the old delivery instead of being fixed in both, since porting the fixes is difficult.

At some stage, the sub delivery will pass the acceptance test, or (more commonly in my experience) it will be dropped, as the next delivery either is about to be delivered or has been delivered.

Due to the work on the sub delivery one, the second sub delivery is unfortunately either late or in such a mess that it cannot pass acceptance test (or, more likely, both).

This means that when sub delivery two is handed in, hotfix hell starts all over again.

So, how can this be fixed?

Well, one way is to do iterations the agile way. Unfortunately, that’s not particularly likely to happen.

Another way is to base deliveries on the date when the acceptance test of the earlier sub delivery has passed. So in the above example, the second delivery will be handed in six months after the first delivery has passed the acceptance test.
Given the nature of the projects using sub deliveries, this is also unlikely to happen. Often the last deadline is defined by a new law or regulation, and is firm (until it becomes completely apparent that it cannot be done).

A more likely solution would be to take the overhead to hotfixes into account when planning. This would mean that the time spent on hotfixes on sub delivery one wouldn't take time set aside to sub delivery two. The problem with this approach would be that this would make the price higher when bidding, since more people would be needed to finish the work on time, than if one assumes that not hotfixes are necessary. On top of that, it is also hard to estimate just how much time is needed for this (in my experience, everybody vastly underestimates this).

My suggestion would be something more simple. Timebox the acceptance test - and call it something else.

Before the project starts, the customer and the contractor decide how much time will be used for testing the sub delivery in order to make sure that the fundamentals are fine, but without resulting in the developers having to fix obsolete problems.

When the timebox is done, the customer will either accept or reject - a rejection would mean that the customer think the code is so fundamentally flawed that it cannot be used. If that’s the case, the customer and the contractor will need to sit down together and figure out how to get on from there. Perhaps the final deadline will have to be moved, the contractor will have to add more people to the project in order to get it back on track, or the customer will have to become more involved in the development process (e.g. by providing people who can help testing during the development of sub delivery two).

I am well aware that my suggestion breaks with the concept of sub deliveries, but I would claim that the concept of sub deliveries is fundamentally flawed, and instead of helping a problem, it actually makes it worse. Since this is the case, I think we have to re-think how they are used, if at all.

Ending error-driven development, part 4 - setting the team

My last post was focused on problems which might exist in a team, and what measures which can be done to avoid, or at least compensate, for these problems. This post focuses on the roles which should be filled in a team which has problems, and has turned into the error-driven development phase.

As always, these are my opinions and my opinions only. I’d love to get feedback, and hear what roles other people find necessary or unnecessary.

I am not claiming that the roles that I mention are always necessary for a project to succeed, or even that they are necessary when in an EDD project. They are, however, roles that I find will help the project along the way towards becoming a better project.

The roles are not necessarily full-time jobs, and a person could fulfill several roles at the same time, as long as that person has the right skill-set for each of those roles.

So, apart from a project manager, software architects and developers, what roles do I think should be part of a team’s setup?

• QA manager


• Configuration manager


• Deployment manager


• Defect manager


• Test manager


• Release manager


• End user representative

QA Manager

Let’s be honest, when a project is under a great deal of pressure, and has been so for a while, the developers start cutting corners. This is the only way they can remain any hope of making it on time.

This means that it’s necessary to have someone who is responsible for the code quality, and who have the jurisdiction to enforce coding standards.
Depending on how bad shape the code is, this can either be a small task among other tasks, or it can be a full-time job. If there is a lot of legacy code, the QA manager will have a hard time getting the code up to par, even if the legacy code is ignored. It’s a sad fact that code which is being written for a crappy code base will most likely also be crappy – it’s the “broken window syndrome”. Why bother with writing clear, beautiful code, when it’s going to be part of a messy, unclear code base?

The QA manager needs to be respected by the other developers, and must be able to help mentoring less experience developers on how to solve problems in the correct way.

Configuration manager

We probably all know this situation. The tests have gone well, everything seems to work, but when it’s deployed out to the production environment, the system suddenly fails, and no one knows why. After some frantic work, it’s found out that one of the settings in the configuration was wrong – it might have been overwritten by a test setting, it might not have been deployed, or there might be something else wrong. Whatever the reason, it’s a situation which should be avoided at all costs.

This is where the configuration manager comes in. That’s the person responsible for ensuring that the configurations are correct and the right version on all systems. Every time a change is made to the configurations, the configuration manager should be involved.

Deployment manager

Deploying systems is a hard, thankless task, and most developers don’t want to do it. The deployment manager is responsible for the deployment of the system, ensuring that all the correct information is filled out, that the people who need to know that the deployment is happening get to hear it, and that the deployment went well. In case the deployment didn’t go well, the deployment manager is responsible for finding out what went wrong, and to fix the problems.

If you get the right deployment manager, he or she will try to ensure that the deployment procedure gets automated, so he or she will have fewer things to worry about in the deployment phase.

A deployment manager is not the same as a release manager (see below), as the deployment manager is focused on the entire system, while the release manager should only focus on the parts of the system affected by that specific release. This means that while the deployment manager should ensure that each and every subsystem is running after a deployment, the deployment manager is not responsible for smoke testing the system – that’s up to the release and test managers.

Defect manager

When you are in an EDD project you will get a large number of reported defects (called defects for short), and it’s a quite good idea to have a single point of entry for those defects. The purpose for this is to ensure that the developers don’t get disturbed by defect reports all the time, and to make it easier to weed the defects for duplicates and hidden change requests (a future post will go into these differences).

The defect manager should also be the single point of contact, regarding defects, for people outside the team. This means that if a tester has a question regarding some behavior which might be a defect, the tester should talk with the defect manager.

The defect manager should have clear guidelines from the project leader about how the defect should be prioritized, both in relation to each other, but also in relation to other tasks, such as new development.
My rule of thumb would be that critical defects (e.g. parts of the system doesn’t work), should be have highest priority, but otherwise defects should be solved when someone is working on the general area they are connected to. Of course, in the case of defects found in new functionality for the release, they should always be fixed straight away.

Test manager

Given my heavy focus on testing (see my earlier post on the subject), I guess it should not come as a surprise that I feel that there should be a person dedicated to coordinating the tests.

The test manager is responsible for ensuring that all relevant tests are run every time a new release is on the way, that tests are modified as required, and that new tests are added as new requirements appear.

A good test manager should focus on automating as much of the test process as possible (but not so much that the test quality suffer), and ensuring that everything is tested sufficiently. The test manager should also always be on the lookout for new tools and techniques for testing.

Release manager

A release manager is the person responsible for a particular release, keeping track on everything related to that specific release.

During deployment it might be hard to tell the differences between a deployment manager and a release manager, but there are some key differences. A deployment manager is responsible for ensuring that the deployment process goes as it should, while the release manager is responsible for ensuring that the correct things are deployed for that particular release, and that that particular release works as it should after deployment. In other words, the release manager is responsible for making the package which should be deployed, while the deployment manager ensures that the package is actually deployed.

End user representative

If at all possible, there should be at least one representative from the end users in the team. This person should be available to answer any questions the other team members might have regarding the domain the system should work in.

Sometimes a requirement seems ambiguous to a developer, while being quite clear to someone who knows the domain better. If there isn’t an end user representative available, the developer will often end up guessing how the requirement should be understood, and design and implement based on that guess – often resulting in a wrong design/implementation – or the developer will have to waste time finding someone who correctly understand the requirement and pass the information on to the developer.

Having an end user representative on-site, will allow the developer to quickly get feedback on how to understand the requirement, reducing the development time and/or number of wrong design decisions and implementations.

How should the work be divided between the roles?

There is a certain amount of overlap between what can be considered the area of responsibility of the different roles, so I thought I should try to explain how I see the divide.

• Defects should be handled by the defect manager, but the release managers should be aware of what defects are solved in their releases. The test manager gets to say whether a defect has been solved or not (do they pass the tests).


• Configuration managers should handle configuration of servers etc., but should work closely with the release managers to ensure that all configuration changes in the releases are taken care of. The configuration manager also needs to work with the deployment manager to ensure that configuration changes are deployed correctly.


• Release managers should work closely with the deployment manager to ensure that the releases are deployed correctly. By “correctly” I mean that they should not only make sure that the system is up and running afterwards (this is the deployment manager’s job), but ensure that it’s the correct releases which are up and running. The test manager should be the one responsible for testing that all the additions, modifications, and corrections are deployed, but the release manager is in charge of keeping track of what additions, modifications, and corrections are part of the deploy.

For a different way of explaining this, let’s go through the process:

1) A new release is defined, and a release manager is appointed. The release manager work together with the rest of the team to ensure that the release is finished on time.

2) The test manager starts ensuring that there are tests covering new functionality, and modifies any existing tests as warranted.

3) While developing the new release, the code is continuously committed, built and tested. The test manager ensures that all errors found during the testing are reported to the defect manager.

4) The defect manager prioritizes the errors, and make sure that they are added to the tasks for the release if relevant. When doing this, the defect manager ensures that the release manager and QA manager are aware of the open defects.

5) Any fixed defect goes to the test manager, who either confirms the fix or reopens the defect as not solved.

6) The release manager informs the configuration manager of any changes to the configuration.

7) During the development of the new release, the QA manager will do general QA work, enforcing coding standards, do code reviews etc. If any area gets a large number of defects, this area will be in the QA manager’s focus.

8) After the release is done, passing the tests sufficiently, the release manager makes sure that all the relevant code is made into a deployment package, and hands this over to the deployment manager.

9) The deployment manager goes through the deployment steps, ensuring that the deployment goes well.

10) After the deployment has gone well, the configuration manager ensures that the configuration is correct.

11) The test manager tests that the deployed version corresponds with the release version [go through steps 8 through 11 as many times as necessary until it’s right]

12) Finally, the release manager signs off the release, filling out whatever papers are necessary, informing the necessary people about the release etc. [this might be handled by the project leader together with the release manager]

Note that there frequently will be several test environments which the deployment should go through, but the release manager is only responsible for the first deployment, while the rest presumably can be handled by the deployment manager alone.

Ending error-driven development, part 3 - evaluating the team

As I wrote in my original post about error-driven development, there are usually multiple underlying problems which have lead to the project becoming an EDD project. These problems can be technical in nature, but they might also be people-or role-related. While my earlier posts in this series focused on the technical problems, it’s important not to ignore the person- or role-related problems as well, be they internal or external to the team.

When working on an error-driven development project, there is a tendency for the people in the team to focus outwards when trying to finding this sort of problems. This is of course a natural response when a team comes under pressure (aka. “Circling the wagons”), but it’s not really a good response, since it won’t help solve the problems.

In my opinion there are two types of problems which can be related to the people in the team.
1) Qualifications, competences, and inter-personal skills of the team members
2) Roles of the team members

Of course, as always, both sorts of problems might be in play at the same time.
This post will only focus on the first type of problems, while my next post will focus on the roles in the team.

Qualifications, competences, and inter-personal skills of the team members

When a project starts up, a great deal of attention is usually given to the people who become part of the team, since these people will to a large degree create the fundaments of the system, and help shape the future development in the project.
This allows for a false sense of security, since everybody assumes that the people who were picked had the qualifications and competences required to do their job. This is unfortunately not always the case.

An example, which I’m always presented for as a consultant, is the case where there is a key position that’s filled with an external ”expert” (read: consultant with documented experience). Given the fact that this expert obviously have the qualifications (documented experiences in the position), it’s assumed that the expert also has the competences for the position. This is not always the case. Sometimes they have been lucky in the past, and not run into problems; at other times they might have embellished their résumé (putting too much emphasis on one aspect on their work, without making clear that this was only a minor aspect).

Since the external expert was hired exactly because there were no-one available who qualified, there will be no one around to ask questions about problematic decisions.
At some stage, it becomes clear that the decisions might be sub-optimal, to put it mildly, and people will work hard to either compensate for the problems they cause (the typical scenario) or actually have to re-do the work (less typical, but sometimes the better scenario).

I should add that while it might sound like I am trying to insinuate that people like external expert are lying about their qualifications or competences, they might genuinely believe that they are good at what they are doing (see my post on the Dunning-Kruger effect on my other blog for more details on why this could be).

All what I wrote about the hypothetical external expert can go as well for internal team members who misjudge their own qualifications.

Later, when the project has run into problems, more people will be added to the project, without as much focus on their skills, which might cause similar problems.

Here are some steps to avoid these problems.

Peer review

Peer review is of course well-known from scientific publications, but in reality, there is absolutely nothing wrong in doing the same with systems architecture, software architecture, and code. What it simply means, is that something is reviewed, by someone on a similar level (a peer), to find obvious flaws, before it’s accepted. It doesn’t mean that there won’t be any errors, but it means that at least one other person have gone through the product, and tried to understand it, before it’s accepted.

The more fundamental to the project the product is (e.g. the overall layering of the software architecture) the more important it is to get it peer reviewed. Any flaws in the product will cause a lot of grief later and the earlier they are removed, the better.

In case that there is not anyone inside the project who are able to do the peer review, try to look outside the project. Is there anyone else in the same company who can do it? Or can you hire someone for that particular job? The latter option will cost money, but it might save a lot in the long run.

Ensure that the team members have the necessary domain knowledge

Unless you work for a company which only makes a certain type of systems, it will always be necessary to have some kind of domain knowledge when developing a system. It’s not necessary for the developers to know every detail about the domain, but they should understand the fundamentals, and they certainly should understand the details about what they are implementing.

How can you realize that what you’re doing is not making sense, if you understand neither the problem you’re trying to solve, nor the solution to the problem?
In my first blog post in this series, on testing, I made one recommendation on how to ensure that people had the necessary domain knowledge.

With regards to the test cases, my general suggestion is that they are not written by technical people, but rather by people with an understanding of the business domain. Optimally they should be written at the same time as the requirements or at least before the coding really start, and only be in general terms. Before the developer starts developing, he or she should read the relevant test cases, making sure that he or she understands the requirements as stated in general terms. If there are some business concepts that appear unclear, it’s possible for the developer to acquire the necessary domain knowledge before starting on the development.

This is one way, but there are of course others. How you do it, depends on the project and the people in the team. It might require a course, or it might be solved by having one of the end users close at hand, ready to answer any questions which might pop up (something which I would recommend anyway).

Only take on new people if you have time to integrate them into the team

Time after time, projects run into trouble, and management starts throwing more people at the project, in the hope that it will make it possible to finish on time.

That’s not going to work – it just won’t happen.

Adding a new developer to a team is not cost free, and adding several developers at the same time, can be extremely expensive. A team which has worked together for a while, knows each other, and often runs smoothly, even if there are problems with the project as a whole. Adding new team members, without given thought to when and how that’s done, will cause problems.

In 1965, Bruce Tuckman proposed the Forming – Storming – Norming – Performing model for teams, where teams move through different phases (see Wikipedia article). Every time new group members are added, the team will get thrown backwards through the phases, and it will take a time for them to re-group.

Of course, if the team is not running well, adding new team members might be less of a problem, but then, there are probably other issues which should be addressed as well.

Don’t be afraid to get rid of people from the team

This is probably the most controversial of my suggestions, but I know that many people will agree on this.

People in teams depend upon each other, and if a team member can’t get along with the others, or underperforms, it can ruin the entire team. If it appears that’s the case in your project, try to talk with the team member about the problem(s), and see if there is something which can be done, but if there isn’t (or if the problems continue) then don’t be afraid to get rid of the team member.

It might cost you a member of the team, but you’ll find that a good team will perform a lot better than a team which doesn’t work properly. This difference will more than compensate for the loss of man-hours.

Don’t consider mentoring a cost, but an investment

Unless you’re very lucky, not everyone in your team will be experienced and excellent developers when they start on the project. This means that they will have to learn while working on the project.

Learning while working on a project means two things:
1) They will make mistakes
2) They will draw upon the other team members to help them solve problems.

When the going is rough, it’s easy to resent the time the experienced team members have to use to help the less experienced team members. Don’t do that. Consider the time spent on helping the inexperienced team members an investment, helping them reach a level where they can work on equal level with the rest.

If it gets out of hand, boundaries needs to be set, but as long as the experienced team members’ performance doesn’t suffer too much (you’ll have to decide what this constitutes), I recommend letting the other team members draw upon them whenever is necessary.

Create an “ideal” code example

This is something I saw working with success at a customer. They had some general architecture and coding guidelines for the whole company, but realizing that each project would have differences, they also used a concept called “the good code example”. This was a code example in the projects, showing how a typical code implementation would be done correctly in that particular project. The example had to be picked with care, to be representative, and it had to be updated every time changes were made to the coding standards in that particular project, but it served as a great references.

Every time a new member got added to the team, they were given the code example, and told to try to understand it. This worked great for giving people a basic understanding of the software architecture they were going to work in.

Ending error-driven development, part 2 - adapting agile practices

This post is not about whether agile methods are better than the waterfall model; rather it’s about adapting to the situation, and using the most useful tools for solving the problem. Error-driven development is often plagued by a number of problems, which might be solved, or at least reduced, by using some of the practices from the agile methods, such as scrum or eXtreme programming.

If you are already using an agile method, it might be worth evaluating whether it’s the correct method for the situation.

Anyway, here are some agile practices it might be worth adapting to your project.

Daily stand-up meetings

A lot of the problems in EDD projects are related to breakdowns in communications - not only between the team and the customer, but also between the team members and the team leader or between different team members.

The purpose of daily stand-up meetings is to spread knowledge of what each person has done since last time, what they are doing now, and what problems they have encountered. This allows team members to either help solving the problems or to plan their work accordingly – there is no need to start working on something which will run into the same problem before the problem gets fixed.

In the book Manage It, Johanna Rothman distinguish between status meetings, where everyone tells everyone else what their status is (something she consider a waste of everybody’s time) and daily stand-up meetings, where people tell the others about a) what they have just finished, b) what they are going to do now, c) what problems they encountered (something she consider very valuable). In other words, daily stand-ups can be said to give a status of the current progress for each team member.

While I certainly agree with Rothman that daily status meetings are not as good as daily stand-ups, I would hesitate to make the claim that they are a waste of time. In a project with a real breakdown in communications, status meetings can help get everyone up to speed on the project as a whole. Of course, such status meetings probably shouldn’t be daily, but rather weekly, and they should be discontinued (or at least held less frequently) when the project is back on track.

Work in short iterations

Agile methods focus on working in short iterations (usually 2-4 weeks long) where there is a finished product at the end, which can be tested. By finished, I mean a product with fully integrated functionality which can be tested through the entire system (e.g. from the GUI all the way down to the database).

This allows for continuous testing, and will give early warnings about problems in the requirements, architecture, or technology. On top of that, it has the benefit of demonstrating progress to people inside and outside the team – the psychological value of this cannot be overestimated, in a project where everybody feels that they have been working hard without showing any progress.

This approach also works in systems which are fully implemented, but which are full of bugs. Here the functionality should be fixed so they are bug-free.

No matter whether it is new functionality or existing functionality, they should be prioritized accordingly to how important they are for the customer, and where they are in the workflow of the user. If you have a workflow where the functionalities are used in the order A->B->C->D then you should implement them in that order, even if the customer feels that B is more important than A. The exception of course being if there are alternative workflows which will take the user either directly to B or to B through some other functionality – then it might make sense to implement B before A.

If the project is just one of several intra-dependent projects (e.g. if the service provider and the service consumer is implemented at the same time), it’s important to coordinate the iterations, so any dependencies between the projects are taken into consideration when planning iterations.

Implement functionality rather than architecture

This pretty much follows from the last point, but it’s important to keep in mind anyway. When developing a system, there are a lot of frameworks that need to put in place (caching, data access layers etc.), but once that’s done, the team should stop thinking in terms of architecture. Instead the team should focus on functionality.
An example of developers thinking of architecture instead of functionality is the case where new requirements are implemented layer-wise. E.g. first all the changes are done to the database, then to the ORM etc. This means that none of the new requirements (or changes to the old ones) is done before they are all done. Not a good way to make visible progress, and not something which can be easily tested before the very end.

Consider pair programming for complex problems

I must admit that I am not particularly hooked on the concept of pair programming, as I am not sure that the costs are in proportion with the benefits. If you have two programmers on equal level, then pair programming can make sense, since it can create a synergy effect, but if the programmers are on different levels, then it will quickly turn into a mentoring process. While I find mentoring processes valuable, they have their time and place, and it’s not necessarily in the everyday programming in a project that has hit problems.

Still, if there are complex problems which need to be solved in the system, then pair programming might very well be a very good idea. The benefits of pair programming should in most cases easily be worth the reduced productivity for a time (given most people don’t pair program most of the time, there will be a cost in productivity when doing this). The benefits of having two people doing the code is that they will work together to solve the problems, making it more likely that it’s done correctly, and there will be two people who understand both the problems and the solutions to them (as they were implemented).

Use continuous integration

To my mind, there is nothing that beats instant feedback when there are problems. Continuous integration is a powerful tool in helping giving developers instant feedback about problems with their code, allowing them to fix those problems as soon as they occur.

Continuous integration is, simply put, the practice of continuously committing your code to the code base, where it’s build and have tests run on it (unit tests and smoke tests) to see whether it works as it should.

This doesn’t absolve the developers of running unit tests etc. before they check in. Instead it’s a safeguard against any issues that might have occurred during check-in (forgotten to check in a file etc.).

For more on continuous integration, go read Martin Fowler’s article on the subject.

Error-driven software development

When developing software systems, there are a number of systems development types out there, e.g. test-driven development (focuses on making tests before implementing), and what might be called requirements-driven development (focus on finding all the requirements before implementing). Unfortunately, there is a type of development that I all too frequently come across, which I've come to call error-driven development.

Error-driven development is systems development, where everything is done in reaction to errors. In other words, the development is reactive, rather than proactive, and everybody is working hard, just to keep the project afloat, without any real progress being made.

I should probably clarify, that I am not speaking about the bug fixing phases, which occurs in every project, but rather the cases where the project seems to be nothing but bug-fixing (or change-requests, which is to my eyes is a different sort of bug reports), without any real progress being made.

Unsurprisingly, this is not very satisfactory for any of the people involved. What's more, it's often caused by deep, underlying problems, where the errors are just symptoms. Until these underlying problems are found, the project will never get on the right track, and will end up becoming a death march.

The type of underlying problems, which can cause error-driven development, could be things like:

• Different understanding of the requirements for the software among the people involved. Some times the people who make the requirements have an entirely different understanding of what the end system should be like than the end users.


• Internal politics. Some departments or employees might have different agendas, which might lead to less than optimal working conditions.


• Lack of domain knowledge among the people involved. If you're building e.g. a financial system, it helps if at least some of the people involved in the development have a basic idea of the domain you're working within.


• Bad design. Some times early design decisions will haunt you for the rest of the project.


• Unrealistic time constraints. If people don't have time to finish their things properly, they will need to spend more time on error fixing later.

There are of course many other candidates, and several of them can be in play at the same time, causing problems.

No matter what the underlying problems are, the fact is, that just focusing on fixing bugs and implementing change requests, won't help. Instead it's important to take a long hard look at the project, and see if the underlying problems can be found and addressed.

This seems trivial, but when you're in the middle of an error-driven development project, it's hard to step out and take an objective look at it. What's more, you might not be able to look objectively at the process. Often, it requires someone who hasn't been involved from the start, to come and look at things with fresh eyes.

As a consultant who often works on a time-material basis, I often get hired to work on error-driven development projects. The reason for this is simple: often it appears to the people involved, that the project just need a little more resources, so they can get over the hurdle of errors, and then it will be on the right track. When hired for such projects, I always try to see if there are some underlying problems which needs to be addressed, instead of just going ahead and fixing errors/implementing changes. Unsurprisingly there often are such problems.

Frequently these problems can be fixed fairly simply (reversing some old design decisions, expanding peoples' domain knowledge, get people to communicate better, implement a test strategy, use agile methods etc.), while at other times, they can't be fixed, only taken into consideration, allowing you to avoid the worst pitfalls.

So, my suggestion is, if you find yourself in a project which over time has turned into an error-driven development type project, try to take a long hard look at what has caused this, instead of just going ahead and try to fix all the errors/implement the changes. Error reports and change requests are just noisy symptoms in most cases, and will continue to appear as long as the real problems aren't addressed in one way or another.