Tearing down the walls

It is pretty widely accepted that one of the major challenges for IT projects is communications between the different parties involved.

The above figure demonstrates how a traditional IT project works, and why communication is such a big issue.

The business have some wishes, which they give to the requirement people, who turns them into requirements, which are handed over to the development team. The development team then develops the system, and hands it over to testing. When the system is tested, it is handed over to operations, which puts it into production.

Such a process is characterized by each group being walled off from each other, often in a real physical sense. It is also characterized by a lack of communications between the groups.

This means that it can take a long time to find misunderstandings, and that the requirements tend to become outdated before they reach production.

SCRUM and other Agile methods have tried to address this in different ways, but even so, communications problems still exists.

The above figure illustrates an IT project using SCRUM. Here the developers and testers form one team, and the requirements specification is handled by a PO who represents the business.

Due to the processes in SCRUM, the communication barrier between the PO and the SCRUM team is reduced, but there is still a barrier. Also, in most organizations, the barrier between the business and the PO is unfortunately as great as the barrier between business and requirements people in traditional projects.

There is also a communication barrier between the SCRUM team and operations.

The fairly recent concept of DevOps have tried to address the communication barrier between the SCRUM team and the operations people.

As the above figure illustrates, this is handled by embedding the operations people in the SCRUM team (this is obviously a simplified description).

Last November I was at a conference in Orlando, where one of the speakers were the PO for the SCRUM process, and he told us that there is a focus on adding DevOps to the SCRUM methodology.

Unfortunately, he also said that they are not going to do anything about adding requirements specification to the methodology.

This is, in my opinion, unfortunately, as it would help reduce the barrier between the business and the rest of the project, and between the PO and the SCRUM team.

So, what can be done instead, in order to remove the barrier between the business and the rest of the project?

Well, I think it will be hard to completely remove the barrier, but I think there are different ways of reducing the communication barrier.

These are, in random order:

• Embedding a business person in the SCRUM team
• Regular meetings
• Techniques such as story mapping

These can obviously be combined.

Embedding a business person in the SCRUM team

This is an idea that was introduced with eXtreme Programming, though it probably existed before that as well.

It is simply getting someone from the business sitting together with the SCRUM team.

This is often hard to do, as it can be hard to get people released from their normal position in order to place them in an IT project, even if they continue to work on what they usually work with. This is partly due to the, probably reasonable, concern that the person will have much less time to do their job. But it is also often partly due to the fact that many bosses don't like not having people sitting apart from their usual team. On top of that, it might be hard for the business to see the value of embedding the business person in the SCRUM team.

If you manage to get a business person embedded with the SCRUM team, it is worth regularly evaluating whether the business person is the right person for the role, or whether there should be a rotation of the business people, in order to keep the focus fresh.

Regular meetings

A simple, but effective way of enabling communication between the business and the rest of the project, is to plan frequent meetings.

The sprint reviews in the SCRUM methodology is one type of frequent meetings, but there should be other meetings, such as requirements meetings between the PO and the business.

These meetings can be used for presenting user stories before they are taking into a sprint, in order to get the input from the business, and clarify any unclear requirements. It can also be used to prioritize the requirements, based on the current situation, and to adjust the scope of the project, based on any developments.

Techniques such as story mapping

Requirements in SCRUM projects are often written as user stories, and there are some techniques that can be used while writing these, allowing the business to get involved.

One of these methods is user story mapping.

No matter what technique is used, it is important that to ensure that the business are willing to get fully involved. If they are not committed, no technique will help reduce the communication barrier.

In conclusion, it is unlikely that it is completely possible to reduce the communication barriers completely in an IT project, but there are methodologies, concepts, and techniques that can greatly reduce them, allowing for a greater likelihood of the successful development of a system which fulfills the needs of the business.

One aspect I haven't touched on in this post, is the extra complications if the project isn't aimed at the internal business, but rather at customers outside the organization. External customers obviously adds an entirely extra complexity to communications, and is a subject for an entirely different blogpost.

Project managers vs Product owners

I frequently come across discussions on the internet about using project managers as product owners in Agile projects.

These discussions often spring out of discussions on what the role of a traditional project manager is in an Agile project, and since project managers often prioritize work in traditional projects, it makes sense to think of using them as product owners, who, after all, should prioritize the work that the developers do.

I disagree.

When Scrum was created, and the roles of Scrum Master and Product Owner came into existence, the people who got those roles, were the traditional project managers - at least according to what Jeff Sutherland said when I got my Scrum Master certification at his course.

That probably made sense back then, but I think it is a bad idea to do so now, for several reasons.

First, and foremost, I think it is a bad idea, because you need the project manager to focus on the project and its priorities. Unless you run an extremely agile project, there is still a need for a project manager taking care of the non-development aspects of the project (e.g. resource planning, dependency management with other projects, and other tasks ignored by Scrum and other Agile methods). In other words, project managers need to focus on the restraints and barriers of the project, and ensure that these doesn't affect the project and its progress.

Product owners, on the other hand, should represent the business/customers, ensuring that they get the most value out of the project as they can, within the boundaries set by the project manager.

Secondly, I think the skillset required to be a good project manager is orthogonal to the skillset required to be a good product owner, and it is best to use people in the role they fits the best.

A good project manager is good at forecasting and evaluating risks for the projects, ensuring that the people inside the project can perform optimally. A good product owner, on the other hand, understands the needs of the business/customers, and can translate this in a way that the developer team (including testers) can understand. This is why good product owners often are business analysts with a developer or tester background.

Thirdly, in most projects, being a product owner is a full time job. The number of developers in a project generally outnumbers the product owner by a fair margin, and having the product owner do project management as well, will slow the project down - or at least limit the number of developers that can be in the project.

All of the above is based on the premise that projects still need project managers, even if the development process (e.g. Scrum) doesn't call for it. If that's not the case, then the only relevant objection is my second point, about the necessary skillset being quite different. But this is a major point! In my experience, the role of the product owner is essential for the success of an agile project, and all too often, the selection of the product owner is taken too lightly.

If you want to find a good place for your project manager, look at the Scrum Master role, which has many similarities, focusing on sheltering the developers and create the framework for the development process.

Agile and pseudo-science

Back before I started blogging about IT-related stuff, I used to write about pseudo-science on my other (very quiet) blog Pro-science, and before I had that blog, I used to comment a lot on a number of science and skeptic blogs.

First commenting and later blogging about science and pseudo-science has helped enable me to be better at detecting woo and pseudo-science in my daily life. Woo is a term used in the skeptic community to indicate something which is based on anti-science and have no scientific validity or data to back it up.

Woo and pseudo-science exists everywhere, and I have found that the field of IT is absolutely no exception. Rather, there are certain areas which is rampant with pseudo-science, often bordering on woo.

One such area is the field of Agile.

The field of Agile is not a well-defined field, but is a loosely collection of methodologies, techniques, and tools, used for systems development. It is commonly understood as having formed as a result of the agile manifesto, but in reality, it started developing before then, and would probably have come into existence as a field, even if the manifesto hadn't been written.

As a software developer, I use Agile methodologies, techniques, and tools on a daily basis. I also go to conferences and meetups about Agile frequently.

Due to that, I can't help noticing that there is not a whole lot of science behind Agile. Not only that, many of the arguments people use to promote Agile is not exactly based in science or data, but rather on pseudo-science, often in the form of popular science.

This summer, I went to a conference which had a talk on Agile and NLP. I didn't go to the talk, so I can't say anything about the actual content, but let's be perfectly clear: NLP is pseudoscience (.pdf) - see also Thirty-Five Years of Research on Neuro-Linguistic Programming. NLP Research Data Base. State of the Art or Pseudoscientific Decoration? by Tomasz Witkowski (.pdf).

If you do a search on the words "Agile" and "NLP", you'll find articles like NLP Patterns and Practices for High-Performance Teams and Achievers by J.D. Meier. If you look at the blogpost, and are in any way familiar with the writing of people pushing pseudo-science, you'll notice the patterns there. There are appeals to effectiveness and popularity, but no actual data to back those claims up. 

It is highly likely that Meier actually believes that NLP helps, but there is no science to back it up - neither in the form of an explanation of how the NLP mechanisms works or in the form of data, backing up the claims of improvement.

This is, unfortunately, widespread in the field, and not just when it relates to NLP.

Agile (as all things related to systems development) is very much dependent on people, and since most realize this, there is a lot of focus on this aspect. 

This results in many speakers on the subject on Agile, will pepper their talk with references to neuroscience, but rather than looking at the actual science papers, they will get their knowledge from popular science books, by people like Malcolm Gladwell.

In 2012, Steven Poole wrote an article for The New Stateman on the subject of popular science books on neuroscience, called Your brain on pseudoscience: the rise of popular neurobollocks. While he, in many peoples' opinion, painted with too broad a brush, there was an important message in the article, which we'll do well remembering - popular science books often have an agenda, and they are simplistic, and, in many cases, get the science completely wrong.

E.g., as a rule, one should expect anything written by Malcolm Gladwell to be wrong.

I understand why speakers and authors tend to use popular science books as sources, rather than the real science articles. With popular science books, there is a good chance that the audience has already read them, and the science is presented in a form which is easy to digest and regurgitate.

Just a pity that the science is often presented simplistic at best and wrong at worst.

Some speakers, such as Linda Rising and Kevlin Henney, instead read the original science papers, and refer to them instead. Given that those two speakers are among my favorite speakers, I hardly think it takes anything away from their talk. It just makes them more accurate.

Having addressed the use of pseudo-science and popular science, I think it is also important to address the fact of data on which to evaluate things.

This is obviously connected to the other problems. Lack of data, leads to bad conclusions.

A simple example of the lack of data, is this: How do we know Agile works? 

Or put in another way: On what basis do we conclude that Agile works better than what was there before?

As far as I've been able to find out, there is no real data to support such claims.

Yes, there have been studies that indicate that Agile works better, but those are somewhat doubtful, since there was no baseline to compare to, no clear definition of what went before Agile, or even what Agile is.

People tend to compare Agile to Waterfall, but as Laurent Bossavit wrote in his excellent book The Leprechauns of Software Engineering, there is no clear evidence that Waterfall was ever used - at least not in the form that people commonly think of when talking about the Waterfall method. 

Personally, I believe there is great value in using Agile, but I am not willing to make any claims without data to back me up, and in order to have that, we need to try to not base the field on pseudo-science and misrepresentations in popular science, and instead try to use real science, and to collect and evaluate data.

Not everything fit into the scientific method, and in a field that is so heavily dependent on people, there will be times where personal preferences will lead the way, but this doesn't mean that we can't do better.

Think of it this way: We would not accept a new UX design without an AB test or some other empirical data to back it up. So why would we accept it when it comes to our methodologies, techniques, and tools?

Is the Agile Manifesto outdated?

Looking through the program for GOTO Aarhus, I saw that one of the tracks is about people and processes. This is a track they have had at GOTO Aarhus for some years, and one that I usually go to most talks at. After looking at the program, I don't think this year will be any different.

The reason I go to this track, is that I feel that the greatest challenges in software development is not related to technology, but rather to the interaction between people - exactly what this track is all about.

Looking back at all the conferences I've been to the last few years, it has been talks about organizations and processes that has challenged my world-views the most, forcing me to re-evaluate my assumptions, and decide whether or not they were right or not.

A simple example - last year I listened to talks by both Jez Humble and Dan North, where they mentioned the fact that one has to understand the trade-offs in order to make informed decisions. Otherwise you don't know whether it is the right choice for your situation or not. This is a simple message, and one which is easy to grasp on the surface, but also one it is easy to ignore, when there is a choice that seems obvious.

Lets take source control, which most of us would always insist on.

Source control is without a doubt a must in just about all projects where there is more than one person working on code (and the majority of projects where there is just one person working on the code). Does that mean that we should always use a source control system? Well, no - we need to look at the individual situation, and decided whether it is appropriate or not, evaluating the trade-offs.

In most cases the trade-off is between risk-reduction versus speed and/or cost. Here most would err on the side of risk-reduction, but it could be that speed is of such paramount importance, that the time to set up the source control would make the project worthless, and in that case, then risk-reduction would be the wrong choice.

Personally, I have never been in this situation, and find it highly unlikely I ever will, but it is important to keep it in mind, even when the choice seems obvious.

This is just one of the ways talks related to people and processes has changed my way of thinking.

Another obvious way such tracks have changed my way of thinking, is to make me more cautious about Agile and especially the Agile Manifesto.

At GOTO Amsterdam 2013, Kai Gib gave a interesting talk How to Focus Agile so it delivers Value to your Stakeholders, where he correctly pointed out that the Agile Manifesto has very little focus on actually providing value to the business side (though it does pay attention to value in the principles). Since providing value is the actual reason for doing a project, it would seem problematic that this is left out of the actual manifesto.

Kai Gibs talk, and similar talks I've heard the last few years, have left me wondering if perhaps it is time to retire the Agile Manifesto, or at least put less emphasis on it. Given the fact that I see it presented less and less often as part of slides at a talk, I don't think I am alone in feeling this way.

Actually, looking at the GOTO Aarhus program, it seems like that even the original signees of the Agile Manifesto might feel this way, since there are two sessions on the first day of the people and processes track called A retake on the Agile Manifesto (part 1 and part 2), where five people will be takein "a closer look at what has happened in the last 13 years since the Agile Manifesto was published and evaluate where the development community is going in the future".

Three of those five people are co-signers of the original manifesto (Martin Folwer, Andrew Hunt and Dave Thomas).

I am very much looking forward to these sessions, and to to what they will bring. Maybe something new and exciting will come out of it.

One thing is sure, I expect that I, and everyone else listening, will learn a lot.

Ending error-driven development, part 2 - adapting agile practices

This post is not about whether agile methods are better than the waterfall model; rather it’s about adapting to the situation, and using the most useful tools for solving the problem. Error-driven development is often plagued by a number of problems, which might be solved, or at least reduced, by using some of the practices from the agile methods, such as scrum or eXtreme programming.

If you are already using an agile method, it might be worth evaluating whether it’s the correct method for the situation.

Anyway, here are some agile practices it might be worth adapting to your project.

Daily stand-up meetings

A lot of the problems in EDD projects are related to breakdowns in communications - not only between the team and the customer, but also between the team members and the team leader or between different team members.

The purpose of daily stand-up meetings is to spread knowledge of what each person has done since last time, what they are doing now, and what problems they have encountered. This allows team members to either help solving the problems or to plan their work accordingly – there is no need to start working on something which will run into the same problem before the problem gets fixed.

In the book Manage It, Johanna Rothman distinguish between status meetings, where everyone tells everyone else what their status is (something she consider a waste of everybody’s time) and daily stand-up meetings, where people tell the others about a) what they have just finished, b) what they are going to do now, c) what problems they encountered (something she consider very valuable). In other words, daily stand-ups can be said to give a status of the current progress for each team member.

While I certainly agree with Rothman that daily status meetings are not as good as daily stand-ups, I would hesitate to make the claim that they are a waste of time. In a project with a real breakdown in communications, status meetings can help get everyone up to speed on the project as a whole. Of course, such status meetings probably shouldn’t be daily, but rather weekly, and they should be discontinued (or at least held less frequently) when the project is back on track.

Work in short iterations

Agile methods focus on working in short iterations (usually 2-4 weeks long) where there is a finished product at the end, which can be tested. By finished, I mean a product with fully integrated functionality which can be tested through the entire system (e.g. from the GUI all the way down to the database).

This allows for continuous testing, and will give early warnings about problems in the requirements, architecture, or technology. On top of that, it has the benefit of demonstrating progress to people inside and outside the team – the psychological value of this cannot be overestimated, in a project where everybody feels that they have been working hard without showing any progress.

This approach also works in systems which are fully implemented, but which are full of bugs. Here the functionality should be fixed so they are bug-free.

No matter whether it is new functionality or existing functionality, they should be prioritized accordingly to how important they are for the customer, and where they are in the workflow of the user. If you have a workflow where the functionalities are used in the order A->B->C->D then you should implement them in that order, even if the customer feels that B is more important than A. The exception of course being if there are alternative workflows which will take the user either directly to B or to B through some other functionality – then it might make sense to implement B before A.

If the project is just one of several intra-dependent projects (e.g. if the service provider and the service consumer is implemented at the same time), it’s important to coordinate the iterations, so any dependencies between the projects are taken into consideration when planning iterations.

Implement functionality rather than architecture

This pretty much follows from the last point, but it’s important to keep in mind anyway. When developing a system, there are a lot of frameworks that need to put in place (caching, data access layers etc.), but once that’s done, the team should stop thinking in terms of architecture. Instead the team should focus on functionality.
An example of developers thinking of architecture instead of functionality is the case where new requirements are implemented layer-wise. E.g. first all the changes are done to the database, then to the ORM etc. This means that none of the new requirements (or changes to the old ones) is done before they are all done. Not a good way to make visible progress, and not something which can be easily tested before the very end.

Consider pair programming for complex problems

I must admit that I am not particularly hooked on the concept of pair programming, as I am not sure that the costs are in proportion with the benefits. If you have two programmers on equal level, then pair programming can make sense, since it can create a synergy effect, but if the programmers are on different levels, then it will quickly turn into a mentoring process. While I find mentoring processes valuable, they have their time and place, and it’s not necessarily in the everyday programming in a project that has hit problems.

Still, if there are complex problems which need to be solved in the system, then pair programming might very well be a very good idea. The benefits of pair programming should in most cases easily be worth the reduced productivity for a time (given most people don’t pair program most of the time, there will be a cost in productivity when doing this). The benefits of having two people doing the code is that they will work together to solve the problems, making it more likely that it’s done correctly, and there will be two people who understand both the problems and the solutions to them (as they were implemented).

Use continuous integration

To my mind, there is nothing that beats instant feedback when there are problems. Continuous integration is a powerful tool in helping giving developers instant feedback about problems with their code, allowing them to fix those problems as soon as they occur.

Continuous integration is, simply put, the practice of continuously committing your code to the code base, where it’s build and have tests run on it (unit tests and smoke tests) to see whether it works as it should.

This doesn’t absolve the developers of running unit tests etc. before they check in. Instead it’s a safeguard against any issues that might have occurred during check-in (forgotten to check in a file etc.).

For more on continuous integration, go read Martin Fowler’s article on the subject.