I very much liked the idea, that a set of very low level instructions, carefully chosen and carefully followed, could create something so complex and beautiful. Jurgen Appelo has expressed similar ideas regarding complexity theory, and how ant hills or bee hives are complex structures created without a master plan through individual bees’ actions. Yet another connection would be local search algorithms, which find maxima in a landscape through looking only at their immediate surroundings.

Next, my brain started linking these ideas to software development. For example, Test-Driven Development. It is in its essence, a set of very simple rules. Write a test, write needed implementation, refactor. These three simple (at least in theory) steps is all that there is to it, when properly adhered to, it can give great results. Designs tend to be much more easy to use, contain less bloat, be more testable (duh!), and so on. Refactorings are another great example. A set of rules or recipes for how to transform a code base into a better code base.

Obviously, choosing the right set of rules to follow is the important part, as a bad set of rules will lead to crappy code just as an unwanted mutation in a human body may lead to e.g. cancer. But I guess the point I was trying to make is that there is immense power in follow simple rules.

Constraints are good.

To me, that statement sounded rather odd. Constraints is a negative word to me, so intuitively I want to remove constraints. Thus, I asked him to clarify. And here’s what he said (or rather, my interpretation of it).

A big advantage of software development is that we can build virtually anything. However, that same fact also represents one of the biggest problems. “Anything” includes all the great things that we envision, but also all the crappy things which we typically produce.

Since the possibilities are endless, in theory nothing prevents us from creating something truly great. But — and here’s the thing — by adding wisely chosen constraints, we can eliminate some of the undesirable outcomes. That means, that even if we screw up a bit, we’re more likely to create something which is at least not half-bad because we’ve eliminated the really bad outcomes. If we stay within the constraints, that is…

Another way to see it is that just as any society needs some laws and rules in order to function smoothly, a software development team does too. Whereas an example in the first case may be “do not kill”, an example in the second could be “do not commit code which breaks the build”.

The rule

• Make a good estimate of the work (in any time unit).
• Multiply that number by itself in months plus one.
• What you get is a reasonable estimate for release date (in the original time unit).

Examples

Some examples:

• Work estimated to one week will take one week.
• Work estimated to two weeks you can expect to be ready in close to a month.
• Work expected to take two months weeks will take about six months.
• Anything estimated to half a year or more won’t be finished. The scope will be changed too much before the project is finished.

While not perfectly scientific, it is actually more or less empirically derived from my own company’s history.

A software project’s bus factor is a measurement of concentration of information in a single person, or very few people. The bus factor is the total number of key developers who would if incapacitated, as by getting hit by a bus, send the project into such disarray that it would not be able to proceed.

How would your company do without you if you became sick for a month? Six months? Quit completely? Or put the other way, if you want to quit, can you get out without leaving a complete mess behind you?

If you’re an entrepreneur, this is even more important. Especially if you’re looking into one day selling your company. Nobody wants to buy a company which is worthless without the seller.

What is the bus factor on your project or company?

I’m getting frustrated by people I talk to who seem to misunderstand the whole point of agile software development (IMHO).

Different ways of reducing risk

The point with agile, as far as I understand, is to reduce risk and thereby costs by postponing decisions until we have as much information as possible. In older waterfall processes, pretty much everything was decided upfront. That wasn’t because people thought it was so much fun to decide everything up front, but because they believed it to be the best way to reduce risk and cost. Because a bug found just before delivery is much more costly than one found earlier.

Agile, on the other hand, tries to use development practices which enable us to postpone decisions, practices which makes the system easy to change even later in the development process. Thus, the code you write should have two qualities:

• doesn’t force you into making unnecessary decisions, and
• be easy to change.

How to reduce risk the agile way

Quotes such as “You Ain't Gonna Need It” (YAGNI) and “Do The Simplest Thing That Could Possibly Work” (DTST) emphasize the first part — not making premature decisions. But while you can typically find people talking about the second part too, it doesn’t nearly get as much attention in people minds as the first one. Most likely simply because it’s not included in the name, in the mantra. I guess “You Aren’t Gonna Need It, But Do Not Forget To Make Sure Your Code Is Flexible” just doesn’t stick as easily.

Actually ensuring that your code is easy to change requires some work — especially while trying to avoid making premature decisions! It means using things like table driven code and putting abstractions in code an meta-data in configuration. It means thinking about what things are likely to change, and what things are not. It means not assuming too much, since you never know how things are going to be used in the future.

Where many people misunderstand agile

Rules and practices are good, but they are just the means to an end. I find that too many people interpret the agile mantras of YAGNI and DTST too literally. They create solutions which are “simple” and “works” and only includes “what’s needed”.

However, the next day, or next month, when requirements and expectations have changed (as they no doubt will) these solutions prove to be hard to adapt to the new situation. That’s because their developers forgot about the second part, making the system easy to change! And I believe forgetting that is forgetting the very core of what agile is about!

You agree? Got another opinion? How do you ensure you’re code is easy to change?

Definitions

According to Wikipedia, a cross-functional team is:

A group of people with different functional expertise working toward a common goal. It may include people from finance, marketing, operations, and human resources departments. Typically, it includes employees from all levels of an organization.

If we instead look at the definition of a cross-functional team in Wikipedia’s Agile Software Development article, we learn that such a team “is usually cross-functional and self-organizing without consideration for any existing corporate hierarchy or the corporate roles of team members.” This seems to suggest that the meaning of cross-functional in agile is often interpreted a bit differently.

Why confusion arises

Part of the problem might be that leaders in the field are sometimes a bit unclear. For example, reading Henrik Kniberg’s otherwise excellent piece on agile testing quickly, you can easily get the perception that he is arguing for the generalist approach. Quotes such as the following certainly helps one reaching such a conclusion.

In agile projects, testing is considered to be far too important to be confined to a single role, a single team, or a single project phase. Having a role called Tester implies that others don’t need to test. In an agile project almost everyone tests.

On the other hand, if you read it more carefully, he is actually arguing for a specialist approach. (Although he plays safe — and increases confusion — by calling them generalizing specialists.) Anyhow, this quote gives a rather other feeling than the previous.

Since the programmers have automated all the boring, repetetive tests, you are free to focus on the hard stuff. The stuff that is difficult to automate. Exploratory testing. Usability testing. Performance test scripts. Figuring out those tricky test cases that only a veteran like you could come up with.

Finally, in his own words:

Agile teams are cross functional, they are made up of generalizing specialists. People who are experts at certain areas, but have basic knowledge of a whole bunch of other areas as well. A team of generalizing specialists is extremely fast at learning and adapting.

Why specialists?

For me, I’m in the specialists camp. As some form of evidence, I’d like to quote Marcus Buckingham and Curt Coffman from the great book First, Break All the Rules.

Great managers do not believe that a productive team has camaraderie as its cornerstone and team members who can play all roles equally well. On the contrary, they define a productive team as one where each person knows which role he plays best and where he is cast in that role most of the time. … The founding principle here is that excellent teams are built around individual excellence.

The nice part about this last quote is that it is not simply anecdotal “evidence”, but actually conclusions drawn by the Gallup organization after interviewing over 80 000 managers!

Your experience?

What’s your experience in the matter? Is specialist or generalist the way to go?

The problem with this comparison is that it’s simply wrong. I’m fine with comparing creating software with creating a car. But the closest thing to an assembly line in software development is the work done by build and deployment scripts, turning source code into executable software. The actual analysis, design, and coding corresponds to developing a brand new car model. It is a creative process without a single given “right answer” and of which experimentation and iteration are natural ingredients.

To use another metaphor, and at the same time cite Mary and Tom Poppendieck from their book Lean Software Development:

Development is like creating a recipe, while production is like making the dish. Recipes are designed by experienced chefs who have developed an instinct for what works and the capability to adapt available ingredients to suit the occasion. Yet even great chefs produce several variations of a new dish as they iterate toward a recipe that will taste great and be easy to reproduce.

If a commit message feels very long, or you feel a resistance against writing it because you don’t know what to write, it has probably been too long since you last committed! If you commit after each task is done, writing a good one-sentence message is a no-brainer.

Then, somewhere about the turn of the century, agile development became the preferred way of creating software in the more trendy corners of the development community. Today, mainstream developers have started to follow and software shops all over the place are introducing Scrum, doing test-driven development or using other agile techniques.

In the second edition of his popular book Code Complete, author Steve McConell describes this by saying that while he was writing the first edition, design zealots argued for “big design up-front” (BDUF), and as he was writing the second, software swamis instead argued for “emergent design”, which basically means no initial design at all. As he put it:

In ten years the pendulum has swung from “design everything” to “design nothing”.

Now it seems to me that the pendulum is on its way back.

In an article in IEEE Software, the July/August 2008 issue, Rebecca J. Wirfs-Brock argues in favor of up-front design.

These days, some developers shy away from even suggesting that a project might need up-front design and experimentation. They equate any up-front work with “big design up front” (BDUF) and totally wasted effort. Up-front design needn’t be wasteful if you develop a design rhythm that balances thinking, learning, and doing. You’ll build confidence and sleep better at night.

On a similar note, Justin Etheredge argues that some agile developers might be a little bit naïve.

A lot of developers think that agile development represents doing no design up front. That we will figure out what we want to do and then just jump right into coding it without any concern for what we are going to build. Then we will just come back later, refactor the whole thing, and voila, instant working application. Well, it just doesn’t work that way.

Frans Bouma adds to the theme, possibly suggesting that even laziness might be a part of it.

Yes, Software Engineering is hard, deal with it. Don’t use the excuse that because goals are unknown, because the client changes his mind every day the design therefore has to be done in the code editor. If those are your problems, solve them, deal with them.

My own earlier posts deal with similar issues, such as developers who do TDD without thinking, and that you can't skip design.

To be fair to gurus arguing for the principles of agile development (which to a large degree I myself believe in), these comments apply more to average developer Joe than the gurus themselves. But then, the gurus are probably smart enough to succeed in building their software no matter what process they use. There are however way more average Joes than gurus, so the points above are still very valid.

Given this shift of thinking, where will we head now? Will the pendulum actually swing right back to BDUF? Will we find the sweet spot between emergent design and up-front design? Or will we rush off once again in a new chase for design nirvana? If so, where will we end up this time? I sure hope for the sweet spot option, but given human nature I’m not convinced. What are your thoughts?

I can’t help to wonder, however, what’s so magical about 40 the hour workweek? Why not 35, 45, or even 55? Also, it will naturally vary among different people. So I want to know where to draw the line. Just playing safe on 40 seems a waste, if my team can do better.

The capacity model

In the figure above, I’ve created a simple model for how I imagine things work. It’s pretty safe to assume that everyone has a maximum productivity level, over which it is simply not possible to get. I’d say this maximum is defined by the programmers talent. No matter how much experience, practice or training one gets, there’s still some basic inherent talent level which defines one’s maximum. (If nothing else, let it be the number of words per minute the developer can type.)

More interestingly, we’ve got a current capacity which is the highest amount we can currently produce without lowering quality. A sustainable pace is any pace which is below this current capacity. That is, we can keep going at this rate forever, without accumulating any technical debt. We can go above it once in a while too, in a coding crunch, but then we start accumulating technical debt. This forces us to work at a lower rate later to pay back.

How do you go faster?

There are at least two interesting questions in this model. First, how do you find your current capacity? I believe that question can be answered empirically by increasing development speed until developers start getting (too) tired and bugs start to slip in. The second, and more interesting, question is: how do you raise your current capacity? To answer this, we must first answer another question which is: what defines your capacity? (In the figure, this unknown entity is called x.) My best bet is motivation with a tad of discipline thrown in for good measure.

To sum up, my point is that while we want to achieve a sustainable pace in development, we shouldn’t have to limit ourselves to 40 hours per week. If we can hold a sustainable pace of say 45 hours per week, as opposed to 40, we gain more than one month of extra development time during a year! Especially in start-ups, such as where I work, 40 hours a week is a luxury one cannot afford. Large companies such as Microsoft and Google also very much rely on peoples desire for and ability to work (much) more than 40 hours per week. So the question is, how do we increase our development pace while keeping it sustainable? Is it possible? (I sure hope so.) If so, how do you do?

Traditional vs Test-Driven

The traditional way of developing (if there is such a thing) typically follows this pattern.

1. Implement a feature.
2. Test follows from implementation.

Whereas Test-Driven Development reverses that order.

1. Test a (missing) feature.
2. Implementation follows from test.

The idea here is that if we design and write the test before concerning ourselves with a particular implementation, we are more open-minded and the resulting feature will be designed as we want to use it rather than how it’s easiest to implement.

Thoughtless Development

Some people (myself included, in an earlier life), tend to get this idea slightly wrong. They end up with, what I call, Thoughtless Development. When converting from traditional development to TDD, they know the implementation is supposed to go last and follow from the tests.

For some reason though, they keep the “test follows from implementation” part. But as the test is supposed to be written first, there is no longer anything it can follow from. To make it even better, it is also supposed to be as simple as possible, which is easily interpreted as easy to do as possible. The flow typically ends up something like this.

1. Test follows from (vague expectations of) implementation.
2. Implementation follows from test.

Do I need to add that they typically end up with a heap of junk? That’s because there is no room for thought in this process. All the thinking and care that should go into designing tests is forgotten.

As I’ve written about earlier, some people tend to use TDD as an excuse for doing no design at all. Either purposefully or by accident. This is ironic, as TDD is really all about design! So let us all make sure that we are doing Test-Driven Development, and not Thoughtless Development.

This procedure is typically repeated a whole bunch of times. Thus, my thought process is a very iterative one. Unfortunately, I’m not in charge of the iterations. They usually end not because I’ve reached a new milestone, but because I’ve realized I’m hopelessly lost and need to go back to the beginning. Somehow still, (on good days) each iteration seems to make things at least a little bit clearer. Or at least every other iteration or so. Some of them just end in a big hopeless sigh.

Anyway, I finally either realize that I just can’t get it right, or I find a solution. If the latter, the solution typically has very little resemblance to my starting ideas. My thoughts have been distilled and refined over and over, and my initial assumptions have been overturned more or less completely. Actually, the solution typically is far more elegant and simple(!) than what I started with.

So why am I telling you this? Well, I’m not sure (still trying to get my head around that one ). But somewhere in the back of my head I wonder if this is not how I ought to develop software as well. Not that all iterations end in catastrophy, but that whenever I’m not quite sure of what I’m doing, I should throw whatever code I’ve been writing away and start over. Then, finally, the code I end up with will be simple enough for me to grasp in one go and only contains what I’m sure is absolutely necessary. Then of course, it is very hard to actually throw away code you’ve written. But maybe it’s worth it?