Joel Spolsky, of Joel on Software is one of my favorite writers in the software industry. He’s insightful, pithy, and practical. But sometimes he and I part ways, at least in practice, and “Evidence Based Scheduling” is one of those times.

The trouble with the “break it into tiny pieces” approach is that the number of pieces quickly explodes so large that the act of tracking them swamps the project. And now, rather than getting wildly behind schedule because you underestimated how long a particular piece will take, you’re behind schedule because you underestimated how many pieces there were. And now you have more project management overhead….

Many of Joel’s points are good, but the basic truism is that our industry has no idea how to schedule anything non-trivial. This has led me to advocate very rapid iterations, which basically breaks everything down into things trivial enough that we can schedule them (similar to what Joel is saying), but this doesn’t help when the goal is to answer “when can we ship a product that we can sell?” We can tell you when the first feature-incomplete iteration will be ready, but not when something actually customer-ready will be.

I agree with Joel about “evidence based” scheduling at the macro-level (rather than the individual task level). That’s how I do all my most accurate scheduling. “Project Hellfire is about as big as Project Snowstorm, and Snowstorm took about 6 months to complete, so Hellfire should be able the same.” But generally we don’t institutionalize that memory, and we ignore those who remember project Hellfire. We assume we won’t make those mistakes again, so of course it’ll be faster. It won’t be. We’ll make different mistakes. And in the end, it’ll probably take about the same amount of time.

But then again, if our employers actually knew and believed how long it would take to really deliver the product, they would never approve the project…. A bizarre fact. If a project is really going to take 2 years to complete, and you say it’s going to take 18 months, you will be accused of sandbagging and run out of the room. If you say it will take 6 months, they will fund you, and be frustrated when it takes 3 years (because it always takes longer when you try to go too fast), but they’ll fund you all the same, and keep funding you for the three years. We tend to accuse engineers of overestimating the schedule, but the truth is we almost always overestimate our skills.