We've all been through the cycle. Horrendous "waterfall" project management, with insane deadlines, endless crunching, overtime, stress, mistakes everywhere, only to finish up with a sub-standard product that despite all the time, effort and suffering that went into it, just isn't very good.
And then... then we stumble upon a great prophet. He teaches us - we listen. He utters truths so profound you wonder how you didn't realise this yourself already. What is this magic I am hearing oh teacher? The room is in silence, everyone on the edges of their seats, hanging on his every word. He pauses dramatically, before he speaks the word that will change your life forever - agile.
It's almost ecstasy as you hear it. Suddenly it all makes sense, everything is clicking into place. Your mind is racing, your enthusiasm and vigour for software development is renewed, you immediately login to GitHub and open a PR. This time it's going to be different. This time we will reach the promised land!!! No more midnight coding sessions, no more technical debt, we are FREE!!!!
Then you hear it. At first you're confused - you don't understand. You look behind you and realise there was someone there you didn't notice before. An older man, longer hair, a scruffy beard, sipping a cup of Yerba Mate. Then it dawns on you, he is actually scoffing. How can this be? Why does he not rejoice with the rest of the team? You strain to hear him above the clamour of the other devs celebrating, popping champagne and high-fiving, but you do. "What a load of bollocks" you just about catch as he gets up and walks out of the room.
Very strange. He must not understand. I guess some people are just dinosaurs, unwilling to change their ways. He's probably the reason the last project was so behind, we can do without the naysayers - onwards!
You arrive at work next Monday morning energised and ready to go. You soak in the glorious site of your new Kanban, read your first user story, and the rest is history.
Fast-forward and you're in a stand-up meeting, again. They're going around the table, it's about to get to you. You're desperately thinking of the best way to explain why your ticket is still behind schedule, again. The scrum master calls out your name, and you are summoned to give your status update. It's bad, it's a red. You're asked if you can stay behind to get it shipped before the end of the sprint. You reluctantly agree, the sprint deadline is in 3 days, maybe if you can work extra to get it done today you can actually take a bit of time off! Except a part of you knows, deep-down, that isn't going to happen. You can already see the disappointment in the scrum-master's eyes on Friday morning when you are going to have to explain the ticket still isn't done, and that it's going to roll into yet another sprint. Oh the shame.
Slowly but surely, life settles into the exact same pattern of work, deadlines and stress that the waterfall project managers rained down upon your heads. You're wondering if it's all worth it, when, like a movie, 5 words ring in your ears. You can hear them as clearly as the day you heard them - "what a load of bollocks". You look around, but the wise old developer is nowhere to be seen. How did he know? What went wrong?
Is the old man right?
Kind of.
The big challenge in commercial software engineering is that the domain experts don't understand code, and the software developers don't understand the domain they are operating in. Fundamentally that's it.
You can have as many meetings as you want but that gap will always exist, and it's a source of endless friction in development teams.
However, I would say none are more guilty of this than accountants. It's not their fault, to be fair. They have been trained on an accounting technology that's been around since the renaissance. The thinking goes something like this:
You have a factory, it makes widgets (like all good factories do). You can purchase a new widget making machine, it costs $1m. It can produce 1000 widgets a day, which we can sell with a net profit margin of $1. Do we buy this machine?
Well, it's going to net us $5000 per week, which over the course of the year equates to basically a quarter of a million dollars. The machine will have recouped the investment in 4 years, and will last in operation 10 years before needing to be replaced, so we get 6 years of profit out of it. Sign the check please!!!
The accountant will then produce their budget at the start of the year, submit it to the board of directors for approval, then track each income and expenditure against this budget to see if they are "on track". If it's under, that's good, if it's over, that's bad. End of story. Rinse and repeat next year.
Software is different
But software doesn't work that way. If we were to treat the application being built as a machine, the numbers would go something more like this:
We have a factory, it makes widgets. We are not exactly sure what the profit margin on widgets is. We want to build a new widget making machine. How much that costs depends on how good you want it, how many widgets it will turn out, and how much profit margin you want on each widget.
That's it basically, there are no numbers.
Accountants will not like this. If you've ever been in a management role you know they will press you for an answer. You try to give vague estimates, you try to explain how we are "doing agile", and that we need something more flexible.
You are over-ruled, and instructed to submit an estimate. You make up some numbers, fill in the spreadsheet, and hope for the best. Here's the problem. Now the business managers and finance team have an expectation, they are expecting some software to be built. It should be ready by a certain time (maybe Q3), it should be to a certain standard, and it should generate the agreed-upon amount of revenue. So as the deadline approaches, the pressure mounts. Decisions made earlier in the year come back to bite you and add delays. People get annoyed, you get pressured into making the devs do overtime so you can "hit the deadline".
The only thing that gets hit is your beautiful agile methodology. You keep doing stand-ups, but you turn your kanban into a bullshit backlog and revert to classic "waterfall" development.
I put that word in quotes because of the hilarious fact that waterfall development never existed. I encourage you to go and read the original paper "Managing the Development of Large Software Systems" published by Winston W. Royce in 1970. The diagram that gets faithfully wheeled out by agile advocates is part of a chapter where he's explaining the worst possible way to develop software, and goes on to explain what basically amounts to agile development.
So agile is not really new, and good engineers have always known that the only way to build complex systems is to incrementally work your way towards the destination, discovering as you go rather than trying to know everything up front.
Agile Budgeting
The truth is you can't do agile development in a company that does not do agile budgeting. They go hand in hand and failure to realise this is what leads agile projects to fail. Ultimately if the guys that write the checks are not on board, then it's not going to happen. End of story. But what is agile budgeting?
Think of it more like exploring a cave for minerals. Think of the insanity of being asked, when all you can see is a hole in a mountain and a dream of mineral wealth, exactly how much it's going to cost to extract the minerals, how long it will take, and how much money you're going to make.
You can't do that. It's impossible.
What would make more sense would be to break the project up into stages. Maybe you agree to spend $5000 on an expedition into the cave, to map out some of the main tunnels and take some samples.
You then bring this data back, and you find a large fairly straight tunnel with some promising mineral deposits. Because of it's proximity to the surface, you can estimate a lower initial mining cost. Maybe it's only going to cost $100,000 for the machines you want, only 6 weeks to dig out the tunnel entrance, and based on the samples you estimate at least $1m worth of minerals in that area.
Or maybe you find it's a sprawling network of thin tunnels, each with tiny deposits. You suggest an open pit mine that needs different equipment. Probably the initial cost will be much higher, but the estimates of the amount of minerals is in excess of $5m.
So now you can get authorisation to spend $250,000 to mine for 6 months and see if the mineral returns are in line with what you expect. You know that this money might go down the drain as you learn that the seams are not as rich as you calculated.
Think of agile budgeting as something where, initially, you are humble enough to admit you don't really have a good idea of what this is going to cost. You can make up some numbers, and you can probably get in the right ball-park (mining is never going to be done on a budget of $50, but is also never going to cost $1tn).
But you agree to invest chunks of money into learning about the problem domain, so that you can create a better estimate that is more accurate. And as the project goes on you get a better and better idea, until you have enough information to decide if it's worthwhile continuing or not.
Obviously at the start of the project you have a very wild estimate, and on the last day of the project you know with 100% certainty what it cost to deliver. So somewhere in between that you will have a confidence interval that you are happy with to make a decision.
You should know what that interval is, and realise it's going to cost you money to find out.
Reality doesn't care about you
You can argue with people about this until you're blue in the face. They will insist you can create accurate estimates from "similar projects" and so on.
And yes, that can help. Yes the more experienced you are, the better your initial estimates will be. But until you start digging, you honestly have no idea. You could stumble upon anything.
You might find customers won't buy something of the quality you built, and the market demands something much better. You might find performance issues you hadn't considered, you might realise that your initial data structures aren't a good fit for the problem domain.
People also have quite limited imaginations, and often it's not until you start showing them the front-end do they start to think of all the great bells and whistles that they really really have to have in the project right now!!!!!
So you can be bullied into making inaccurate estimates, but that doesn't make them accurate. It just makes accountants feel better. When the project gets into the weeds and the delays start coming, there's nothing they can do about it because it's just the reality of it. If the minerals are 200m deeper than you thought, then that's where they are. No spreadsheet is going to change that.