They write the stuff - what we can learn from the way the Space Shuttle software was developed

As a youngster I watched the movie "The right stuff" about the Mercury Seven. The film deified the picture of an Astronaut in my mind as some kind of superhero. Many years later I read and just absolutely loved an article written by Charles Fishman in Fast Company in 1996, entitled "They write the stuff", which did something similar for the developers who wrote the code which put those guys up there.

Read the article here : https://www.fastcompany.com/28121/they-write-right-stuff

I find myself going back to this article time and again when confronted by the question about how many bugs is acceptable in our code, or even just on what is possible in terms of quality. The article explores the code which ran the space shuttle and the processes the team responsible for this code follows.

This team is one of very few teams certified to CMM level 5, and the quality of the code is achieved not through intelligence or heroic effort, but by a culture of quality engrained in the processes they have created.

Quote

In the shuttle group’s culture, there are no superstar programmers. The whole approach to developing software is intentionally designed not to rely on any particular person.

Here are the important numbers for the Space Shuttle code:

The moral of the story here is that the quality everyone desires is indeed achievable, but we tend to severely underestimate the cost which would go along with this level of quality. I have used the numbers from this article countless times in arguments about the trade-off between quality and cost on my projects.

The most important lesson to learn from this case study is that quality seems to depend primarily on the development processes the team follows and the culture of quality which they adhere to.

Quote

It’s the process that allows them to live normal lives, to set deadlines they actually meet, to stay on budget, to deliver software that does exactly what it promises. It’s the process that defines what these coders in the flat plains of southeast suburban Houston know that everyone else in the software world is still groping for. It’s the process that offers a template for any creative enterprise that’s looking for a method to produce consistent – and consistently improving — quality.

As Fishman points out, in software development projects most people focus on attempting to test quality into the software, and of course as Steve McConnell pointed out this is "like weighing yourself more often" in an attempt to lose weight. 

Quote

“Most people choose to spend their money at the wrong end of the process,” says Munson. “In the modern software environment, 80% of the cost of the software is spent after the software is written the first time — they don’t get it right the first time, so they spend time flogging it. In shuttle, they do it right the first time. And they don’t change the software without changing the blueprint. That’s why their software is so perfect.”

Another lesson to be learned is that the process inherently accepts the fact that people will make mistakes. Quality is not ensured by people trying really hard to avoid mistakes, instead it accepts the fact that mistakes will be made and builds the detection and correction of these mistakes into the process itself. This means that if something slips through it is inappropriate to blame any individual for this mistake, the only sensible thing to do is to acknowledge a gap in the process, so when a bug does make it out into the wild the team will focus on how to fix the process instead of trying to lay blame on the person who was responsible for the mistake.

Quote

Importantly, the group avoids blaming people for errors. The process assumes blame – and it’s the process that is analyzed to discover why and how an error got through. At the same time, accountability is a team concept: no one person is ever solely responsible for writing or inspecting code. “You don’t get punished for making errors,” says Marjorie Seiter, a senior member of the technical staff. “If I make a mistake, and others reviewed my work, then I’m not alone. I’m not being blamed for this.”

Over all this article is a great story which is chock full of lessons to learn and pass on.

References

The full article is archived here : https://www.fastcompany.com/28121/they-write-right-stuff

NASA has a great piece published on the history of the software https://history.nasa.gov/computers/Ch4-5.html

Fantastic read- the interview with Tony Macina published in Communications of the ACM in 1984 -  http://klabs.org/DEI/Processor/shuttle/shuttle_primary_computer_system.pdf