Sunday, February 15, 2009

That NASA Management Video

I recently became aware of this video on NASA management problems (via zdnet and npr). It was apparently was put together by astronaut Andy Thomas and was acted out by actual NASA employees. According to Hobbyspace it's been known about for a while now.

If the issues acted out in the video are representative of NASA, it raises concerns about NASA's ability to complete large projects on time and on budget that are much more serious than just the usual problems of large organizations.

Specifically, my concern is this: suppose an organization with the management problems outlined in the video takes on a large project. This project is then divided into separate sub-projects with each sub-project corresponding to a subsystem that is specified with a list of requirements strictly based on a set of top level requirements that represent the goals of the system as a whole. The subsystems are then in turn broken down and more requirements are derived for them.

As a simple example, suppose there are 10 subsystems, each with its own project and management team. Each team is working hard to ensure that the subsystem meets its requirements, but no more than that. Because the people in the organization are dedicated and competent, 9 out of 10 projects result in a subsystem that meets the requirements, however one unlucky project team happens to get the subsystem with unknown problems, that couldn't have been predicted at the start of the project. They miss the requirements goals by about 30%. Now, because all the other teams were only trying to meet the requirements, not exceed them, and only managed to exceed them by a few percent at most, there are no easy ways to make up for the loss in performance or capability. It may be possible to ask some of the other teams to do a redesign, but the other teams, not wanting to make it seem that their initial design was flawed, all vouch for their own designs, based on the unarguable fact that their current design meets the requirements.

The only alternative is to push the problem up into the next higher level of integration, where problems with other sub-systems meeting their requirements are starting to accumulate and cause serious performance/cost/performance issues. Eventually, it is found that the entire system design is flawed as measured against the requirements, forcing either costly design changes or reducing the performance requirements to an obtainable level.

Short of deconstructing the entire organization, I can think of two possible mitigation measures that may be taken to protect against the above scenario. The first method is just to ensure that all performance requirements have generous margins. This involves giving the design team a set of performance requirements that are tighter than strictly necessary, with the option of loosening them later on, however it does somewhat corrupt the whole concept of having the requirements fully derived from the top level, and unless communication with the organization is very good, the individual teams may not have a clear idea of how critical the requirement actually is.

The second is to form 'tiger teams' that actively go out and search for bad designs, and correct them. I suppose this could be quite effective, and could also be used as a formidable 'stick' to threaten low level managers with if their team's design is suboptimal, given the ego bruising that would occur if one's own design were overhauled. However, I doubt that tiger teams are often used this way, although I could be wrong. And even if it were the case, a manager might just think it easier to weather the storm, when the tiger team members come knocking on his/her door, judging the risk of being 'caught' with a poor design preferable to a pre-emptive design change making it look like the team got it wrong from the start.

In any case this approach could have the unwanted, serious side effect of fostering a herd mentality amongst the staff, who learn never to stand out from the crowd for fear of being picked out by a bunch of domain experts hunting for a reason to justify their jobs. If done badly, a tiger team approach could be disastrous for employee morale.

Ultimately the best solution is 'organizational redundancy', by which I mean utilizing multiple organizations with differing management cultures and design methodologies, and allocating work strictly on the basis of past performance. Clark Lindsey commented in his blog entry about this video, saying "The only way I see NASA overcoming this problem is to move product development out to commercial companies and letting them compete to build the products". If this were the case, the US taxpayer could have a lot more confidence that their money is being used effectively, and the USA would have a much more vibrant and viable space program.

Unfortunately, given the political realities, an outsourced space program seems unlikely, in which case we can only hope that either the problems shown in the video are exaggerated, or that somehow NASA can reform itself. If the video is being used at management retreats, at least they have gotten to the stage of acknowledging the problem. The next stage is treatment, but I guess we won't know the results until the next big government mandated project. Personally I don't think the US taxpayer will want to wait that long.

Will Ares I and V be NASA's last launch vehicles?


At 2:02 PM , Blogger Robert Horning said...

No, The Ares I/Ares V launch systems will not be the last rockets designed by NASA engineers. They won't even be the last boondoggle/wastes of talent done by NASA. There is a long and glorious past for previous attempts to get something into orbit that never really happened, from paper studies to even working prototypes.

I can go all of the way back the to "Big G" or "Big Gemini" project that tried to put a 5-man crew up into an expanded Gemini capsule to something more recent like the DC-X project or even the X-33. To suggest that any single project proposed by NASA is inevitable and will simply have to be built is betrayed by plenty of examples to the contrary.... although I haven't seen anything given so much focus at NASA since the Apollo projects (including Skylab and the Apollo/Soyuz missions) ended.

The embarrassment facing NASA right now is that private companies may be able to build similar systems (at least to a congressman... the folks that really count here) at a mere fraction of the cost of the Ares launch systems. If Elon Musk gets the Dragon vehicle up to the ISS successfully (even unmanned), it will be hard for even die-hard NASA supporters to continue Ares funding, much less congressmen following in the tradition of Walter Mondale and William Proxmire. May God help NASA if that situation happens, because even somebody with the charisma of JFK (Obama?) couldn't stop NASA from getting gutted to the bone at that point.

At 9:19 PM , Anonymous Anonymous said...

Or the subsystem design teams could talk to each other in an intelligent manner and understand how the systems interact. They could discuss how each systems requirements and operational plans drive the performance of the other related subsystems. Sort of like we just did today. And yesterday... And last week.

Oh, and maybe we could have other teams of people who are more operationally (or task) oriented. They could have a voice on the design team of each subsystem that they relied on to perform the task. Sort of like the team I work on.


At 11:31 PM , Blogger David Riseborough said...

Thanks for your comment. It's the first for the year :-)

I don't think that Ares is inevitable either. But then, on the other hand, I wouldn't like to suggest that the Ares projects are in dire straits, considering the additional funding that has been allocated to NASA as part of the 'stimulus', and since we don't even know who the next administrator will be. Elon's vehicles are potential game changers, no doubt, but sometimes the wheels turn more slowly than we may wish for.
I guess a lot of space enthusiasts and newspace supporters are waiting and hoping for the wheels to fall off the NASA juggernaut it a way that doesn't obliterate the hopes of newspace as well. It's a rather sad state of affairs. (but much preferable to just about any other country's space program) Since NASA is not my country's space agency and I don't put any taxpayer dollars towards it, I try not to be too negative toward it. As I see it, while it is quite reasonable for a US taxpayer to rail against the paradigm currently in place, I feel it would be presumptious to say the least for me to join in the chorus:-) However I do hope at least for a productive and 'peaceful coexistence' between NASA and newspace companies, where it is not assumed by default that NASA somehow offers a safer development path than any private company.
In any case, if NASA were to be gutted as you describe, we can only hope it is with the aim of reconstructing the path to development of space and not terminating it.

At 12:37 AM , Blogger David Riseborough said...

Seems like you work in a nice, well functioning company/organization. Would you care to name it? :-)
I guess it depends on the degree of vertical and horizontal integration within the project. In the video, one thing that stood out was the several layers of management within NASA. Perhaps in a highly stratified project breakdown, the related projects at the same layer talk to each other, and understand the necessary tradeoffs, but beneath that layer there can be less communication and ultimately less interest in the effect of the subsystem's performance on the system as a whole. (my example was probably way oversimplified) This is why Elon Musk talks about employing a minimum number of highly motivated, capable people for his company.
Having someone on each team who is aware of the operational context seems like a very practical and necessary setup. Even so, would you agree that there needs to be some extra encouragement for engineers to shoot for the best solution, rather than the easiest that meets spec. and doesn't depart from the established design paradigm? Or is it more a matter of corporate culture and individual ethic?


Post a Comment

Subscribe to Post Comments [Atom]

<< Home