GAO: Orion Cost & Schedule Estimates “Not Reliable”

Orion_components

By Douglas Messier
Managing Editor

A new Government Accountability Office (GAO) report (GAO-16-620) says that NASA has used bad on estimating the cost and schedule for its Orion Multi-Purpose Crew Vehicle, leaving the program open to budget overruns and cost delays.

“GAO found that the Orion program’s cost and schedule estimates are not reliable based on best practices for producing high-quality estimates,” the report stated. “Cost and schedule estimates play an important role in addressing technical risks.”

In September 2015, NASA used a joint cost and schedule level (JCL) analysis to determine the agency would need $11.3 billion to meet an April 2023 launch date for its second crewed Orion flight, the GAO wrote. The analysis had a 70 percent confidence level that NASA could meet that goal, as required by agency policy.

“However, NASA’s JCL analysis was informed by its unreliable cost and schedule estimates,” the report stated. “GAO found that the Orion cost estimate met or substantially met 7 of 20 best practices and its schedule estimate met or substantially met 1 of 8 best practices.

“For example, the cost estimate lacked necessary support and the schedule estimate did not include the level of detail required for high-quality estimates,” according to the report. “Without sound cost and schedule estimates, decision makers do not have a clear understanding of the cost and schedule risk inherent in the program or important information needed to make programmatic decisions.”

Orion_Schedule_GAO_July2016GAO found that NASA that working on an aggressive schedule for the first crewed Orion, which is scheduled for August 2021. However, the space agency’s confidence level in meeting that goal is only 40 percent.

“Working toward a more aggressive goal is not a bad practice; however, increasing cost and schedule risk to the program in order to pursue such a goal may not be a beneficial strategy to the program in the long term,” the report stated.

GAO found that NASA is spending most of its Orion budget funding current work and holding the majority of the reserves set aside to deal with cost overruns and delays until the end of the schedule.

“The lack of cost reserves has caused the program to defer work to address technical issues and stay within budget,” the report stated. “As a result, the Orion program’s reserves in future years could be overwhelmed by work being deferred.”

The report made two recommendations to NASA:

  • “perform an updated JCL analysis including updating cost and schedule estimates in adherence with cost and schedule estimating best practices; and,
  • “perform an analysis on the cost of deferred work in relation to levels of management reserves and unallocated future expenses and actual contractor performance.”

In its response, NASA wrote that it did not see a need to update the JCL analysis at this time. The agency said it believes the analysis was sufficiently rigorous, and that the Orion program’s performance metrics are reviewed regularly.

“If the metrics were to show a significant deviation from the plan, then NASA would initiate a formal rebaselining process, which would include a re-assessment of the fundamental program assumptions and associated recalculation of the JCL,” the report stated. “Until that time though, NASA stated that performing a new JCL is not warranted.”

NASA concurred on GAO’s second recommendation, agreeing to perform an analysis on the cost of deferred work.

“NASA characterized its deferral of work to date as task-level deferrals, lasting only several months and not affecting major program milestones or the critical path, but agreed to include an analysis of how these deferrals affect budget reserves and program performance in future routine management reporting,” according to the report.

Key excerpts from the report follow.

Orion Multi-Purpose Crew Vehicle: Action Needed to Improve Visibility into Cost, Schedule, and Capacity to Resolve Technical Challenges

Government Accountability Office
GAO-16-620
July 27, 2016

What GAO Found

The National Aeronautics and Space Administration’s (NASA) Orion Multi-Purpose Crew Vehicle (Orion) program has overcome several technical challenges and made design changes to the crew capsule to reduce risk. Known challenges, however, remain—such as development of the service module and the crew capsule heat shield, among others—that could cause cost increases and schedule delays as the program undergoes integration and test. Technical challenges are inherent in complex programs such as Orion, but if not carefully managed, they could result in cost overruns and schedule delays. For example, the program has identified software development as an area of substantial risk with a potential cost impact of more than $90 million and which may result in schedule delays.

GAO found that the Orion program’s cost and schedule estimates are not reliable based on best practices for producing high-quality estimates. Cost and schedule estimates play an important role in addressing technical risks. In September 2015, NASA established a commitment baseline of $11.3 billion and an April 2023 launch readiness date for the program’s second exploration mission. NASA used a joint cost and schedule confidence level (JCL) analysis—a point-in-time estimate that, among other things, includes all cost and schedule elements and incorporates and quantifies known risks—to establish the commitment baselines at a 70 percent confidence level, as required by NASA policy. However, NASA’s JCL analysis was informed by its unreliable cost and schedule estimates. GAO found that the Orion cost estimate met or substantially met 7 of 20 best practices and its schedule estimate met or substantially met 1 of 8 best practices. For example, the cost estimate lacked necessary support and the schedule estimate did not include the level of detail required for high-quality estimates. Without sound cost and schedule estimates, decision makers do not have a clear understanding of the cost and schedule risk inherent in the program or important information needed to make programmatic decisions.

NASA and the Orion program have made some programmatic decisions that could further exacerbate cost and schedule risks. The Orion program is executing to an internal schedule with a launch readiness date of August 2021, which has a lower confidence level than its commitment baseline. This means that NASA is accepting higher cost and schedule risk associated with executing this schedule. Working toward a more aggressive goal is not a bad practice; however, increasing cost and schedule risk to the program in order to pursue such a goal may not be a beneficial strategy to the program in the long term. According to program officials, the program employs most of its available budget to fund current work and holds most of its cost reserves at the end of the internal schedule. The lack of cost reserves has caused the program to defer work to address technical issues and stay within budget. As a result, the Orion program’s reserves in future years could be overwhelmed by work being deferred. Program officials told GAO that they have not performed a formal analysis to understand the impact that delaying work might have on the available reserves since the program was confirmed. Without this type of analysis, program management may not have a complete understanding of how decisions made now will affect the longer-term execution of the program.

What GAO Recommends

We recommend that the NASA Administrator take the following two actions:

To provide the Congress and NASA a reliable estimate of program cost and schedule that are useful to support management and stakeholder decisions, direct the Orion program to perform an updated JCL analysis including updating cost and schedule estimates in adherence with cost and schedule estimating best practices.

To have a full understanding of the cost, schedule, and safety impact of deferring work, direct the Orion program to perform an analysis on the cost of deferred work in relation to levels of management reserves and unallocated future expenses and actual contractor performance, and report the results of that analysis to NASA management.

Agency Comments and Our Evaluation

In responding to a draft of our report, NASA partially concurred with one recommendation and concurred with a second recommendation. NASA partially concurred with our recommendation to update the JCL analysis, including updating cost and schedule estimates for the Orion program in adherence with best practices.

In response to this recommendation, NASA stated that the agency reviewed, in detail, the Orion integrated cost/schedule and risk analysis methodology during the KDP-C decision process and determined the rigor to be a sufficient basis for the agency commitments. Further, NASA noted that the program’s performance metrics are reviewed regularly. If the metrics were to show a significant deviation from the plan, then NASA would initiate a formal rebaselining process, which would include a re-assessment of the fundamental program assumptions and associated recalculation of the JCL. Until that time though, NASA stated that performing a new JCL is not warranted.

We still contend that NASA should update its JCL analysis that informed its baseline because we found that the cost and schedule estimates underlying those baselines are not reliable as they did not conform to best practices. For example, the program did not conduct a cross-check of its cost estimate, nor did it ensure that source data and estimating techniques were sufficiently documented so that they could be reviewed and replicated. Further, the program’s schedule estimate had significant logic faults and did not convey a valid critical path. Thus, we continue to believe that NASA will be well-served by updating the Orion program’s cost and schedule estimates to adhere to best practices and to perform an updated JCL for the program. With respect to NASA’s statement in its response that it “is concerned that the GAO may not have consistently evaluated all available data in the baseline program when assessing the reliability of Orion’s cost and schedule estimates,” we met multiple times with the program, reviewed extensive documentation, and provided the program with two opportunities to provide additional documentation based upon its review of preliminary results of our analysis. We made changes to our analysis based on additional information provided to us by the program following its review of the preliminary results and believe this report represents a fair and accurate assessment of the extent to which the program met best practices.

NASA concurred with our recommendation to have the program perform an analysis of the cost of deferred work as compared to available cost reserves—contractor-held management reserves and program-held unallocated future expenses—and contractor performance levels, and to report that analysis to agency management. NASA characterized its deferral of work to date as task-level deferrals, lasting only several months and not affecting major program milestones or the critical path, but agreed to include an analysis of how these deferrals affect budget reserves and program performance in future routine management reporting. Given the finite funds available each year and the low level of in-year cost reserves, we believe that until the results of such an analysis are available, it will be difficult to understand the impacts of this deferred work.

Finally, in its response to our recommendations, NASA officials made reference to a statement that we made in this report but did not include its full context. Specifically, NASA included the following statement in its response: “GAO also noted that NASA’s management of schedule reserves on Orion is a viable approach to managing large human spaceflight programs. GAO noted that ‘working toward a more aggressive internal goal is not a bad practice.’” We would like to clarify that the full context of the statement is that while we do not believe working toward a more aggressive internal goal is a bad practice, increasing cost and schedule risk to the program in order to pursue such a goal may not be a beneficial strategy to the program in the long term.

Save

Save

Save

Save