By Douglas Messier
NASA’s Inspector General (OIG) has criticized the agency’s practice of allowing SpaceX and Orbital ATK to lead investigations into their own launch failures involving commercial cargo ships, citing a lack of independence and the potential for serious conflicts of interest.
“While not required in the CRS-1 [Commercial Resupply Services-1] contract, this lack of independence does not meet Government best practice standards for NASA, NTSB, and USAF investigations and could impact the board’s ability to identify the root cause and make corrective actions,” the OIG said in an audit released on Tuesday.
“NASA’s official policy for investigations requires all official Mishap Investigation Boards to be independent,” the report adds. “NTSB and USAF have similar requirements. In contrast with these best practices, the CRS-1 contract and FAA license requires SpaceX to conduct its own investigation but does not require company investigation boards to screen for conflicts of interest or maintain independence.”
The criticisms are contained in a new OIG audit, “NASA’s Response to SpaceX’s June 2015 Launch Failure: Impacts on Commercial Resupply of the International Space Station.”
The failure of SpaceX’s Falcon 9 booster sent a Dragon supply ship bound for the space station to the bottom of the Atlantic Ocean. Eight months earlier, an Orbital ATK Antares rocket exploded shortly after launch, destroying a Cygnus cargo vehicle headed for the station.
The Inspector General found that not only was SpaceX allowed to investigate its own failure, but that 11 of the 12 voting members of the investigation board were SpaceX employees. The report acknowledges that expert knowledge is required to investigate failure, but said there were other ways to obtain it.
“We acknowledge SpaceX’s investigation was transparent and the observers from FAA, ISS, LSP, NTSB, and USAF had access to the investigation’s data and analysis,” the report concluded. “However, an investigation led by the employee responsible for the SPX-7 launch and run by the contractor responsible for the failure raises questions about inherent conflicts of interest.”
The report also criticized NASA for not having a clear policy on how commercial cargo failures should be investigated. This has resulted in the space agency deciding on how to conduct accident inquiries on an “ad-hoc’ basis.
“As such, for the SPX-7 launch failure there were up to seven possible investigation authorities depending on when the failure occurred and the extent of damage to coordinate and prioritize,” the report states. “Due to a lack of standardization or NASA policy, the contractor and NASA investigations into the SPX-7 and Orb-3 failures had different scopes and produced varying findings and corrective actions.”
SpaceX’s investigation concluded the “most probable cause for the mishap was a strut assembly failure in the rocket’s second stage. Specifically, the failed strut assembly released a helium tank inside the liquid oxygen tank, causing a breach in the oxygen tank’s dome and the release of gas that in turn disabled the avionics and caused release of the Dragon 1 capsule and break-up of the launch vehicle….
“The company’s post-mishap testing of strut parts from the same purchase order as those used on SPX-7 found material flaws due to casting defects, ‘out of specification’ materials, and improper heat treatment,” the report states.
NASA’s Launch Services Program (LSP) conducted its own investigation into the failure. Officials concluded there were several “credible causes” for the accident, including poor quality control at Elon Musk’s launch company.
“In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure,” according to the report. “LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors.”
LSP’s findings concerned top officials at NASA, which had awarded commercial cargo and crew contracts to SpaceX to service the space station.
“In February 2016, the NASA Administrator and the Associate Administrator for the Human Exploration and Operations Mission Directorate sent a letter to SpaceX expressing concerns about the company’s systems engineering and management practices, hardware installation and repair methods, and telemetry systems based on LSP’s review of the failure,” the report read.
The report concludes that the disparate findings are likely the result of NASA’s lack of a clear policy on how to conduct commercial cargo accident investigations. This caused the two inquiries to diverge in their scope.
“The findings of the contractor-led investigation boards were generally limited to determining the “technical cause” of the failures and implementing corrective actions to replace failed parts or systems,” the report reads.
“NASA’s investigations had broader objectives, but varied in scope and purpose,” the report added. “For example, LSP not only evaluated the technical causes of the SPX-7 failure but also made findings related to the selection, use, and lack of testing of the failed strut assembly as well as a general finding recommending additional measures when using commercial grade parts on launch vehicles.”
The OIG report makes two recommendations on how NASA can improve its investigation of future accidents. The first is to improve communications and coordination with other federal agencies that deal with commercial space, including the Federal Aviation Administration (FAA), National Transportation Safety Board (NTSB), and the U.S. Air Force.
NASA’s management concurred with this recommendation. It said the four agencies are drawing up a memorandum of agreement “identifying investigative authorities, responsibility, relationships, notification procedures, coordination requirements, and reporting responsibilities of the FAA and NASA in conjunction with commercial space transportation mishap investigation.”
The OIG also recommended that NASA “determine the extent to which official NASA mishap policies apply in commercial space launches with NASA payloads”, “describe what types of investigations may occur and the processes to be followed”, and “clarify the scope and purpose of each investigation.”
NASA partly concurred with the recommendation. The space agency said it would “include an appendix on commercial space launches with NASA payloads. The annex will further define how NASA will accomplish parallel investigations to FAA licensed launch investigations.”
However, NASA said it did not want to make wholesale changes to the commercial cargo program. The agency said that “to apply significant portions of the NASA mishap policy to the current CRS contracts would be a fundamental change to the procurement and resupply strategy laid out by NASA in 2008 and which as created an ISS commercial resupply cpaability and fostered U.S. commerce in low-Earth orbit. NASA will not agree to these updates at this time.”
The relevant excerpts from the report follow.
NASA Office of the Inspector General Report No. IG-16-025
NASA’s Response to SpaceX’s June 2015 Launch Failure: Impacts on Commercial Resupply of the International Space Station
NASA Could Improve Investigation Policies and Coordination for CRS-1 Launches
NASA’s official policy for mishap investigations does not directly address the process for failures of FAA-licensed commercial space launches.52 As a result, when a CRS-1 mission fails NASA determines on a case-by-case basis whether to form an ad hoc investigation through its various discretionary authorities (as shown in Appendix D, Figure 7 and Table 12). This determination is based on FAA licensing requirements, CRS-1 and LSP contract requirements, and the ISS Contingency Action Plan. As such, for the SPX-7 launch failure there were up to seven possible investigation authorities depending on when the failure occurred and the extent of damage to coordinate and prioritize.53
Due to a lack of standardization or NASA policy, the contractor and NASA investigations into the SPX-7 and Orb-3 failures had different scopes and produced varying findings and corrective actions. The findings of the contractor-led investigation boards were generally limited to determining the “technical cause” of the failures and implementing corrective actions to replace failed parts or systems.54 NASA’s investigations had broader objectives, but varied in scope and purpose. For example, LSP not only evaluated the technical causes of the SPX-7 failure but also made findings related to the selection, use, and lack of testing of the failed strut assembly as well as a general finding recommending additional measures when using commercial grade parts on launch vehicles.
While the Agency’s SPX-7 and Orb-3 investigations had elements of a traditional NASA Mishap Investigation, they were not as comprehensive as the process described in NASA policy. For example, the Orb-3 Independent Review Team made programmatic recommendations to the ISS Program, while the LSP SPX-7 Investigation did not. In addition, neither of the investigations was directed to determine all elements of a full “root cause” determination – defined by NASA as determination of the cause of the failure, including technical, organizational, and programmatic issues by reviewing the actions of the contractor and all related parties.55 Accordingly, the Orb-3 Independent Review Team used root cause analysis to develop a fault tree that included findings and recommendations related to programmatic and organizational issues, while the LSP SPX-7 investigation team stated they did not conduct root cause analysis but rather focused on the technical aspects of the failure. Had NASA undertaken an official Agency Mishap Investigation for the failures, Agency policy would have required a root cause analysis with comprehensive corrective actions directed at the contractor and the ISS Program to prevent the specific technical cause from reoccurring and to address any programmatic weaknesses that contributed to the failure. Table 7 compares the four investigations to the NASA Mishap Investigation standards.
While a complete NASA Mishap investigation is not required for launch failures under the terms of the CRS-1 contract, in our judgment the absence of more formal guidance for CRS investigations increases the risk that contractor corrective actions may not fully address broader contributing causes.
In addition, NASA lacks a memorandum of understanding with the FAA to coordinate and delegate accident investigation authority during CRS launches involving the FAA, NASA, NTSB, USAF, and contractor.56 After the Orb-3 failure, there was confusion among FAA, NASA, and Orbital on how to immediately respond and impound evidence. While these issues were resolved relatively quickly, NASA officials identified the need for a more formalized understanding between all the affected parties involved in an FAA-licensed commercial space launch failure. Moreover, FAA officials stated there is the potential for the FAA to relicense a company’s launch vehicle before reviewing NASA’s independent investigation of the failure. For example, although FAA officials had access to LSP meetings during the SPX-7 investigation, the FAA did not receive LSP’s final report with findings and recommendations for corrective actions before SpaceX obtained FAA approval to return to flight in December 2015. While the FAA was not required to review LSP’s findings before issuing a license, this uncoordinated approach increased the risk the FAA approved a launch without fully understanding the LSP investigation’s findings and recommended corrective actions. Moreover, according to officials from NASA’s Office of Safety and Mission Assurance, the Agency will always conduct an independent investigation of failures of commercial launches. As such, the Office is currently in the process of developing a memorandum of understanding to better coordinate with all relevant parties and is updating the NASA mishap policy to clarify the process for independent investigations.
Lack of Independence Could Inhibit Contractor-Led Investigations
In our report examining NASA’s response to the Orb-3 launch failure, we found that Orbital’s Accident Investigation Board was not independent.57 Similarly, we found SpaceX’s investigation board was not independent because 11 of the 12 voting members were SpaceX employees.58 While not required in the CRS-1 contract, this lack of independence does not meet Government best practice standards for NASA, NTSB, and USAF investigations and could impact the board’s ability to identify the root cause and make corrective actions.
NASA’s official policy for investigations requires all official Mishap Investigation Boards to be independent.59 NTSB and USAF have similar requirements.60 In contrast with these best practices, the CRS-1 contract and FAA license requires SpaceX to conduct its own investigation but does not require company investigation boards to screen for conflicts of interest or maintain independence. FAA officials stated NASA can implement additional independence requirements for contractor-led investigations through its contracts as long as they do not conflict with FAA regulations.
NASA and SpaceX officials responded that specific expertise in the failed launch vehicle is important for an accident investigation board and that this factor should be taken into consideration even though it may impact the board’s independence. We agree that engineering expertise is invaluable for determining the causes of a failure and developing corrective actions. However, other processes such as NASA’s Anomaly Engineering Board or USAF’s Engineering Analysis Group are available to obtain this expertise without compromising the independence of an investigation board.61
We acknowledge SpaceX’s investigation was transparent and the observers from FAA, ISS, LSP, NTSB, and USAF had access to the investigation’s data and analysis. However, an investigation led by the employee responsible for the SPX-7 launch and run by the contractor responsible for the failure raises questions about inherent conflicts of interest. To independently verify and review the contractor investigations, NASA created its own investigation boards for both the Orbital and SpaceX CRS-1 mission failures. While NASA, Orbital, and SpaceX, have similar incentives to safely and quickly return to flight, the structure of the contractor-led investigations may not result in a full review of all programmatic and organizational contributing factors and consequently these factors may not be fully addressed to prevent future failures.
52 For failures of NASA-owned or -operated launches, Agency policy provides that NASA form a Mishap Investigation Board to determine the root cause and recommend corrective actions. However, this policy only applies to CRS-1 launches once the spacecraft reaches the proximity of the ISS.
53 We also found that four different accident investigation plans existed among NASA, the FAA, and SpaceX (see Appendix D, Figure 6).
54 Technical cause is the condition that directly resulted in the failure and is usually limited to determining what physical part or system literally caused the failure. Orbital’s investigation for Orb-3 recommended replacing the whole engine system and SpaceX’s investigation for SPX-7 recommended replacing the failed strut part.
55 Root cause is an event or condition that is an organizational factor that existed before the technical cause and directly resulted in its occurrence (thus indirectly it caused or contributed to the proximate cause and subsequent undesired outcome) and, if eliminated or modified, would have prevented the technical cause from occurring and the undesired outcome. Typically, multiple root causes contribute to an undesired outcome. Root cause analysis is a structured evaluation method that identifies the root causes for an undesired outcome and the actions adequate to prevent recurrence. Root cause analysis should continue until organizational factors have been identified or until data are exhausted.
56 In 2004, the FAA, NTSB, and USAF formed a Memorandum of Understanding, which did not include NASA, before the CRS-1 contract was initiated.
57 NASA OIG, IG-15-023.
58 SpaceX’s FAA-required investigation plan requires the chairperson and board members to be impartial in their analysis. Although there was no definition of impartiality in the investigation plan, SpaceX explained that the impartiality requirement means board members are able to evaluate evidence critically and objectively to reach conclusions without being subject to financial, political, legal, or interpersonal influences.
59 NPR 8621.1B. Specifically, the board chair must be independent of the activity, the majority of board members must be independent, and contractors cannot be voting board members. According to ISS Program officials, NASA’s official mishap investigation policy is not applicable to CRS launches.
60 In the event of a launch failure under the Evolved Expendable Launch Vehicle contract, USAF has authority to conduct the accident investigation. Depending on the situation and as outlined in the contract, a contractor could conduct an investigation, but USAF has monitoring standards for the contractor-led investigation board.
61 NASA’s official investigation policy allows for creation of a nonvoting Anomaly Engineering Board to examine technical engineering and factual issues. USAF also creates a nonvoting Engineering Analysis Group to assist its Safety Investigation Boards. LSP officials told us they had access to all required technical expertise for their analysis of the SPX-7 failure.
OIG Recommendations & Management Responses
4. Review all investigation authorities and plans during commercial launches with NASA payloads to ensure they are standardized. In particular, NASA should review the contract requirements, ISS Program Office plans, the FAA Accident Investigation Plan, and contractor submitted plans to ensure each references the other and are coordinated and incorporate programmatic and organizational root cause analysis.
NASA Management’s Response: Concur. NASA will be assigned an action through the Space Station to review all plans and recommend updates, as required.
Estimated Completion Date: August 30, 2016
In order to clarify the division of roles and responsibilities in the event of a mission failure, we recommended the Office of Safety and Mission Assurance, in conjunction with ISS Program officials:
5. Improve coordination with other Federal agencies involved in commercial space. For example, consider
a. creating a formal Memorandum of Understanding with the FAA, NTSB, and USAF to coordinate accident investigations;
b. coordinating with other Federal agencies to determine the hierarchy and roles of different investigation authorities during all phases of commercial launches with NASA payloads; and
c. communicating investigation findings and corrective actions to all interested Federal agencies to allow full and informed decisions.
NASA Management’s Response: Concur. NASA continues to improve communication with other Federal agencies involved in commercial space through ongoing cooperative agreements with NTSB, FAA, and USAF. The Quad Agency Working Group was developed to work issues with commercial launches such as, the hierarchy and the roles for mishap investigations. The FAA, NASA, and NTSB, as representatives of Quad Agency Working Group (FAA, NASA, NTSB, and USAF), are currently drafting a Memorandum of Agreement (MOA) identifying investigative authorities, responsibility, relationships, notification procedures, coordination requirements, and reporting responsibilities of the FAA and NASA in conjunction with commercial space transportation mishap investigation, and identifying areas in which the exchange of information, data, and use of resources or services of one agency by another may be requested. The MOA will address how NASA will assist in communicating investigation findings and corrective actions to all interested Federal agencies to allow full and informed decisions. The MOA is being reviewed by the involved Agencies at this time.
Estimated Completion Date: June 30, 2017
6. Update NPR 8621.1B to include commercial space launches with NASA payloads in official mishap policies. In particular, NASA should
a. define commercial space launches with NASA payloads;
b. determine the extent to which official NASA mishap policies apply in commercial space launches with NASA payloads;
c. describe what types of investigations may occur and the processes to be followed in lieu of an Official Mishap Investigation Board, such as an independent investigation board created by NASA; and
d. clarify the scope and purpose of each investigation, such as a NASA defined root cause compared to a technical root cause analysis, and consider the inclusion of programmatic and organizational root cause analysis.
NASA Management’s Response: NASA partially concurs with this recommendation. OSMA will update NASA NPR 8621.1 Revision C to include an appendix on commercial space launches with NASA payloads. The annex will further define how NASA will accomplish parallel investigations to FAA licensed launch investigations. However, updating the NPR to apply significant portions of the NASA mishap policy to the current CRS contracts would be a fundamental change to the procurement and resupply strategy laid out by NASA in 2008 and which as created an ISS commercial resupply capability and fostered U.S. commerce in low-Earth orbit. NASA will not agree to these updates at this time.
Estimated Completion Date: August 31, 2017