NASA Investigation into SpaceX’s Falcon 9 Explosion Questions Single Strut Theory

Dragon capsule separated from Falcon 9 launch vehicle.
Dragon capsule separated from Falcon 9 launch vehicle.

While SpaceX blames a faulty strut supplied by a contractor for the explosion of a Falcon 9 rocket in June 2015, an independent investigation by NASA Launch Services Program (LSP) concluded there were several “credible causes” for the accident, including poor quality control at Elon Musk’s launch company.

“In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure,” according to a report published on Tuesday by the NASA Office of Inspector General (OIG). “LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors.”

The information is contained in a new OIG audit, “NASA’s Response to SpaceX’s June 2015 Launch Failure: Impacts on Commercial Resupply of the International Space Station.” The report says LSP failed to find a probable cause for a failure that sent a Dragon supply ship carrying cargo for the International Space Station to the bottom of the Atlantic Ocean.

SpaceX’s investigation concluded the “most probable cause for the mishap was a strut assembly failure in the rocket’s second stage. Specifically, the failed strut assembly released a helium tank inside the liquid oxygen tank, causing a breach in the oxygen tank’s dome and the release of gas that in turn disabled the avionics and caused release of the Dragon 1 capsule and break-up of the launch vehicle….The company’s post-mishap testing of strut parts from the same purchase order as those used on SPX-7 found material flaws due to casting defects, ‘out of specification’ materials, and improper heat treatment.”

LSP’s findings alarmed officials at NASA, which awarded commercial cargo and crew contracts to SpaceX to service the International Space Station.

“In February 2016, the NASA Administrator and the Associate Administrator for the Human Exploration and Operations Mission Directorate sent a letter to SpaceX expressing concerns about the company’s systems engineering and management practices, hardware installation and repair methods, and telemetry systems based on LSP’s review of the failure,” the report reads.

The OIG reports that SpaceX has taken a number of corrective actions to address concerns about the strut and its processes.

“The company also reviewed the certifications of all spaceflight hardware and altered its quality control processes to better align with NASA technical standards,” the report reads. “In order to track completion of its corrective actions, SpaceX is updating its process for identifying and resolving work-related tasks, which allows for improved auditing, prioritizing, and tracking of fracturable hardware.

“To administer its updated quality control process, SpaceX has reorganized into three teams called ‘Design Reliability,’ ‘Build Reliability,’ and ‘Flight Reliability.’ Besides monitoring corrective actions taken as a result of the SPX-7 failure, these teams are tracking the significant upgrades SpaceX has made to the Falcon 9 launch system for future launches, including increased thrust capability with a new fuel mixture and corrective actions on software implementation plans, which are both rated as low risks by the ISS Program,” according to the report.

The relevant excerpt from the report follows.

NASA Office of the Inspector General

Excerpt from

NASA’s Response to SpaceX’s June 2015 Launch Failure: Impacts on Commercial Resupply of the International Space Station

SpaceX’s Return to Flight Plan

Following the SPX-7 failure, SpaceX recovered parts of the Falcon 9 rocket and, through telemetry analysis and other testing, determined the most probable cause for the mishap was a strut assembly failure in the rocket’s second stage. Specifically, the failed strut assembly released a helium tank inside the liquid oxygen tank, causing a breach in the oxygen tank’s dome and the release of gas that in turn disabled the avionics and caused release of the Dragon 1 capsule and break-up of the launch vehicle. SpaceX completed an extensive analysis of the SPX-7 failure, consulted with NASA and the United States Air Force (USAF) regarding their analysis, and provided a mishap report and Return to Flight Plan to the FAA and NASA in November 2015. The company’s post-mishap testing of strut parts from the same purchase order as those used on SPX-7 found material flaws due to casting defects, “out of specification” materials, and improper heat treatment.23

NASA’s Launch Services Program (LSP) conducted a separate, independent review of the failure, briefing its results to senior NASA leadership on December 18, 2015.24 LSP did not identify a single probable cause for the launch failure, instead listing several “credible causes.” In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure. LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors.25

SpaceX has taken action to correct the deficiencies that led to the failed strut assembly and to address NASA’s concerns by conducting inspections, replacing suspect parts, and conducting additional testing. The company also reviewed the certifications of all spaceflight hardware and altered its quality control processes to better align with NASA technical standards. In order to track completion of its corrective actions, SpaceX is updating its process for identifying and resolving work-related tasks, which allows for improved auditing, prioritizing, and tracking of fracturable hardware.

To administer its updated quality control process, SpaceX has reorganized into three teams called “Design Reliability,” “Build Reliability,” and “Flight Reliability.” Besides monitoring corrective actions taken as a result of the SPX-7 failure, these teams are tracking the significant upgrades SpaceX has made to the Falcon 9 launch system for future launches, including increased thrust capability with a new fuel mixture and corrective actions on software implementation plans, which are both rated as low risks by the ISS Program.
___________________

23 A casting defect is an irregularity that occurs when molten metal is poured into a mold and cooled. An “out of specification” material has a technical attribute (e.g., chemical composition, mechanical property) outside of the prescribed values for the type of metal specified for a particular use. Heat treatment at accurate temperatures strengthens metal parts while improper heat treatment can cause deviations or weaknesses.

24 LSP purchases commercial launch services for NASA customers, including missions of the Agency’s Science Mission Directorate. LSP had a contract with SpaceX to use the Falcon 9 to deliver a science mission payload.

25 In February 2016, the NASA Administrator and the Associate Administrator for the Human Exploration and Operations Mission Directorate sent a letter to SpaceX expressing concerns about the company’s systems engineering and management practices, hardware installation and repair methods, and telemetry systems based on LSP’s review of the failure.

Save

Save

Save

Save

Save

  • Hug Doug

    It’s the final report from the OIG of an audit of NASA’s handling of the investigations into both the Orbital and the SpaceX launch failures, it’s packed full of interesting details on the commercial cargo program, and is well worth reading for that alone. Far from a rehash, it’s an analysis of what NASA did well and what NASA did not do so well. Not a waste of time and resources, as it has recommendations for what NASA could do to improve investigations in the future.

  • Hug Doug

    The strut was over-designed by a factor of 3. It should never have failed.

  • TomDPerkins

    If that’s what the report says, why is Mr. Messier obsessing with and only writing about the unsupported claims SpaceX did not do a thorough enough examination, and that it changed practices found to be unsatisfactory?
    Cause he doesn’t talk about what you’re talking about.

  • Hug Doug

    This wasn’t a NASA report. It’s the final report from the OIG of an audit of NASA’s handling of the investigations into both the Orbital and the SpaceX launch failures, it’s packed full of interesting details on the commercial cargo program, and is well worth reading for that alone. It’s an analysis of what NASA did well and what NASA did not do so well, and it has recommendations for what NASA could do to improve investigations in the future.

  • redneck

    I did not mean to imply based on nothing. I meant that there are always things visible later on, especially to nit pickers. Wrench dropped on foot and “was he wearing his hard hat” type stuff.

  • Hug Doug

    What he says isn’t really supported by the report. I’d recommend reading it in full.

    There’s a link to a PDF of the report here:

    http://forum.nasaspaceflight.com/index.php?topic=40621.0

  • Christopher James Huff

    It’s stored in the LOX tank to keep it cold, allowing the helium tank to hold more helium. It’s warmed as its drawn off and used to pressurize other rocket components.

    No idea if the LOX tank itself was autogenously pressurized or used the helium. Seems like there might be issues maintaining pressure in a tank full of subcooled LOX and gaseous O2, and the same would go for subcooled CH4.

  • Douglas Messier

    Tom’s comments here have become abusive. He’s banned.

  • I can’t report on documents that haven’t been released and that I do not have. You want details from the LPS report. Request that NASA release the document.

  • Douglas Messier

    It is difficult to report on the details of unreleased documents that I do not have copies of. Why don’t you take all the energy you’ve directed toward this argument to trying to convince NASA to release them? Yelling at me isn’t going to do that.

  • Douglas Messier

    Can report on documents that NASA hasn’t released and that I don’t have.

  • Aerospike

    No humor, intentional or unintentional, could be found in this discussion.

  • Aerospike

    A possible “mistake” according to public opinion: hiring an “imperfect” company to do stuff instead of doing it in house or contracting a more traditional and “mature” contractor like ULA.

    I really don’t read that report (or summary of reports) as “SpaceX’s conclusion is wrong or questionable”, it even reinforces the probably failure scenario.

    There is a fundamental difference however: SpaceX seems to have been focused on finding and fixing the technical cause of the problem. The NASA panel looked at the incident from a more broad perspective and points to possible (not proven) problems regarding SpaceX’s procedures. AKA “how could this happen in the first place?”
    Namely:
    Selection of suppliers, inspection/review/control of suppliers,
    quality control of parts, handling of parts, etc.

    As you pointed out, those are the relevant points where the full report needs to be released to determine if those are just unspecific concerns (someone pointed out that differences in cultures could be at the source of those concerns) or if some specific problems/shortcomings have been identified that need to be addressed.

  • Bill Clawson

    You’re right, but I’m wondering if there was a better way to mount the tank that had more redundancy.

  • Hug Doug

    I didn’t yell at you, lol. Defensive much?

    Probably you should have reported on the interesting things rather than latching on to your own interpretation of a small section of the report.

  • Hug Doug

    Figuring that out would require detailed engineering schematics for the tank wall attach points, the helium tank, and the struts, and we definitely aren’t going to get any of that. My guess is the way they did it is the lightest way possible to attach the helium tank to the tank wall, and for safety they overdesigned the strut. Multiplying struts means multiplying weight and there are, according to Elon Musk, “hundreds” of these struts on the rocket already.

  • windbourne

    I think that you missed my point.

    I am saying that OIG has an axe to grind with new private space. They have always kept quiet on the shuttle, ULA, Boeing, Ball, Atlas V, Delta IV, etc. yet, scream murder about new space, esp. SpaceX. TO be fair, it is probably helpful to spacex, and foolish not to be that harsh on old space as well.

    Not sure which group it is that is directly helping SpaceX, ATK, etc with COTS/CCDev/etc. but that group has given invaluable help to them. It is for that reason that I maintain that NASA is so very valuable, even though it is soo large and soo top heavy, that groups are fighting each other.

  • OldCodger

    Understand the gut reaction to this but no it is not always the case. A specialist in that particular field is usually better and cheaper than investing the resources in to an area that is not your specialty, and cos it isn’t your thing usually you don’t do as good a job.
    There are always exceptions but here it looks like either bad supplier selection (as I said a bad procurement failure) or the contracted firm outsourced (possibly against the contract terms) to an inadequate supplier and passed off non specification parts. Which means either SpaceX procurement get fired or some one gets their arses sued!

  • Douglas Messier

    I don’t think OIG has a axe to grind. The full report both praised and criticized NASA. Some of the recommendations NASA has greed to implement. Others it disagreed with. That’s not unusual.

  • Douglas Messier

    Footnotes are really interesting. You find things like this:

    The SpaceX accident board included 11 SpaceX employees and a lone FAA representative.

    “In comparison, the seven-member contractor board that investigated the Orb-3 failure included four Orbital employees, two NASA employees, and one third-party expert.”

    Then there’s this:

    “Only the 11 SpaceX board members signed the final accident investigation report.”

  • Siderite

    While I am not a SpaceX fanboy, I am certainly strongly opposed to bureaucratic and political games. While I would welcome some sort of safe way to double check the company’s findings, I can not think of a way that would not devolve into finger pointing and chest beating and committee forming. I am thus forced to believe that a company that has just started a multi billion business would be the most motivated to find the true reason for a technical failure and a solution on how to avoid it.

  • Kay Cee

    Really, the first question that comes to my mind is; Why are accidents of space-going transportation craft not subject to a National Transportation and Safety Board investigation, the same as every other machine involved in transport? Privatization of space travel is no different than privatization of any other travel industry, the rest of which are investigated by a board not answerable to anyone. You can’t even use NTSB reports in lawsuits, that’s how separate they keep themselves. Their reports are excruciatingly thorough, as well as publicly available, right down to the video of the final hearing these days. NTSB, after all, is the group that makes recommendations to the FAA, not the other way around. Compare what’s been released on this accident so far to the NTSB’s report on the SpaceShipTwo accident. Night and day as far as completeness and speed go.

  • Kay Cee

    BTW Siderite, I liked your comment as a whole. But unfortunately history tends to indicate the business itself is the least motivated to implement a solution, even when it bothers to find the root cause. Significant numbers have instead reasoned that paying damages or fines is cheaper than re-engineering. Heaven forbid anything should cut into profits.