NASA’s investigation into the Falcon 9 launch failure that destroyed a Dragon cargo ship in June 2015 keeps getting more and more interesting.
I checked in again last week with the space agency about when it would be releasing a public report on the 18-month old accident. This is what a NASA spokesperson told me (emphasis mine):
NASA’s final report on the SpaceX CRS-7 mishap is still in work. While the report is important in providing NASA historical data of the mishap, the accident involved a version of the Falcon 9 rocket that is no longer in use. Furthermore, while the public summary itself may only be a few pages, the complete report is expected to exceed several hundred pages of highly detailed and technical information restricted by U.S. International Traffic in Arms Regulations and company-sensitive proprietary information. As a result, NASA anticipates its internal report and public summary will be finalized in the summer 2017.
That is a rather long time, even for a sometimes pokey government agency investigating the failure of a booster variant no longer in use.
It’s an especially long period given what SpaceX’s separate investigation concluded was the cause of the accident. According to a NASA Office of Inspector General (IG) report, SpaceX’s accident investigation into Falcon 9’s in-flight failure found the
most probable cause for the mishap was a strut assembly failure in the rocket’s second stage. Specifically, the failed strut assembly released a helium tank inside the liquid oxygen [LOX] tank, causing a breach in the oxygen tank’s dome and the release of gas that in turn disabled the avionics and caused release of the Dragon 1 capsule and break-up of the launch vehicle.
SpaceX completed an extensive analysis of the SPX-7 failure, consulted with NASA and the United States Air Force (USAF) regarding their analysis, and provided a mishap report and Return to Flight Plan to the FAA and NASA in November 2015. The company’s post-mishap testing of strut parts from the same purchase order as those used on SPX-7 found material flaws due to casting defects, “out of specification” materials, and improper heat treatment.
The solution sounded simple enough. Switch strut suppliers and test all the struts when they come in to weed out defective ones rather than relying on the contractor’s guarantee. At least that’s how SpaceX and founder Elon Musk described the fix, anyway.
SpaceX completed the investigation into the failure and was back flying the Falcon 9 only six months after the crash on Dec. 21, 2015. Three days earlier, NASA’s Launch Services Program (LSP) had briefed the space agency’s senior management on the results of its own separate investigation into the failure.
So, everything appears to have been largely wrapped up before Christmas 2015. So, why would it take another 18 months for NASA to complete its investigation and issue a public summary?
Yes, the version of the Falcon 9 that failed is no longer being flown, so perhaps there is no urgency. But, that was also true of the Antares booster that exploded just after launch in October 2014. Orbital ATK re-engineered the first stage with new engines after that accident. But, it only took a year for NASA to release a report on that accident.
The really intriguing question is: how could NASA’s investigation into the failure of a single defective strut produce a report with “hundred pages of highly detailed and technical information” restricted by export regulations?
That seems like overkill, doesn’t it? Unless, of course, the failure was more complicated than that.
No Probable Cause Found
The NASA investigation delved deeper into SpaceX’s operations and reached a different set of conclusions. According to the NASA IG report [emphasis mine]
LSP did not identify a single probable cause for the launch failure, instead listing several “credible causes.” In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure. LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors….
In February 2016, the NASA Administrator and the Associate Administrator for the Human Exploration and Operations Mission Directorate sent a letter to SpaceX expressing concerns about the company’s systems engineering and management practices, hardware installation and repair methods, and telemetry systems based on LSP’s review of the failure…
SpaceX has taken action to correct the deficiencies that led to the failed strut assembly and to address NASA’s concerns by conducting inspections, replacing suspect parts, and conducting additional testing. The company also reviewed the certifications of all spaceflight hardware and altered its quality control processes to better align with NASA technical standards. In order to track completion of its corrective actions, SpaceX is updating its process for identifying and resolving work-related tasks, which allows for improved auditing, prioritizing, and tracking of fracturable hardware.
The first paragraph of the above excerpt is almost the entire description of what NASA’s investigation found in terms of credible causes. The IG’s 54-page report was focused on how the space agency responded to the accident, not on what caused it.
As a result, the information raises more questions than it answers. For example, it’s not clear whether “improper material selection” refers to the struts or other parts of the booster around it.
Until NASA releases more definitive information next summer, we’ll have to rely largely on SpaceX’s finding that a single defective strut caused the accident. That conclusion was reached by an investigation board stacked with 11 SpaceX employee and a single FAA representative. The FAA official was the only board member not to sign the final report.
“We acknowledge SpaceX’s investigation was transparent and the observers from FAA, ISS, LSP, NTSB, and USAF had access to the investigation’s data and analysis,” the NASA IG report notes. “However, an investigation led by the employee responsible for the SPX-7 launch and run by the contractor responsible for the failure raises questions about inherent conflicts of interest.”
A Second Accident Raises More Questions
The question NASA’s investigation raises is whether a defective strut was the “root cause” of the failure. Did the strut alone cause the helium tank to break free? Or did something else cause the strut to fail other than the normal stresses of launch?
Stories have circulated within the space community for more than a year that it wasn’t the strut alone that caused the accident. The speculation was heightened after a Falcon 9 caught fire and exploded on the launch pad on Sept. 1 while it was being fueled for a pre-launch engine test.
The location of the fire and explosion this time? The second stage LOX tank again. And the problem? A large breach of a cryogenic helium tank inside of it.
Musk said the investigation pointed to the formation of solid oxygen during the loading of the helium tanks, which are covered with carbon composite materials. The oxygen then ignited the carbon composites, starting a fire that led to the explosion of the second stage.
The company has said it has reproduced the accident in tests “entirely through helium loading conditions. These conditions are mainly affected by the temperature and pressure of the helium being loaded.”
SpaceX hopes to return to flight next month using modified fueling procedures once the FAA grants the company a launch license. That decision will depend upon the results of the on-going accident investigation.
But, is simply changing the fueling process enough? Following the launch pad failure, SpaceNews consulted several experts who raised questions about the cause of the June 2015 accident. (Emphasis mine)
“The question is whether the strut was really the origin of the June 2015 failure. The struts were of below-specified quality, but it was not demonstrated that this was the root cause of the failure.
“It remains possible that a helium bottle burst in June 2015 and that, in parallel, the struts supporting it were of poor quality….
“But there is always the risk of correcting things that are not the real cause, such as correcting a badly built strut, which might not have prevented the June 2015 failure…”
If this theory is correct, then SpaceX has experienced two separate breaches of helium tanks in Falcon 9 second stages under very different conditions — in-flight and during fueling. It raises the possibility the booster has a basic design flaw that will not be fully addressed by procedural changes.
SpaceX denies any commonality between the two accidents. However, SpaceNews’ sources said that merely meant that the Sept. 1 failure was not caused by a broken strut but a tank breach.
Questions Swirl Around Composite Tanks
Despite SpaceX’s assurances, questions remain about the composition, location and durability of the helium tanks used in the Falcon 9.
SpaceX uses carbon over-wrapped pressure vessels (COPVs), which consist of a thin liner of aluminum covered with carbon composite fiber. The helium vessels are lighter and stronger than tanks made from aluminum or other metals, giving the Falcon 9 greater payload capacity.
The COPVs are placed inside the LOX tank, where they are soaked in extremely cold liquid oxygen. This arrangement is different from the ones used on many other rockets, where helium tanks are placed outside of the LOX tank.
Beginning last year, SpaceX began to densify Falcon 9’s propellants, a process under which they are chilled to an even lower temperature than they were for previous flights. The process reduces the volume of the LOX, allowing SpaceX to place more of it in the tank. The change helps to counteract the performance hit the Falcon 9 takes when the booster’s first stage is recovered.
The problem with submerging COPV’s in densified LOX is they can become brittle and develop cracks when exposed to the types of extremely low temperatures. (Emphasis mine)
Despite years of testing, concerns still exist about the potential for leaks, due to microcracking of traditional carbon/epoxy composite laminates at cryogenic temperatures. Microcracks can occur in any laminate because of the difference between the axial and transverse coefficients of thermal expansion (CTE) in each ply, explains Brian Wilson, president of Wilson Composite Technologies (Folsom, Calif.). “Transverse microcracks in the resin can be generated between the fibers as the laminate cools after cure, and as temperature is lowered to cryogenic levels.”
Even at moderate pressure, the composite’s exposure to temperature extremes and repeated fill-and-drain cycles causes thermo-mechanical loading, which exacerbates cracking and leads to permeation leak paths, easily traversed by small hydrogen and oxygen molecules. The issue isn’t helped by the fact that most thermoset systems lose strain capacity and become brittle at cryogenic temperatures.
Designing for containment of LOX poses particularly difficult problems. Any permeation leak poses a potent ignition-and-explosion hazard. There is unresolved debate in the industry as to whether LOX, an aggressive oxidizer, actually reacts with and degrades composite materials — some, including Wilson, say that a cured composite is not affected. Composites also have a tendency to pop, flash or char if impacted mechanically while in contact with LOX, because the mechanical energy of the blow causes a rise in heat within the material, which can, in turn, cause auto-ignition.
So, COPV’s are light and strong but vulnerable to failure, particularly when in contact with LOX. Thus, strict quality control measures must be observed during their production, installation and fueling to avoid. The question is whether those measures are enough to avoid additional bad days.
Commercial Crew Concerns
NASA’s International Space Station Advisory Committee, led by Apollo veteran Tom Stafford, has raised the question of whether it is a safe practice to place the helium bottles inside the LOX tank instead of outside the tank.
Joseph Cuzzupoli, a former senior NASA manager who worked on the agency’s Apollo, Gemini and space shuttle programs, also expressed misgivings about SpaceX’s plans and the lack of NASA response. “Are we in the dark on this whole thing?” he asked.
Mr. Cuzzupoli told the committee that installing helium containers within fuel tanks—which entails putting wiring, sensors and tubing inside a potentially explosive environment—is “very unusual in my world, in my experience.” Such designs, he said “have been a no-no ever since Apollo.”
In 1970, a spark from an exposed wire inside an oxygen tank caused a life-threatening fire on board Apollo 13, bound for the moon. The crew managed a safe return to Earth, but NASA changed designs to prevent a similar incident.
This is a particular concern with the Crew Dragon spacecraft that SpaceX is building under NASA contract to take astronauts to the International Space Station. It’s also an issue for Dragon cargo ships and other NASA payloads the space agency has contracted SpaceX to launch.
The densified LOX has created a further concern for the committee. In order for the oxidizer to remain cold, it needs to be loaded just before launch. That means putting the crew on board the Dragon first and then fueling the booster.
“There is a unanimous, and strong, feeling by the committee that scheduling the crew to be on board the Dragon spacecraft prior to loading oxidizer into the rocket is contrary to booster safety criteria that has been in place for over 50 years, both in this country and internationally,” Stafford said in a Dec. 9, 2015 letter to William Gerstenmaier, NASA’s associate administrator for Human Exploration and Operations.
“Historically, neither the crew nor any other personnel have ever been allowed in or near the booster during fueling,” Stafford added. “Only after the booster is fully fueled and stabilized are the few essential people allowed near it.”
The advisory board is an outside group. However, these concerns are also shared by some within the space agency. It’s not clear how NASA and SpaceX are going to resolve these issues.
The Road Forward
One solution would be to simply not use densified propellants on the Crew Dragon flights. However, it is not clear what changes SpaceX might have to make in the booster to meet NASA’s requirements, or what they might cost to implement. For example, a second assembly might be required.
Not using densified propellants wouldn’t address the concern about having the COPVs inside the LOX tank. There are two possible solutions to this worry. One would be to replace the COPVs inside the LOX tank with ones made of metal. The second would be to move the helium vessels outside the tank.
SpaceX has not shown any indication of wanting to pursue any redesign of the second stage tank. Any redesign of the system would be costly and time consuming while adding complexity to the system. Additional failure modes would be introduced that would need to be tested.
Replacing carbon composite tanks with metal ones would add weight, resulting in lower payload capacity. Leaving them inside the LOX tank could still result in failures similar to the one Apollo 13 experienced.
The company might have to ground the Falcon 9 for an extended period of time. In all likelihood, NASA and the U.S. Air Force would need to re-certify the modified booster to carry government payloads, a costly and time consuming process.
SpaceX is already significantly behind schedule on flying its launch manifest due to a low flight rate and the two booster failures. The company has around 70 missions to fly in the years ahead, Some customers have grown impatient. Last week, Inmarsat moved a satellite from the perpetually delayed Falcon Heavy booster to Ariane 5.
There’s the potential for further delays to the Crew Dragon program, which is already years behind schedule due to a combination of under funding by Congress, technical challenges and bureaucratic delays at NASA. Major changes in the Falcon 9 could delay that program even further.
There’s also the question of SpaceX’s finances. With rock bottom prices, a low launch cadence, repeated failures, a head count exceeding 5,000 employees, and programs that include Crew Dragon, massive satellite constellations and human Mars missions, it’s not clear whether SpaceX is actually profitable. In fact, the company removed the claim that it was profitable and cash-flow positive from its own website.
If there’s another launch failure any time soon, SpaceX could be in serious trouble.
Sorting It All Out
SpaceX had ambitious plans for 2016. It was going to begin catching up on its manifest by launching 18 Falcon 9 boosters — triple the number of successful flights from each of the previous two years. After many years of struggling to launch regularly, this was going to be the breakthrough year.
As it turned out, the company managed eight successful flights before its ninth booster blew up on the launch pad during fueling. It was the type of on-pad accident that hadn’t happened to an American satellite launch provider in more than 50 years.
So, as 2016 comes to a close, SpaceX finds itself with the Falcon 9 grounded as it tries to complete its investigation into the accident. The company has questions swirling around about the reliability and safety of its launch vehicle that will need to be answered before the FAA grants a launch license or NASA puts crew on board it.
Meanwhile, NASA is increasingly dependent for cargo and crew services upon a company that has experienced two launch vehicle failures over the past 18 months, one without even leaving the ground. And SpaceX is asking the space agency to overturn a half century of safety practices in order to launch NASA’s most valuable assets.
One thing is clear: 2017 is going to be a very interesting year for Space X and NASA.