A “very tiny change” to three check valves during manufacturing caused the malfunction that disabled three of four thruster pods on the Dragon spacecraft that launched earlier this month, SpaceX CEO Elon Musk said on Thursday.
Speaking at a joint post-mission press conference with NASA officials, Musk said three check valves on the oxidizer tank became stuck due to the changes. Programmers were able to write software that commanded an increase of pressure upstream of the valves, forcing them open in a spacecraft version of the Heimlich maneuver, he added.
It took four to five nerve-wrecking hours to resolve the problem. The spacecraft was drifting during part of that time, making communications difficult. The U.S. Air Force provided some powerful communications systems that allowed SpaceX to upload the software to Dragon, Musk said.
The change in the valves resulted from a tiny design change made by the supplier that SpaceX did not notice. The company ran the valves through low pressurization functionality tests, but it did not test them under the high pressurization conditions that caused them to stick, Musk said.
The valves functioned normally once they opened. Musk said the solution on future flights is to make sure the valves adhere to their original design.
NASA Administrator Charles Bolden, who was with Musk in the SpaceX control room as engineers worked through the anomaly, said it was an incredible experience to watch the company’s young, energetic team resolve the problem. He praised the teamwork between SpaceX and NASA employees.
Musk also praised the working relationship between the two groups, saying they were deeply integrated in how they operate together. He admitted to being much more nervous during the early hours of the flight than the NASA employees.