NASA: Software Errors Almost Led to Loss of Starliner Twice

Starliner OFT-1 capsule after landing at White Sands Missile Range. (Credit: NASA/Bill Ingalls)

WASHINGTON (NASA PR) — Following the anomaly that occurred during the December Boeing Starliner Orbital Fight Test (OFT), NASA and Boeing formed a joint investigation team tasked with examining the primary issues, which occurred during that test. Those issues included three specific concerns revealed during flight:

  1. An error with the Mission Elapsed Timer (MET), which incorrectly polled time from the Atlas V booster nearly 11 hours prior to launch.
  2. A software issue within the Service Module (SM) Disposal Sequence, which incorrectly translated the SM disposal sequence into the SM Integrated Propulsion Controller (IPC).
  3. An Intermittent Space-to-Ground (S/G) forward link issue, which impeded the Flight Control team’s ability to command and control the vehicle.

The joint investigation team convened in early January and has now identified the direct causes and preliminary corrective actions for the first two anomalies. The intermittent communications issues still are under investigation. NASA reviewed these results on Friday, Jan. 31 along with multiple suggested corrective actions recommended by the team. While NASA was satisfied that the team had properly identified the technical root cause of the two anomalies, they requested the team to perform a more in-depth analysis as to why the anomalies occurred, including an analysis of whether the issues were indicative of weak internal software processes or failure in applying those processes. The team is in the process of performing this additional analysis, as well as continuing the investigation of the intermittent communications issues. NASA briefed the Aerospace Safety Advisory Panel on the status of the investigation this week.

Regarding the first two anomalies, the team found the two critical software defects were not detected ahead of flight despite multiple safeguards.  Ground intervention prevented loss of vehicle in both cases. Breakdowns in the design and code phase inserted the original defects. Additionally, breakdowns in the test and verification phase failed to identify the defects preflight despite their detectability. While both errors could have led to risk of spacecraft loss, the actions of the NASA-Boeing team were able to correct the issues and return the Starliner spacecraft safely to Earth.

There was no simple cause of the two software defects making it into flight. Software defects, particularly in complex spacecraft code, are not unexpected. However, there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects. Due to these breakdowns found in design, code and test of the software, they will require systemic corrective actions. The team has already identified a robust set of 11 top-priority corrective actions. More will be identified after the team completes its additional work.

The joint team made excellent progress for this stage of the investigation. However, it’s still too early for us to definitively share the root causes and full set of corrective actions needed for the Starliner system. We do expect to have those results at the end of February, as was our initial plan. We want to make sure we have a comprehensive understanding of what happened so that we can fully explain the root causes and better assess future work that will be needed. Most critically, we want to assure that these necessary steps are completely understood prior to determining the plan for future flights. Separate from the anomaly investigation, NASA also is still reviewing the data collected during the flight test to help determine that future plan. NASA expects a decision on this review to be complete in the next several weeks.

NASA and Boeing are committed to openly sharing the information related to the mission with the public. Thus, NASA will be holding a media teleconference at 3:30 p.m. EST Friday, Feb. 7.

In addition to these reviews, NASA is planning to perform an Organizational Safety Assessment of Boeing’s work related to the Commercial Crew Program. The comprehensive safety review will include individual employee interviews with a sampling from a cross section of personnel, including senior managers, mid-level management and supervision, and engineers and technicians at multiple sites. The review would be added to the company’s Commercial Crew Transportation Capability contract. NASA previously completed a more limited review of the company. The goal of the Organizational Safety Assessment will be to examine the workplace culture with the commercial crew provider ahead of a mission with astronauts.

Boeing’s Orbital Flight test launched on Friday, Dec. 20, on United Launch Alliance Atlas V rocket from Space Launch Complex 41 at Cape Canaveral Air Force Station in Florida. The mission successfully landed two days later on Sunday, Dec. 22, completing an abbreviated test that performed several mission objectives before returning to Earth as the first orbital land touchdown of a human-rated capsule in U.S. history.