Report: NOAA Errors Led to Diminished Weather Satellites

GOES-17 satellite during processing by Astrotech. (Credit: NASA)

by Douglas Messier
Managing Editor

NOAA’s poor management of the Geostationary Operational Environmental Satellites-R (GOES-R) program has resulted in less accurate meteorological data from the GOES-16 and GOES-17 weather satellites now in orbit, according to an audit by the Commerce Department’s Office of Inspector General (IG). [Full Report]

NOAA’s failure to properly address an overheating problem discovered during ground testing in 2017 led to the degraded performance of GOES-17’s main instrument, the Advanced Baseline Imager (ABI). The GOES-16 satellite, which was already in orbit at the time, is also suffering from overheating of its ABI to a lesser degree, the report found.

The ABI instruments were designed to provide continuous imagery of the Earth from their 16 data channels with outages of less than six hours per year. Due to a blocked heat pipe, nine channels of GOES-17’s ABI will suffer outages of two to six hours per day for roughly six months of the year, the report stated.

NOAA has said it has been able to mitigate some of the degradation in the instruments through operational changes and by applying artificial intelligence. GOES-17’s ABI is returning more than 97 percent of the data it was designed to provide, the agency said.

The IG audit spotlighted a number of additional issues with NOAA’s management of the GOES-R program, including:

  • officials altered ABI contractor performance criteria that if applied could have reduced performance payments to the manufacturer by up to 75 percent;
  • magnetometers on GOES-16 and GOES-17 that help to forecast solar storms are less accurate than the three GOES spacecraft that proceeded them into orbit;
  • less accurate magnetometers are likely to be installed on the GOES-T and GOES-U spacecraft now under development;
  • NOAA was forced to move GOES-16 and GOES-17 from orbital positions U.S. weather satellites had occupied for 40 years due to interference from other spacecraft; and,
  • the agency’s plan to replace ground system servers will cost more and pose greater management risks than anticipated.

A Fix That Didn’t

The GOES-S satellite being lowered into a thermal vacuum chamber. (Credit: Lockheed Martin)

In March 2017, engineers discovered that the ABI thermal subsystem on GOES-S (later renamed GOES-17) was not operating as designed while conducting testing prior to launch. At about the same time, GOES-16’s ABI experienced a thermal anomaly in orbit.

“In an attempt to resolve the issue on GOES-S, the program modified its ABI thermal subsystem, while noting that it did not understand all aspects of the anomaly,” the report stated. “To test the modifications, the program only verified that the thermal subsystem would start at ambient temperature and pressures; it did not confirm the modification corrected the thermal anomaly that occurred during the satellite thermal vacuum test.”

Officials obtained a waiver from the Test as You Fly — Fly as You Test rule from the NASA Goddard Space Flight Center’s Safety and Mission Assurance Directorate. NASA is a partner with NOAA on GOES-R.

“It accepted the risk of not re-performing the thermal vacuum testing of the satellite and proceeded to launch,” the IG report stated. “We found this level of verification for ABI to have been insufficient given its importance to the KPP [key performance parameter] capability for the mission.”

Launched on March 1, 2018, GOES-17’s ABI overheated the following month, resulting in a loss of imagery. The thermal subsystem was not transferring heat away from ABI sufficiently due to what a joint NOAA-NASA mishap investigation board identified as a blocked heat pipe.

“NOAA’s requirement is that GOES-R series satellites provide continuous data (with outages less than 6 hours per year) from the KPP instrument,” the audit stated. “As of October 2018, NOAA predicted that all 16 ABI channels on GOES-17 will be available all day during the cold season, which occurs twice per year—roughly the months of May through July and November through January.

“However, during the warm season—roughly the months of August through October and February through April—only seven channels will be available all day, with the other nine channels having outages of 2–6 hours per day,” the report added.

“Similarly, but to a lesser extent, GOES-16’s ABI is also not optimally managing heat transfer while on orbit. In both cases, the thermal subsystems are not operating as designed,” the audit found.

In a press release issued last month, NOAA said it has been able to resolve some of the ABI’s performance issues.

“Engineers were able to mitigate the issue through operational changes to the instrument and mission operations, including the use of Artificial Intelligence techniques, to restore most of the ABI’s functionality,” NOAA said in a press release. “The GOES-17 ABI is now projected to deliver more than 97 percent of the data it was designed to provide.

“Today, GOES-17 is providing faster, more accurate, and more detailed observations used by NOAA National Weather Service forecasters to predict Pacific storm systems, severe storms, fog, wildfires and other environmental dangers. Also, GOES-17 is monitoring tropical cyclones in the eastern Pacific Ocean, including Hawaii,” the statement said.

The press release was accompanied by a public summary of the NOAA-NASA mishap investigation board’s report. The summary includes a long list of recommendations for how to improve the development of future weather satellites.

However, the document makes no specific mention of NOAA’s discovery of and failure to properly resolve the problems with GOES-17’s ABI instrument before launch. Nor is there any mention of NASA’s waiver that allowed the instrument to fly without additional tests.

Changing Contractor Performance Criteria

The audit found that program “revised the ABI contract’s performance evaluation plan (PEP) and removed evaluation criteria that pertained to the instrument’s control of temperature and ability to calibrate” following discovery of problems on the GOES-16 and GOES-17 satellites. Harris Corporation Space and Intelligence Systems built the instrument.

Credit: Department of Commerce

“The specific evaluation criteria removed from the PEP would have classified instrument performance as either ‘degraded,’ meriting a reduction of up to 40 percent of award fees or ‘severely degraded,’ which merits up to 75 percent reduction,” the audit found.

“A NASA memorandum indicated that one of the purposes of the revisions was to ‘clarify the evaluation criteria for on-orbit performance to align with the expected performance inputs from the funding agency,’ (that is, NOAA),” the document stated.

“The GOES-R System Program Director told us that the program removed the criteria because it was considered a subset of other criteria related to the instruments’ radiometric performance,” the report stated.

The review found that the problems with the instruments would “likely” have reduced the contract award without the revisions. The IG noted it was not able to fully explore the issue, and recommended senior management investigate the changes further.

Less Accurate Magnetometers

GOES magnetometer boom with innboard and outboard Instrument (Credit: NOAA/Commerce Department OIG)

The IG audit found that the magnetometers on the two GOES-16 and GOES-17 satellites are less accurate than the ones on the three previous GOES satellites that NOAA launched.

The magnetometers measure the strength and direction of the Earth’s magnetic field at the satellite’s position in space. The data help forecasters measure the buildup and release of energy from the Earth’s magnetosphere, and predict the onset of geomagnetic storms that can disrupt communications.

“The original requirement for GOES-R series’ geomagnetic field measurement was for an accuracy of 1 nanotesla19 (1 nT) over the full instrument sensing range through the end of its life,” the audit reported. “This was the same accuracy requirement specified in NOAA documents for prior GOES series missions.

“For the GOES-R series, however, the program determined the preliminary observatory design could not achieve 1 nT accuracy without adding an additional orientation sensor to the boom,” the report stated. “However, the program considered the additional sensor too risky to the boom’s operation, so it obtained NOAA approval to provide a lesser 2.3 nT initial accuracy that may degrade to 4.0 nT by end of service life.”

Following the launch of GOES-16, program engineers implemented design adaptations, better magnetic handling procedures, and test improvements to enhance magnetometer performance on the GOES-17 satellite. It is taking the same steps on the follow-on GOES-T and GOES-U spacecraft.

“GOES-17’s magnetometer has shown improvement, but it is still not meeting requirements, and the performance issues will very likely propagate to GOES-T and GOES-U,” the audit found.

Part of the problem resulted from new contractors being brought on board for the GOES-R series.

“Although the basic sensor concept had not changed significantly since GOES-I series, the prime contractor told us that no technical approaches or lessons learned from the earlier series were shared regarding the magnetometer or boom,” the audit found.

“To adjust, the program added expertise to both the contractor and flight project within the program. The program also told us that technical data from the previous GOES series magnetometer contractor was unavailable to the new prime contractor due to contract restrictions,” the report added.

Repositioning the Satellites

Credit: Department of Commerce

The audit reports that NOAA has been forced to relocate the GOES-16 and GOES-17 satellites from the orbital positions U.S. weather satellites have occupied since 1977 because nearby satellites have caused radio frequency interference and conjunction issues.

Satellites launched by the Department of Defense, Brazil and a U.S. communications company were identified as causing problems.

“The GOES-R program did not identify this potential risk and develop risk mitigation plans prior to discovering these issues,” the document stated. “Fortunately, the relocations required only minor changes to ground system software. Risk to its orbital positions, if not managed, could threaten cost and schedule baselines in the future.”

The changes were minor because the two satellites were moved within the plus or minus 0.5° orbital parameters in the ground system’s software code.

Credit: Department of Commerce

“Program personnel stated that if the satellites were to move outside of these parameters, it would require costly software modifications,” the report stated. “How costly and how long those modifications would take is unclear, however, given that the program has not formally documented potential impacts of this risk, something it would likely do under its formal risk management processes.”

NOAA officials accepted the report’s recommended that they “ensure that the GOES-R program formally manages risk to geostationary orbital positions for both current and future satellite programs.”

Ground System Replacement Costs Rise

The audit found that NOAA’s four-year plan to replace its aging ground systems will cost more than anticipated due to changing technical requirements, budget shortfalls, and a law restricting the use of Chinese computer technology by U.S. government agencies.

NOAA’s original upgrade of its ground systems, which was to have been completed last year, were disrupted by a couple of developments beyond its control.

“In January 2014, Congress enacted a law that restricts the use of computer hardware connected to Chinese businesses,” the audit stated. “In September 2014, IBM sold its server business to Lenovo, a Chinese-owned company. As a result, the program must replace all IBM servers in its ground system.

“Original program plans (i.e., before the law was enacted) envisioned a technical refresh of ground system hardware in 2017–2018 with a piecemeal replacement of the IBM servers with newer IBM equipment, which is now not possible,” the report stated.

“The ground system will need to meet new security standards and fulfill requirements to implement other functionality that simplifies the physical architecture,” the audit found. “However, because the complexity of the server replacement has increased, it will cost more than what is included under the current life-cycle cost baseline for the program.”

Meanwhile, NOAA’s life-cycle cost estimate for the replacement hasn’t been updated since 2012.

Credit: Department of Commerce

In August 2018, the agency requested the Commerce Department’s Office of Acquisition Management (OAM) conduct an independent cost estimate of the program. OAM is scheduled to submit its report this fall.

Recommendations

The IG made 11 recommendations for improvements to the weather satellite program (see below). NOAA’s management agreed to implement all of them.

What We Recommend

We recommend that the Assistant Administrator for Satellite and Information Services

  1. Ensure the GOES-R program addresses anomalies on instruments fulfilling essential mission requirements using a distinct process that is proportional to the criticality of a key performance parameter.
  2. Ensure an independent review of changes to the ABI contract’s PEP [performance evaluation plan] occurs to determine their rationale, appropriateness, and need for further actions.
  3. Ensure the GOES-R program updates reliability analyses for ABI, the satellite, and constellation, specifically given the unique conditions of the hardware on GOES-16 and GOES-17 and any design changes for GOES-T and GOES-U.
  4. Ensure the GOES-R program documents its magnetometer design, integration, and on-orbit experience so that it is available to future GOES programs and contractors.

We recommend that the Deputy Under Secretary for Operations

  1. Ensure that NOAA conducts analysis to determine distinct geomagnetic field measurement accuracy threshold and objective requirement specifications and ensure appropriately supported requirements are reflected in GOES-R program documents.
  2. Ensure the NOAA Space Weather Prediction Center updates its geomagnetic field observation accuracy requirement validation documentation.
  3. Ensure NOAA assesses whether GOES are the optimal satellites to achieve geomagnetic field observation requirements, using an analysis of alternatives or similar cost-benefit approach.

We recommend that the Assistant Administrator for Satellite and Information Services

  1. Ensure that the GOES-R program formally manages risk to geostationary orbital positions for both current and future satellite programs.
  2. Ensure the GOES-R program updates its LCCE [life-cycle cost estimate] incorporating results from Department’s independent assessment.
  3. Ensure the GOES-R program completes a prioritized list of off-ramps with triggering dates for server replacement activities.
  4. Ensure the GOES-R program develops a plan to limit the risk of vendor lock-in for ground system sustainment.