It’s Crunch Time for Commercial Crew: Serious Challenges Lay Ahead

commercial_crew_cst100_dragon_iss
Boeing and SpaceX are using a “high risk strategy” to develop commercial crew vehicles to carry astronauts to the International Space Station that could result in costly delays in the program, according to a new Government Accountability Office (GAO) report.

“In addition, the program is concerned that certification could be delayed, in part, because of a lack of design maturity in the companies’ crew vehicles,” the report states. “Both companies held critical design reviews in 2015 and each had several key crew vehicle subsystems designs that were not yet mature.”

NASA’s Commercial Crew Program is on “an aggressive schedule” to certify Boeing’s Starliner and SpaceX’s Dragon crew vehicles by the end of 2017, the GAO says in its report, “NASA: Assessment of Major Projects.” The vehicles would then be used on a commercial basis to transport crews to ISS.

Dragon Version 2. (Credit: SpaceX)
Dragon Version 2. (Credit: SpaceX)

“Boeing and SpaceX have yet to complete the majority of their critical test events and there is little time between test events for the companies to learn from them and make changes,” the report reads.

“Both companies plan to complete their uncrewed and crewed demonstration missions that are intended to test key system capabilities including their ability to launch, dock with the International Space Station (ISS), and return to Earth in 2017 and there are only about 4 months between each of these demonstrations,” the GAO said.

“Boeing does not plan to complete the final lower-level component design reviews for crew life support systems until May 2016 and SpaceX does not plan to completely mature its Dragon seat designs until spring 2016,” the GAO added. “Our best practice work shows that having a stable design prior to hardware fabrication can reduce the risk of rework efforts that could result in cost increases and schedule delays.”

Artist's conception of CST-100 Starliner docking at the International Space Station. (Credit: NASA)
Artist’s conception of CST-100 Starliner docking at the International Space Station. (Credit: NASA)

NASA has secured additional seats aboard Russian Soyuz transports for 2018 to accommodate any delays in the Commercial Crew Program.\

“Moving forward, if the program receives inadequate funding to finance planned contract work, it could be required to renegotiate the contract, which may result in price increases and schedule delays,” according to the report.

The GAO report also found that:

  • Boeing requested variances for aspects of Starliner and Atlas V designs that do not meet fault tolerance requirements;
  • SpaceX has requested a variance to use commercial-off-the-shelf parts in certain applications rather than specially tested “space-rated” parts ;
  • NASA officials are concerned Starliner and Dragon could fall short of the Commercial Crew Program’s 1 in 270 loss of crew requirement.

Space agency officials are also working through issues involving modifications being made to the Atlas V and Falcon 9 launchers. United Launch Alliance is adding a second engine to its Centaur upper stage for Starliner launches. The modified launch vehicle will need to be certified for crew flights.

The concern with SpaceX’s Falcon 9 involves the use of densified fuel.

“SpaceX plans to load crew into the Dragon and then fuel the rocket to keep the densified propellants chilled,” the GAO reported. “The program has reported that loading the crew prior to the propellant is a potential safety risk. SpaceXstated that its approach will improve safety by minimizing personnel exposure to a fueled rocket.”

The relevant section of the GAO’s report is reproduced below.

cctcap_payments_gao16Project Summary

The purpose of the Commercial Crew Program is to facilitate the development of safe, reliable, and cost-effective crew transportation systems (CTS) to carry NASA astronauts and cargo to and from the International Space Station (ISS). The program is a multi-phase effort that started in 2010 to stimulate private-sector interest in providing commercial human space transportation capabilities. The current Transportation Capabilities phase is intended to result in the final certification of CTSs for crewed flights. In September 2014, NASA awarded firm-fixed-price contracts to Boeing and SpaceX for the design, development, test, and operation of CTSs; a minimum of two, but up to six crewed missions to ISS; and special studies, tests, and analysis.

Project Summary

The Commercial Crew Program is working to an aggressive schedule to certify Boeing and SpaceX CTSs by the end of 2017, at which point they could be used for crewed flights to and from ISS. To meet this schedule, Boeing and SpaceX are concurrently developing, testing, and producing their vehicles—a high risk strategy that could lead to costly modifications to systems already being built and delays if problems are identified during testing. NASA has tried to minimize its cost risk on the program by using firm-fixed-price contracts. In addition, the program is concerned that certification could be delayed, in part, because of a lack of design maturity in the companies’ crew vehicles. Both companies held critical design reviews in 2015 and each had several key crew vehicle subsystems designs that were not yet mature. The program put Boeing and SpaceX on contract in 2015 for their first post-certification missions. The companies will need to complete the certification process before these flights.

Launch_America_Commercial_Crew
Cost and Schedule Status

NASA held an agency review in October 2015 to establish cost and schedule baselines for the Commercial Crew Program, but it has not yet finalized its decisions from the review, including the amount of cost and schedule reserves it will hold for risk mitigation activities.

Schedule

The Commercial Crew Program is working to an aggressive schedule to certify Boeing and SpaceX’s CTSs by the end of 2017. To meet this schedule, Boeing and SpaceX are concurrently developing, testing, and producing their vehicles. Overlaps between these activities increase the risk that problems identified during development or testing could lead to costly modifications to systems already being built and schedule delays. NASA has tried to minimize its cost risk on the program by using firm-fixed-price contracts. The program ordered the first post-certification missions from the companies in 2015 before they completed development or tested their CTSs. These orders are made 2 to 3 years prior to actual mission dates in order to provide time for each company to manufacture and assemble the launch vehicle and spacecraft.

Boeing and SpaceX have yet to complete the majority of their critical test events and there is little time between test events for the companies to learn from them and make changes. Both companies plan to complete their uncrewed and crewed demonstration missions that are intended to test key system capabilities including their ability to launch, dock with the International Space Station (ISS), and return to Earth in 2017 and there are only about 4 months between each of these demonstrations. The companies will need to successfully complete these demonstrations and the certification process before they can fly post-certification missions. NASA extended its contract with the Russian Federal Space Agency to procure additional seats on the Soyuz vehicle through 2018 to ensure that it has access to ISS if delays occur.

Boeing Starliner water drop test (Credit: NASA/David C. Bowman)
Boeing Starliner water drop test (Credit: NASA/David C. Bowman)

Design

The Commercial Crew Program is monitoring several design-related issues that could delay final CTS certification, including the maturity of Boeing and SpaceX’s current designs and the ability of the companies to meet NASA requirements and standards. Both Boeing and SpaceX held critical design reviews in 2015, but will not complete design activities until later than planned.

Boeing held a critical design review in March 2015 for its CTS, which includes the Starliner crew vehicle, Atlas V launch vehicle, and ground systems, and a follow-on review in May 2015 that focused on the design of the launch vehicle and launch site.

SpaceX held the first part of a multi-part critical design review in October 2015, which focused on the design of its launch vehicle—an upgraded version of the Falcon 9—and uncrewed ground systems, and the second part in December 2015, which focused on the Crew Dragon capsule and mission operations. An additional critical design review is planned to be completed by August 2016 for any remaining Dragon component or subsystem designs, including an updated seat design, and the crewed ground systems.

The program is tracking risks related to the design maturity of both companies’ crew vehicles because they have several key subsystem designs that are not yet mature.

For example, Boeing does not plan to complete the final lower-level component design reviews for crew life support systems until May 2016 and SpaceX does not plan to completely mature its Dragon seat designs until spring 2016. Our best practice work shows that having a stable design prior to hardware fabrication can reduce the risk of rework efforts that could result in cost increases and schedule delays. For the parts of the CTS designs that are mature, the program is using a design change control process to assess the potential effects of proposed design changes on safety, and to approve or disapprove the companies’ proposed changes.

Dragon abort test with SuperDraco engines. (Credit: SpaceX)
Dragon abort test with SuperDraco engines. (Credit: SpaceX)

Both companies have requested variances, or permission from the program to deviate from certain NASA requirements and design standards. If the program does not approve some of these variances, it could force the companies to make design changes, which could have cost and schedule implications for the companies.

For example, Boeing requested variances for aspects of both the Starliner and Atlas V designs that do not meet fault tolerance requirements, which are requirements related to the ability of a system to continue operating should a component error or failure occur.

SpaceX has requested a variance to use commercial-off-the-shelf parts in certain applications rather than parts that have gone through special testing to be considered “space-rated.” SpaceX officials said that they have used this approach and proven that it is reliable for multiple short-duration cargo missions.

Overall, the program is taking several steps to mitigate these types of issues, including accommodating the companies’ specific ways of doing business, limiting changes to requirements that might lead to design changes, and actively engaging with the companies on requested variances.

The Commercial Crew Program is also concerned that it may fall short of meeting the program’s loss of crew requirement based on the current CTS designs. The program’s loss of crew requirement is 1 in 270, which is a measure of how likely there will be loss of crew on a given mission. This is an increase in the requirement from the end of the Space Shuttle program, which was about 1 in 90. Boeing and SpaceX are responsible for meeting a loss of crew requirement of 1 in 200, and the program is responsible for closing the gap between that requirement and the one for the program.

The program conducted assessments of each company’s designs in order to meet the overall requirement and determined that it would be challenging without additional spacecraft modifications to protect against micrometeoroid and orbital debris. The companies would need to make these design changes soon, as they are both moving into manufacturing their systems. The program has established a team to develop a plan to close the requirements gap.

Artist concept of CST-100 and Atlas V on launch pad. (Credit: Boeing)
Artist concept of CST-100 and Atlas V on launch pad. (Credit: Boeing)

Launch

The Commercial Crew Program is working to address a number of launch vehicle issues that will need to be resolved prior to certifying that the vehicles are safe to transport crew.

  • Boeing’s selected launch vehicle, the Atlas V, is being modified by adding a new emergency detection system and a second engine to its upper stage, so that it can be certified for human space flight and meet fault tolerance requirements.The Commercial Crew Program is working through the variances the launch vehicle provider has submitted for the new upper stage and plans to complete mitigation plans by April 2016 to resolve any remaining risk of certifying the modified Atlas V for human space flight.
  • SpaceX’s launch vehicle, the Falcon 9, has been upgraded to improve its performance by increasing engine thrust and using densified propellants. Among the risks associated with the upgraded vehicle is SpaceX’s planned concept of operations for launching using densified propellants. SpaceX plans to load crew into the Dragon and then fuel the rocket to keep the densified propellants chilled. The program has reported that loading the crew prior to the propellant is a potential safety risk. SpaceXstated that its approach will improve safety by minimizing personnel exposure to a fueled rocket. It has also identified safety and hazard controls to mitigate any risks associated with this approach. In December 2015, SpaceX launched the upgraded Falcon 9 for the first time and successfully landed its first stage on land.

Funding

One of the Commercial Crew Program’s top risks during 2015 was funding uncertainty, but this appears to have been alleviated with the passage of the Consolidated Appropriations Act, 2016. In the Act, the program received funding in the amount requested by NASA. According to NASA, one of the reasons for the funding uncertainty was confusion over the way NASA is financing the contracts. The design, development, test, and evaluation activities in the contract that culminate in certification are fixed-price. The program uses performance-based payments, also referred to as milestone payments, to finance Boeing and SpaceX, and they are only paid after the successful completion of a milestone. The program designated five mandatory milestones, such as the certification review, and the companies developed a set of interim milestones. Under a fixed-price contract with performance-based financing payments, the contractors’ incurred costs are irrelevant, and the milestone payments help finance the contract through development to completion. For example, the companies might use milestone payments received for completing the critical design review milestones to purchase hardware for test articles. Moving forward, if the program receives inadequate funding to finance planned contract work, it could be required to renegotiate the contract, which may result in price increases and schedule delays.

Project Office Comments

In commenting on a draft of this assessment, Commercial Crew Program officials stated that having at least two companies developing different crew transportation systems provides benefits in redundancy, innovation, and cost effectiveness. They also stated that the program was not funded at the levels requested in the President’s Budget request during fiscal years 2011-2015, which were critical years of design and development. They emphasized that adequate and timely funding and maintaining competition between the two companies are essential to ensuring program performance. Commercial Crew Program, Boeing, and SpaceX officials provided technical comments, which were incorporated as appropriate.

  • duheagle

    Yes, it is. Makes one wonder a bit that Boeing didn’t, and still doesn’t, seem to be in any great hurry to get on with doing it. I’m fairly confident SpaceX will have a deliverable spacecraft by the end of 2017. Not as confident that Boeing will.

  • Douglas Messier

    I’m not going to defend Boeing’s decisions regarding what the company put in financially to commercial crew. I think they should have put more funds into it.

    I would note, however, that as a public company, it faces a different standard than SpaceX in terms of justifying spending money on a program than SpaceX, which is privately held. And they likely have higher internal costs than SpaceX. So, getting the business case to close on a program like this with limited a production run and uncertain prospects for other customers (Bigelow might succeed, or might not) is a bigger issue for Boeing.

  • PK Sink

    “I find that scenario almost impossible to believe.”

    Terry, you are messing with my mind, and I don’t have an overabundance of it left. Have a good one, and I’m gonna start Happy Hour a little early today. 😉

  • duheagle

    They “effed up” in trusting a supplier too much. One could say the same, I suppose, about Orbital’s loss of ORB-3. At least SpaceX’s problem had a straightforward and quick fix. A return to flight in six months isn’t bad. Orbital’s problem was a lot more fundamental and is taking over three times longer to fix. SpaceX, in any event, has had no client defections from its Falcon 9 manifest. To hear some people talk, you’d think customers were scattering like roaches when the light goes on.

  • duheagle

    Heh.

  • duheagle

    Nobody at SpaceX probably asked NASA about the possibility of sticking their cranks in a meatgrinder either, because, like boarding a propellant-loaded rocket it’s just so obviously – except to NASA, it would seem – NOT A GOOD IDEA!

  • duheagle

    NASA is overseeing both Boeing and SpaceX on Commercial Crew. There are NASA staffers at Hawthorne full-time. If NASA has doubts about any part of Dragon 2 it has the leverage to require SpaceX to remove said doubt.

    As to losses, cargo vehicles don’t have a lot of the subsystems needed for crewed flight. They have minimal environmental and life support systems, for instance. They also lack abort engines with which to escape a failing booster, either on the pad or in flight. Crew vehicles will have all these things and, at least in the case of SpaceX, the having will also be buttressed by tests to make sure they work. The opinions of neither NASA nor SpaceX staffers will prevail in the absence of support from actual testing.

  • Doug Weathers

    Aaah, don’t worry about it. The Russians have been taking family pictures of people in front of fully-fueled Soyuzes for years!

  • Arthur Hamilton

    Originally, both Boeing and SpaceX had to be ready to fly the first operational U.S. crew mission in December 2017. With NASA’s purchase of additional Soyuz seats into 2018, that gives the CC providers up to a year extra to be ready by December 2018. But, the original contracted ready-to-fly date haven’t been updated to show the extra year from what I understand.

  • Douglas Messier

    I don’t know where the assumption that Boeing is not ” in any great hurry to get on with doing it.” Can you cite any actual evidence?

  • Douglas Messier

    It may have been more complicated than that. We’ll know if we ever see an actual report out of NASA about the crash. We saw one for the Antares crash.

  • Michael Vaicaitis

    Agreed….SpaceX’s argument seems to be that fuelling after boarding means that only the astronauts/passengers that have the benefit of an abort escape system are subject to the risk of a fuelled vehicle.
    This report implies that by subjecting ground crew and unseated astronauts to fuelled vehicle risk is somehow a better/safer strategy.
    I’m mystified as to how anyone would not agree that SpaceX’s plan is obviously superior. Perhaps, they’re concerned that the seats might need changing between fuelling and launch.

  • Michael Vaicaitis

    I’m thinking (that is, my personal unproven theory) that FH delays have been heavily tied to reusability development… and a lack of market urgency.

  • Michael Vaicaitis

    The only explanation for the walk back and sniff the lit firework strategy that I can think of, is that perhaps they’re concerned that the seats might need changing between fuelling and launch.

  • Bernardo_de_la_Paz

    I get that the GAO report was based on the assessments of trained and experienced professionals in the relevant fields of expertise who are directly involved in these programs, but aside from sleeping in a Holiday Inn last night, what is it that qualifies you with the expertise and knowledge to be so dismissive of their report?

  • Larry J

    I suspect that SpaceX used the downtime to look at other potential issues that were identified while examining the telemetry in great detail. If all they had to do was improve the quality control on that tank strut, they could’ve returned to flight much sooner. Instead, they probably found other things to improve and fixed them as well.

  • Larry J

    Some of the difference in contract value is the cost of the rockets. IIRC, the contract called for five flights. SpaceX will be using their Falcon 9 at a cost of $60 million or so. Boeing will use the Atlas V, apparently the 412 version. This includes 1 solid fuel strap on and two RL-10 engines in the upper stage. It’s hard to find pricing info but it’s likely to be closer to $200 million each. The boosters for five flights of the Dragon 2 will cost around $300 million while it’ll cost around a billion dollars for five flights of the CST-100.

  • Athelstane

    Whole different ballgame with crew and a much different set of risk assumptions.

    It is; but I don’t think any NewSpace company would argue otherwise. For one thing, CRS-7 would have had an LAS on it.

    I think some of us are a little skeptical of this GAO report (and the NASA staff who are the source of some of the concerns) because there *is* a real arbitrariness in defining terms like “space-rated” and “human rated,” because it begs the question of just what acceptable risk levels are, and because too often it has the effect of benefiting vested interests within the agency and its existing contractors.

  • ReSpaceAge

    What does effed mean??

  • Hug Doug

    Very true. If there’s no customer pressing for a flight, there’s little reason to build it.

    Same reason the Atlas V Heavy has never been developed for flight.

  • duheagle

    So SpaceX not only makes themselves late, they have some mysterious power to make NASA tardy too. Elon should change his name to Lamont Cranston.

  • duheagle

    Correctimundo. Musk even said as much in some tweets. The company was well along in plans to roll out the F9 Full Thrust so they took a fresh look at all of its engineering as well as that of its failed predecessor just to be as certain as possible they hadn’t missed anything else.

  • duheagle

    I think you and Michael V. are both right.

    To briefly review FH history, there were: 1) the originally announced FH, based on the expendable v1.1, 2) the later cross-feed upgrade which was probably started using the expendable v1.1 as a base, but finished after the v1.1 got legs and became, in theory, reusable, and 3) the currently planned version based on the reusable Full Thrust version but seemingly with the cross-feed option abandoned – at least for now. After the “Mark III” FH enters service, there is still the possibility SpaceX may elect to restore the cross-feed, develop a Raptor-powered upper stage, a larger FH-only payload fairing or some combination thereof. Their decisions will be based on market demand.

    As to market urgency, the early adopters of the FH were two comsat operators which had really big, heavy birds in development and which saw getting an early FH slot as a potential economic win versus the price of a much more expensive dedicated Ariane 5 launch. The FH delays in initial operation have now stretched to the point where one of these has gone with the Ariane 5 option as their bird is now ready and needs a ride sooner than the FH can probably provide a cheaper one. One other such client has booked a Proton option with ILS, but still has the FH slot, too, and will use it, preferentially, if the timing works out.

  • duheagle

    NASA staffers are certainly “trained and experienced professionals,” but they are also people. People are sometimes illogical and/or prejudiced in favor of “the way things have always been done” even if the latter makes no damned sense.

    I don’t think it matters what my credentials are, for instance, to note that NASA’s idea that the “right” way to embark crew and button them into their vehicle is to do it after tons and tons of propellant have been loaded just beneath their feet and not when the vehicle’s tanks are empty and present no explosion risk. NASA’s legacy manned vehicles all took far longer to fuel up than does the Falcon 9, especially the Full thrust version.

    I can certainly understand that astronauts preferred to spend as little time waiting on the pad before launch as possible. Of all NASA’s legacy manned vehicles, only the Shuttle had a toilet and it wasn’t usable while the vehicle was on the pad. The idea of climbing into an empty Saturn V or Shuttle, then waiting six or eight hours before launch without a bathroom break while its propellants are slowly loaded doesn’t appeal to me either. Neither does riding said vehicles to orbit at three or four G’s with an excruciatingly full bladder.

    So NASA went with loading crew after loading propellant. It is certainly understandable that NASA staff tend to regard “tried and true” NASA procedures favorably, but the ones whose opinions are reflected in this report are, I believe, simply being tribal about something that is not really logically defensible as a preferred, let alone superior way of doing things.

    I think we are all, therefore, thoroughly entitled to question the alleged wisdom of people who seem flagrantly captives to tradition rather than objective judges of merit in one obvious case when they scruple to pass “judgement” on other matters.

    Bottom line? I reject your argument from authority because I have serious questions about the quality of that authority. Too many here seem inclined to automatically bend the knee and tug the forelock where NASA is concerned. This is not your father’s or grandfather’s NASA. These are not the guys who got us to the Moon.

  • duheagle

    So far as I know, Boeing has yet to produce a test article of Starliner that is even as complete as the engineering prototype of Dragon 2 SpaceX used to conduct its pad abort test nearly a year ago.

  • duheagle

    True. And NASA has no input at all into safety practices in the Russian space program. So NASA continues to blithely risk the lives and limbs of its ISS crews by paying for rides on the rickety Soyuz, but nitpicks or spews arrant nonsense about the American firms trying to end reliance on increasingly dodgy Russian hardware. That really makes sense.

  • Douglas Messier

    And from that you determine they’re slacking? Without knowing the details of what’s involved in the Starliner program, I don’t know how you can reach that conclusion. Insufficient data.

    But, then again, you’ve been making a lot of assumptions about the Commercial Crew Program’s concerns based on a report that was intended essentially as a schedule update and not a technical document.

  • Douglas Messier

    SpaceX has run 2.5 to 3 years behind schedule on most things. It’s now trying to pare down its launch manifest backlog (18 launches this year is the goal) while simultaneously standing up its Dragon crew vehicle. That’s a big order.

    Space Flight Now had the uncrewed Dragon flight test scheduled for late this year. It’s now slipped to May on the schedule.

  • ReSpaceAge

    Nasa going up against the guys that will take us to Mars
    hmmmm?
    🙂

    Get the hell out of the way!

  • Vladislaw

    I was thinking with the back log of F9 launches, burning up three cores would only delay the backlog even more.

  • Glenn

    effed is French for F’d