Out of the blue and into the black
They give you this, but you pay for that
And once you’re gone, you can never come back
When you’re out of the blue and into the black.
My My, Hey Hey (Out of the Blue)
In his book, “Mastery,” George Leonard provides a fascinating explanation of how people master new skills.
“There’s really no way around it. Learning any new skill involves relatively brief spurts of progress, each of which is followed by a slight decline to a plateau somewhat higher in most cases than that which preceded it,” Leonard writes. “The curve above is not necessarily idealized. In the actual learning experience, progress is less regular; the upward spurts vary; the plateaus have their own dips and rises along the way. But the general progression is almost always the same.”
Leonard recounts his own mastery of martial arts. One day he and three of the other top brown belts in his class discussed the prospect of moving up to the higher level of black belt. At that point, the “worm of ambition” had burrowed into them.
“Maybe it was coincidence, but within three weeks of that conversation all four of us suffered serious injuries — a broken toe, torn ligaments in the elbow, a dislocated shoulder (mine), and an arm broken in three places,” he wrote. “These injuries were effective teachers. After recovering, we settled back into steady, goal-less practice. Another year and a half was to pass before the four of us made black belt.”
What is true for individuals can also be applied to organizations. Although companies do not have the option of “goal-less practice” in the face of competition, they do experience plateaus and growth spurts as they learn how to produce their products, interact with customers, and operate effectively.
Unless, of course, the company is run by someone obsessed with big leaps forward and impatient with plateaus. In which case, you’d be working for SpaceX and Elon Musk.
Hardware as Software
A space launch involves accelerating an extremely valuable satellite from zero to 17,500 miles per hour atop a rocket powered by a series of controlled explosions. The line between success and failure is exceedingly thin during the 10 minutes or so it takes to reach orbit.
As a result, the industry has been traditionally conservative. You develop a launch vehicle, put it through a flight test program, correct any problems that crop up, and fly it many times to understand how the complex machine performs. Changes and major upgrades are done very carefully. Hitting a plateau where you’re launching reliably and regularly, year after year, without any bad days is a good thing.
SpaceX has stood this approach on its head. Coming from Silicon Valley where software is king, Musk has led SpaceX down a path of major upgrades to the company’s Falcon 9 launch vehicle at a pace much faster than is normally seen in the industry.
The table below the Falcon 9’s major variants since the first launch in June 2010.
|SpaceX Falcon 9 Variants|
|Variant||Operating Years||No. of Launch Vehicles||Successes/Failures/ Partial Failures||Notes|
|Falcon v1.0||2010 – 2013||5||4-0-1||Used for Dragon supply ship missions; no payload shroud or geosynchronous satellites launched; Merlin 1-C engines|
|Falcon v1.1||2013 – 2015||8||7-1-0||Landing legs and fins for 1st stage recovery; 60 percent more thrust than the v1.0; upgraded Merlin 1-D engines; re-arranged 1st stage engines; stretched fuel tanks; recoverable first stage with landing legs and fins; upgraded avionics & software; payload shroud; geosynchronous orbit capability|
|Falcon v1.1 (Expendable)||2014 – 2016||7||7-0-0||Expendable version of Falcon v1.1 without first stage recovery|
|Falcon v1.2 (Full Thrust)||2015 – Present||9||8-1-0||30 percent greater performance than v1.1; densified super cold propellants; recoverable 1st stage|
Each of the Falcon 9 upgrades involved significant increases in the booster’s capabilities. The closest thing to a plateau the booster has enjoyed involved the 15 launches of the Falcon 9 v1.1 (expendable and recoverable) between 2013 and 2016. However, even that is a bit of an illusion.
The upgrades were not just about improving performance in order to launch larger payloads into space. The Falcon 9 v1.1 first stages were outfitted with landing legs, fins and other components to allow for them to land back at Cape Canaveral or on an off-shore barge.
None of the Falcon 9 v1.1 rockets completed a successful landing. However, the tests did pave the way for first stage recoveries by the even more powerful Falcon 9 v1.2 (Full Thrust), which uses super-cold, densified propellants to boost the rocket’s performance.
The Falcon 9 v1.2 made its debut four days before Christmas 2015 by successfully by orbiting 11 ORCOMM OG2 satellites. The rocket’s first stage made a spectacular nighttime landing back at Cape Canaveral. It was a historic first; no one else had achieved this feat with an orbital rocket.
In the eight months that followed, SpaceX would reel off another eight successful launches (seven using the new Falcon v1.2) and land five more boosters. By the end of August, the company had the first recovered booster installed outside its Hawthorne, Calif., headquarters. It also had signed up the first customer to fly on a recovered first stage by the end of the year.
SpaceX also had an ambitious agenda for the last four months of 2016. It would launch another 10 rockets — making 18 for the year, a new record — and finally debut its new Falcon Heavy booster, whose first stage consisted of three Falcon 9 boosters. The debut of the rocket had been delayed nearly four years.
Everything was looking great for SpaceX heading into Fall. And then came Sept. 1.
A Startling Accident
The fire and explosion that destroyed the Falcon 9 and Amos-6 satellite was startling for when it happened. This was not a failure in flight. Or during the static fire test Space conducts to test the first stage engines prior to a launch. It happened while the rocket was being fueled for the static fire.
SpaceX said in a statement that the accident originated around the upper stage liquid oxygen (LOX) tank. It is not clear whether the fire started due to a problem with the launch vehicle or in the ground support equipment. The Falcon 9 is grounded while an investigation is conducted to determine the cause of the accident.
The rarity of a satellite launch vehicle exploding during fueling had people racking their brains and scouring the Internet to find out the last time something like this happened. At least in the United States, that turned out to be more than 50 years ago when rocketry was in its infancy and accidents were much more frequent.
The lack of any modern precedents and the speed of the accident — Musk tweeted that engineers were reviewing around 3,000 channels of telemetry and video data that cover only 35-55 milliseconds — are making the investigation challenging. Musk has said it is the most difficult of the six failure investigations the company has conducted since it was founded in 2002.
Multiple Variants, Multiple Failures
A disturbing aspect of the accident is that it was the second catastrophic failure of a Falcon 9 launch vehicle in just over 14 months. It is also the third major incident during the booster’s 28-launch history. The table below describes the failures.
|SpaceX Failures & Partial Failures|
|2012||Falcon 9 v1.0||Partial||Failure of first stage engine resulted in ORBCOMM OG2 secondary payload in lower than planned orbit; unable to boost spacecraft due to mission rules designed to protect International Space Station (ISS); primary mission to send Dragon to space station succeeded|
|2015||Falcon 9 v1.1||Complete||In-flight failure resulting from over pressurization of second stage; Dragon supply ship lost|
|2016||Falcon 9 v1.2 (Full Thrust)||Complete||Fire and explosion on launch pad prior to a planned static fire; Falcon 9 & Amos-6 satellite, launch pad damaged|
SpaceX has experienced failures with each variant of its booster. On the fourth flight of Falcon v1.0, the fuel dome of one of the first stage engines ruptured. Burning fuel escaped, causing the destruction of the engine fairing before the motor could be shut down. A subsequent investigation pinpointed a material flaw in the engine chamber jacket as the most probable cause.
The Falcon 9’s other eight first-stage engines fired longer than scheduled, delivering a Dragon supply ship very close to its planned orbit. However, mission rules designed to protect the space station prevented an additional burn of the second stage to deliver an ORBCOMM OG2 satellite into a higher orbit. The secondary payload re-entered the atmosphere two days after launch.
At the end of June 2015, a Falcon 9 v1.1 broke up in flight after the second stage over pressurized. A Dragon supply ship headed to the space station with $118 million worth of cargo was lost. Under the commercial cargo agreement with SpaceX, NASA was left with the cost of replacing the cargo.
A SpaceX-led investigation blamed an unnamed supplier for providing a defective part. A NASA Office of Inspector General (IG) report stated that the investigation found the
most probable cause for the mishap was a strut assembly failure in the rocket’s second stage. Specifically, the failed strut assembly released a helium tank inside the liquid oxygen tank, causing a breach in the oxygen tank’s dome and the release of gas that in turn disabled the avionics and caused release of the Dragon 1 capsule and break-up of the launch vehicle….
The company’s post-mishap testing of strut parts from the same purchase order as those used on SPX-7 found material flaws due to casting defects, ‘out of specification’ materials, and improper heat treatment.
The SpaceX investigation board consisted of 11 company employees, including the chairman, and a lone FAA official. The SpaceX employees signed the final report but the FAA official did not, according to the IG report.
A separate investigation by NASA’s Launch Services Program (LSP) did not find a “probable cause” for the accident. It concluded there were several “credible causes”, including poor quality control and practices at Musk’s rocket company.
In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure. LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors….
In February 2016, the NASA Administrator and the Associate Administrator for the Human Exploration and Operations Mission Directorate sent a letter to SpaceX expressing concerns about the company’s systems engineering and management practices, hardware installation and repair methods, and telemetry systems based on LSP’s review of the failure.
The IG reported that SpaceX has taken a number of corrective actions to address concerns about the strut and its practices.
The company also reviewed the certifications of all spaceflight hardware and altered its quality control processes to better align with NASA technical standards. In order to track completion of its corrective actions, SpaceX is updating its process for identifying and resolving work-related tasks, which allows for improved auditing, prioritizing, and tracking of fracturable hardware.
To administer its updated quality control process, SpaceX has reorganized into three teams called “Design Reliability,” “Build Reliability,” and “Flight Reliability.” Besides monitoring corrective actions taken as a result of the SPX-7 failure, these teams are tracking the significant upgrades SpaceX has made to the Falcon 9 launch system for future launches, including increased thrust capability with a new fuel mixture and corrective actions on software implementation plans, which are both rated as low risks by the ISS Program.
A Double Edged Sword
Following the fueling accident on Sept. 1, SpaceX sought to reassure everyone that it wasn’t going anywhere despite this latest failure.
“We deeply regret the loss of AMOS-6, and safely and reliably returning to flight to meet the demands of our customers is our chief priority,” the company said in a statement. “SpaceX’s business is robust, with approximately 70 missions on our manifest worth over $10 billion. In the aftermath of yesterday’s events, we are grateful for the continued support and unwavering confidence that our commercial customers as well as NASA and the United States Air Force have placed in us.”
Two of SpaceX’s biggest customers, NASA and Iridium, released statements of support after the accident. However comforting that support might be, SpaceX faces serious challenges in finding the cause of the accident and working down its substantial manifest once the company returns to flight. Customers are showing patience now, but they likely won’t if failures continue after flights resume.
Between the constant upgrades to the booster and stand downs after various incidents, SpaceX hasn’t launched all that often. SpaceX has flown the Falcon 9 a total of 28 times since its debut in June 2010. That is an average of fewer than five launches per year.
The table below shows the distribution of launch vehicles over the past six years.
|Year||No. of Launch Vehicles||Successes||Failures||Partial Failures|
* Includes the Falcon 9 that blew up on launch pad.
Allowances must be granted for ramping up launches of the Falcon 9 and Dragon cargo ships during the initial years. However, it’s also clear that SpaceX has failed to significantly increase the number of launches over the past three years despite repeated public promises to do so. Each time the company attempted to increase the launch cadence, it ran into problems.
After flying only seven times from 2010 to 2013, it appeared as if SpaceX would achieve its long awaited breakthrough in 2014. The company planned 11 launches that year.
However, engineers ran into helium leaks after deciding to bring the production of the bottles in house instead of using its regular supplier. SpaceX launches six times in 2014, double the number from the previous year but well short of its goal.
The year 2015 started out well, with five successful launches through April. Then came the catastrophic in-flight failure on June 28. SpaceX was grounded for six months. The company returned to flight in December 2015, reeling off nine successful launches in a row before the fire and explosion at Cape Canaveral earlier this month.
SpaceX’s attempt to launch 18 rockets this year — tripling the number of successful flights from each of the previous two years combined — was a sign of how far behind the company is on its manifest. Whether the pressure of the increased launch rate contributed to the catastrophic explosion on Sept. 1 is unknown.
One thing is clear: SpaceX has yet to reach a plateau — that sweet spot — where launch vehicle reliability and launch cadence merge to allow the company to confidently send payloads aloft for years at a time without fail.
For customers watching their launch dates once again slide to the right, SpaceX’s problems are worrisome. Another failure any time soon after return to flight would shake their confidence in the company. The last thing a launch provider wants is the reputation of not being able to launch regularly and reliably.
That wouldn’t be a major problem if it were easy to move satellite to other launch vehicles; however, the supply of boosters is limited, and they are usually booked years in advance. Other companies also charge a lot more than SpaceX’s bargain basement prices.
Arianespace is considering adding one additional Ariane 5 rocket to its 2017 schedule due to problems suffered by Falcon 9 and its Russian competitor, Proton. Launch providers generally don’t have the ability to quickly add a capacity when competitors suffer setbacks.
The situation for NASA is even more serious. The space agency is depending on SpaceX to launch astronauts to the space station under the Commercial Crew Program. The plan is to use the reusable Falcon 9 v1.2 (Full Thrust), which uses densified fuels that would be loaded after the crew is aboard the Dragon spacecraft.
NASA officials were not comfortable with either the densified fuels or the late loading prior to the accident. It remains to be seen if the plan changes as a result of the booster’s destruction, which occurred during fueling.
They Give You This, But You Pay for That…
On the same day the Falcon 9 exploded on the launch pad, the NASA IG released a decidedly discouraging report about the space agency’s Commercial Crew Program. SpaceX and Boeing are being paid billions of dollars to develop separate vehicles to transport U.S. astronauts to the space station.
The audit found that the program — already delayed by years of Congressional under funding — is likely to suffer further slips due to technical challenges faced by Boeing and SpaceX and bureaucratic sluggishness at NASA. Instead of 2017, the IG expects the first commercial flights to take place at the end of 2018.
For SpaceX, the problems do not seem to be solely technical. In an op-ed piece at Ars Technica, Eric Berger writes that the Crew Dragon program seems to be suffering from the company’s tendency to focus on many projects at once.
One person I spoke to recently who is intimately familiar with NASA’s commercial crew dealings with SpaceX and Boeing said both companies face major technical challenges. And while this source wasn’t particularly complimentary of Boeing, noting its interest in maximizing revenue from NASA, that company at least had dedicated a team of engineers to the project. When this person meets with SpaceX engineers, however, the team members are invariably working on several different projects in addition to commercial crew. “If we could only get them to focus,” this source told me.
This would seem to make little sense given Musk’s dream of sending people to Mars. If you’re going to do that, sending them to the International Space Station first would seem to provide invaluable experience. And NASA’s paying the vast amount of the cost for the company’s Commercial Crew work.
Even as progress on Crew Dragon lagged, Musk announced plans early this year to send a series of unmanned Red Dragon spacecraft every two years to land on Mars beginning in 2018. The series is to culminate with a mission by two astronauts to the Red Planet in the mid-2020’s.
NASA has pledged the spend about $30 million to help support the first automated mission in 2018 in return for entry, descent and landing data. However, the cost of the mission — estimated at about $300 million — is being footed by SpaceX and not the agency.
And that raises a major issue. Nobody is paying Musk to send people to Mars. Or to make the first stage of the Falcon 9 reusable. These are great programs, but one wonders why they seem to have the priority they do at SpaceX. Why not focus on the programs people are actually paying the company to do?
In two weeks, Musk is scheduled to deliver a talk at a conference in Mexico on his plans to send thousands of colonists to live on Mars. Whether he will stick to his plans is unclear. On the one hand, his talk is a big reason why people will go to Guadalajara in the first place. He will not want to disappoint his hosts.
On the other hand, the optics of Musk talking about Mars while his rocket is grounded and his company is lagging on NASA’s multi-billion dollar Commercial Crew Program will not be pretty. There will be the suggestion that he is more focused on his dream of being the founder of a Mars colony than on fulfilling the contracts he has signed for far less exciting but vital activities. That would not be good for Musk’s image.