The testing program behind every Zipline flight

Zipline has flown more commercial autonomous miles than any other company—over 70 million—all without a serious safety issue. 

Behind our safety record is a rigorous, obsessive testing program. Zipline is bringing something completely new into the world. To safely do so, we test extensively until we’re confident our Zips can reliably fly in conditions we expect in the communities where we operate. 

Our testing process begins the moment a new concept is born and continues beyond the final release. Throughout, we consider the worst possible outcomes. Our engineers predict scenarios where something goes wrong, then map the risk of each failure mode against the likelihood of it happening. Then, we build a program to test each scenario, force-ranked by probability and risk. 

For example, a piece of space junk falling from the sky and hitting the Zip would be catastrophic, but it’s extremely unlikely to happen. On the other end of the spectrum, the Zip will probably run into gusts of wind multiple times every flight, but the potential to affect the Zip and prevent it from flying is nearly zero. Then there’s everything in between.

To make sure we catch every issue we can before starting customer operations, we build and run a testing program for each design phase: prototype, integration, and validation. 

Prototype testing

Well before we lock in any detailed hardware designs, we use our vast past experience to predict the highest-risk aspects of the new concept and develop various prototype systems to de-risk and verify that the final product will work.

These systems do not have to look like the final product. We intentionally focus only on what we need to get critical flight data as quickly as possible. We then combine data from prototype testing with extensive simulations to bridge the gap between testing and final design performance. We do this in parallel with engineers designing the final product, creating a continuous feedback loop.

Integration testing

As we iterate through prototypes and get closer to our final design, we start flying vehicles that look more and more like the final product. During this portion, we shift our focus from proving the concept to developing and testing functionality of the hardware and software modules we need to safely deliver to a customer.

We not only do this in flight but also simulate how software and hardware components should perform using all manner of processes, from wind-tunnel testing to large-scale software-in-the-loop digital twin simulations. This ensures new software works across a variety of conditions before we reach flight testing.

Validation testing

Once we’ve integrated all functionality, it’s time to do high-volume flight testing. After all, the best way to prove something is going to be reliable and work for thousands of customer flights is to conduct tens of thousands of test flights. We do this across many test locations, covering a wide array of weather conditions—from windy summer days to cold, snowy nights—to confirm our systems work as expected when delivering to customers through all types of weather.

In parallel to our extensive flight test campaign, we run hundreds of thousands of tests in simulation to ensure safety in the event of corner cases—situations that are hard to reproduce in test flights—like a flight during hurricane-force winds while a motor is damaged.

From the day we get our first parts, we complement our system-level testing with sub-system validation. This allows us to focus on each specific hardware’s most risky failure modes and stress them well beyond their limits, like flexing a wing above what it would ever experience in flight or putting our battery through out-of-this-world heat to make sure it still operates.

Once we have a design that is validated and ready for safe, reliable customer deliveries,  we need to ensure every vehicle is built to that design and behaves as we expect before every flight. We do this through two phases of testing before delivering to the customer: manufacturing and preflight testing.

Manufacturing Testing

Every vehicle we build goes through a rigorous suite of automated tests, developed in-house, to validate each piece from the smallest part to the final integrated system. We use these testers to make sure that the structure is sound, our flight computer is performing as expected, all the cables are routed to specification, and more.

Preflight testing

All of our testing up to this point gives us a high degree of confidence that our Zips will safely and reliably deliver. Still, we add an extra gate before each Zip launches: preflight. During this process, we conduct extensive, automated checks to determine everything from whether a propeller is chipped to ensuring our wing ailerons are moving through their range. These preflight checks themselves are tested extensively to ensure that we never take off when something is wrong with the vehicle.

The day that we launch a new Zipline system is really only the beginning. We continue iterating on our testing processes in parallel, leveraging new data from operations to help us design new tests, build functionality, and keep improving the customer experience. 

Here, our existing operational experience is a huge advantage. We’ve flown over large bodies of water in Japan, through severe storms in Rwanda, in extremely hot temperatures in Ghana and -30 degree temperatures in North Dakota. We use the wealth of insight we’ve collected from these operations to keep optimizing our testing strategy.

At Zipline, the relationship between testing and operations creates a feedback loop. We test to operate safely, we operate safely in increasingly diverse environments, which leads us to testing against more scenarios, ultimately improving access to critical goods and services, through Zipline delivery, for people everywhere.