Blog / Testing Lessons from the COVID-19 Vaccine Rollout

Blog - Testing Lessons from the COVID-19 Vaccine Rollout

Testing Lessons from the COVID-19 Vaccine Rollout

When Massachusetts opened up COVID-19 vaccines to a broad group of residents last Thursday, people who met the new requirements quickly went online to schedule their shots.

This spike in traffic caused the vaccine appointment portal to crash. So did the state’s health and human services telephone hotline.

Residents called the crash “a nightmare,” “frustrating” and “unforgivable.” Other government agencies suffered similar failures with their online vaccine portals, including the state of Minnesota and Florida’s Pinellas County. Once the Massachusetts vaccination site recovered, many users found the lengthy — and often redundant — data entry process painful.

What went wrong?

Stuart Madnick, director of the cybersecurity center at the MIT Sloan School of Management, told The Boston Globe that the state system was most likely cobbled together too quickly to permit adequate testing of software and servers. “A new system is likely to have flaws in it,” Madnick said. “When you try to rush things out, that’s one of the likely outcomes.”

Leaders from the company managing the Massachusetts COVID-19 vaccination portal said the site did not scale adequately to handle such a dramatic surge in traffic. Nearly one million state residents met the new vaccine requirements; only 60,000 appointments were available through the portal. Neither the website nor the call center was prepared.

The right tests could have prevented mass frustration

Frustrated vaccine-seekers said that much of the aggravation could have been avoided. They’re right: load, stress, and soak testing would have helped developers better understand how the site would perform in real-world scenarios. Rigorous QA processes include non-functional tests to identify a multitude of failure points, such as inability to withstand high volume, errors on certain devices or networks, bugs and faulty code.

Simple tests could have revealed how both the vaccine portal and hotline would respond to heavy traffic. With a clear understanding of limitations, extra servers or load balancing could have let the portal’s operators scale server capacity to meet spikes in demand. Companies rely on Applause to assess readiness for high volume in a range of scenarios, such as concert ticket sales, financial market fluctuations and live pay-per-view sporting events. All too often, downtime means lost revenue. Investing in testing prior to launch reduces the risk of lost sales.

User experience testing would have uncovered opportunities for improvement as well. One of the biggest complaints about the Massachusetts vaccine portal was that users were required to enter the same information every time they tried to book an appointment at a different time or location. Simple usability or UX tests would have revealed the opportunity to streamline the appointment-scheduling process by having users save profiles with the relevant information, rather than starting from scratch each time.

What companies should learn from the vaccine portal crashes

You can’t rush a product to the finish line and expect it to work when thousands of people access it simultaneously — unless you’ve thoroughly tested. Whether you’re a government agency or a highly recognizable brand, you must deliver exceptional digital experiences — or risk loss of revenue and reputation.

To prevent bugs, crashes and angry customers, organizations can adopt a holistic testing strategy. A holistic approach assesses different dimensions of digital quality, such as functionality, usability, accessibility and more — all to ensure the best possible user experience.

In a recent blog post, Applause’s John Kotzian explained how the right testing environment can help organizations avoid costly errors. Many companies can’t sufficiently replicate real-world scenarios in house or a lab, so they turn to a testing partner they can trust to make sure they’re executing a holistic strategy. Outsourcing certain types of tests allows internal teams to focus on additional logistical challenges, such as preparing the call center to better handle a surge.

Taking time up front to make sure things work smoothly pays off, saving costs and building customer confidence. Make sure you have a healthy testing plan in place. Learn to develop a holistic testing strategy from this ebook.

Published: February 26, 2021
Reading time: 4 min

5 Threats to Streaming Content Protection

Protect streaming media revenue against its many emerging threats

Why Retail Digital Quality Matters More Than Ever 2

Against a challenging economic backdrop, our 2023 State of Digital Quality in Retail report highlights global retail testing data trends and how retailers can successfully address their quality issues

‘Tis The Season: What Shoppers Want in Winter 2023

In October 2023, Applause conducted its annual holiday shopping survey to gather research on consumer trends ahead of the festive season.

Make Every Day World Usability Day

Take time today to consider how well your digital experiences serve different audiences

Get Ready for the European Accessibility Act

Read the highlights from our recent webinar on how to prepare for the new law

Crowdsourced Software Testing FAQs

We answer some common questions about crowdtesting