Where are the rapid testing data?
The White House came under fire this week for incorrectly using Abbott ID NOW rapid COVID-19 tests. But there are other issues with how these tests are being used and reported.
Welcome back to the COVID-19 Data Dispatch, where we explore just how confusing testing data can get.
This week, I’m explaining the issues that occur when rapid tests are either not reported or lumped in with other test types in public COVID-19 data. Plus, a CMS update, a resource compilation spreadsheet, and my acceptance in a new course at CUNY’s Newmark Journalism School!
As always, shares are appreciated. If you were forwarded this email, you can subscribe here:
Simplified data lag an increasingly complicated testing landscape
Abbott ID NOW tests, pictured at a health clinic in Williamsburg (my photo).
Two weeks ago, I went to my COVID-19 testing site of choice for a post-Maine trip screening. I walked the now-familiar ten blocks to the site, a private clinic in Williamsburg, and waited at the now-familiar counter for one of the receptionists to be available to check me in.
“Do you want a rapid test?” the receptionist asked.
“No, PCR, please,” I replied.
I had assumed that the “rapid test” she offered was an antigen test. Antigen tests, as I’ve described in this newsletter before, have a low sensitivity, meaning that they may miss identifying people who are actually infected with the novel coronavirus. (These missed cases are called false negatives.) Evidence also suggests that antigen tests will return more false negatives for patients who are asymptomatic. As I was not exhibiting COVID-19 symptoms, and rather wanted to get tested because I had recently traveled out of the state, I was interested in receiving a more accurate test.
But confusion quickly ensued: the rapid test that my clinic offered, as it turned out, was not an antigen test. It was a nucleic acid amplification test—a model manufactured by Abbott called the ID NOW. Like PCR (polymerase chain reaction) tests, this test uses gene amplification to identify genetic material associated with the novel coronavirus in a patient’s mucus sample. But unlike PCR tests, the ID NOW test does not require clinics to send samples out to faraway labs for processing. This test is distributed with small, in-house DNA amplification machines that can provide patients with their results in 15 minutes. I got the result of my ID NOW test later that same afternoon. (And then I got the results of a second test, this one a PCR test which I had asked the clinic to request at a lab, several days later. Both tests were negative.)
I hadn’t heard of Abbott ID NOW tests before last week. But they’re in the news now, connected to what has become America’s most infamous COVID-19 outbreak: President Trump’s White House relied on Abbott ID NOW tests. And they used these tests incorrectly.
No test type specification in molecular testing data
Rapid testing has been the White House’s defense against critiques of COVID-19 carelessness, explains Slate reporter Daniel Politi. Each day, staffers would get tested with Abbott ID NOW tests. Upon receiving negative results, staffers would be cleared to take off their mask and act without consideration for social distancing. Boxes of the ID NOW tests used for this screening have been pictured at the White House since March.
But how accurate are those negative results? If you’re asymptomatic: not very. The ID NOW test is only authorized for use in people with symptoms. A guidance from the Food and Drug Administration (FDA), updated on September 17, specifies:
The ID NOW COVID-19 is authorized for use with respiratory specimens collected from individuals who are suspected of COVID-19 by their healthcare provider within the first seven days of the onset of symptoms.
The majority of White House staffers who were tested with these tests had not been coughing and wheezing, nor had they been suspected of COVID-19 by a healthcare provider. In asymptomatic patients like these, as many as one-third of negatives returned by the ID NOW test may be false negatives. It’s no surprise, then, that the virus spread quickly through an indoor environment where staffers were using rapid tests—but doing little else.
White House staff are not the only people who used the wrong type of test to check their COVID-19 status. I shouldn’t have gotten an Abbott ID NOW test either. But when a nurse at my local clinic, which I saw as a site of trusted health expertise, offered one to me, I didn’t ask too many questions. It’s PCR, I thought. It’ll be accurate.
But first of all: the Abbott ID NOW test is not a PCR test. It’s in the same family as PCR tests (molecular-based, DNA amplification), but it operates on a different paradigm. And second of all, the health care workers at my clinic gave me no indication of how accurate this rapid test was, especially for my asymptomatic self. (Note: I have no hard feelings toward this clinic or any of the workers there. I’m simply using my own experience as an example of how poorly information about test types is conveyed to the public in general.)
What if my PCR test, sent out to a lab, had returned a positive result? I would have traversed Brooklyn, gotten groceries, grabbed coffee at a neighborhood cafe, and attended a protest in Queens that weekend, buoyed by a negative result yet unknowingly spreading the coronavirus to my fellow New Yorkers. And both of my tests would have been accounted for on New York’s testing dashboard in one single tick mark. New York reports tests in encounters, so my two specimens would have been deduplicated as simply “one person tested on September 30.”
I say “would have been” because I am not, in fact, sure that my Abbott ID NOW test made it into the New York Public Health Department’s database at all.
Here’s a line from the press release that Abbott put out on October 2, the day that Trump tested positive:
While we don’t know the details on who was tested and on which devices, we know that more than 11 million Americans have taken the ID NOW test, helping to stop the spread of the virus.
11 million is a pretty big number: if these tests were all included in the national count reported by the COVID Tracking Project, they would make up about 10% of the total. But are the ID NOW tests included in the COVID Tracking Project’s count? It is difficult to say. The majority of state public health departments, the sources for the COVID Tracking Project’s data, simply report total counts of nucleic acid-based tests, if they specify their test type at all.
State COVID-19 dashboards fail to separate out test counts by method or manufacturer. Some states, such as Maine and Illinois, acknowledge that their test counts include all “nucleic acid-based tests,” not only the PCR tests which fall into this category. Other states, such as Oklahoma and Florida, report detailed data about the testing capacity of individual labs and point-of-care facilities, but fail to supply the testing models used at each location. South Carolina acknowledges a small number of “unknown” tests, for which the public health department is investigating possible categorizations. The Department of Health and Human Services, meanwhile, only reports PCR tests, absent counts of any other molecular test type.
And, returning to Abbott’s press release: the manufacturer admits that they “don’t know the details on who was tested and on which devices.” This leaves a lot of open questions about how rapid testing data are being collected; Abbott seems to imply that even the manufacturer itself does not have access to specific information about where and how their tests are in use. If I had to guess, I’d say that 11 million figure comes from bulk test orders sent out by facilities like my local clinic.
It’s great for healthcare responses that Abbott tests can be processed quickly on-site, in facilities like a healthcare clinic or a major political site. But public health departments don’t have relationships with clinics—they have relationships with testing laboratories. When tests aren’t sent out to a lab, it’s easy for them to fall through gaps in a data pipeline which relies on complex electronic reporting systems. (This is also a problem for antigen tests.)
So, the problem stands: Abbott ID NOW tests are in use across the country. 11 million of them have been distributed since March. But where are the tests being used, how many of them have returned positive results, and are they being used correctly—to quickly diagnose patients with COVID-19 symptoms? Without these data, it is difficult to determine whether Abbott’s test should be part of America’s testing strategy going forward.
Conflating antigen and PCR tests
I can’t discuss the data pitfalls of rapid COVID-19 testing without devoting a few paragraphs to antigen tests.
Five days before his positive COVID-19 test was made public, President Trump announced a plan to deploy 150 million antigen tests across the country. 100 million tests will go to state and territory public health departments, while the remaining 50 million will go to facilities supporting vulnerable populations such as nursing homes, hospice care facilities, and historically Black colleges. Trump discussed how useful antigen tests could be for screening, suggesting that teachers could be tested regularly.
The tests Trump praised are rapid antigen tests manufactured by Abbott, which received FDA Emergency Use Authorization at the end of August. Abbott’s antigen tests are cheap—each one costs $5. And, like the ID NOW tests, they’re fast—patients receive results in 15 minutes. But, also like the ID NOW tests, antigen tests are more effective for symptomatic people.
Here is how Alexis Madrigal and Rob Meyer explain it, in an article for The Atlantic published this past week:
If distributed en masse and used to screen asymptomatic people, these antigen tests will deliver hundreds of thousands—if not millions—of false results, they say. False negatives could lead to reckless behavior by people who don't know they're sick. False positives can also put people at risk: If a virus-free nursing-home resident with a false positive is placed in a COVID-19 ward, that person could become infected.
Not even antigen testing’s trial run—nursing home deployment—is going well. Nevada’s public health department recently instructed nursing homes to stop using antigen tests due to their rate of false positives, a move which was heavily criticized by HHS testing czar Admiral Brett Giroir.
This is not to say that antigen tests are useless; their utility is still being debated in epidemiological and public health circles. The advantages of these cheap, fast tests may outweigh the dangers of their low sensitivity. But to truly understand this problem, we need access to better antigen test data—not just 60 tests from Nevada nursing homes (the sample size upon which that state’s decision was made).
If 11 million Abbott ID NOW tests are a data concern, 150 million Abbott antigen tests are a data minefield. For the past month, I’ve been working on an analysis for the COVID Tracking Project which covers how states are reporting—and failing to report—antigen test results. This analysis is set to be released in the next week, in all its detailed glory. But I can tell you now that the majority of states are not reporting antigen tests in separate counts from DNA-based tests, just as the majority of states are not reporting other types of DNA-based tests as separate from PCR tests. In fact, several states now specifically report that their testing counts combine PCR and antigen tests in one figure.
When two different test types are conflated in a single total figure, it is difficult to analyze the progression of how either test is being used. I can’t tell you how many antigen tests are being put to use across the country, or how effective they are at containing COVID-19 in a given community, if these test numbers are lumped in with another test type. Test lumping also presents a(nother) challenge for test positivity calculations, as antigen and PCR tests have very different sensitivity levels.
And even the few separate antigen test counts that states do report are likely significant undercounts of the actual scale of antigen testing going on in the U.S. As I mentioned earlier, no data reporting pipelines are set up for tests that occur rapidly in point-of-care settings. The Center for Medicare & Medicaid Services (CMS) does not report the number of antigen tests that have occurred in nursing homes, for example.
So far, it appears that state and federal public health agencies alike are unprepared to report the results of antigen tests. Before the White House deploys its 150 million Abbott antigen tests, I hope to see that change.
CMS data and reporting updates
The county-level testing dataset published by CMS has become a regular topic for this newsletter since it was released in early September. As a refresher for newer readers: CMS publishes both total PCR tests and test positivity rates for every county in the country; the dataset is intended as a resource for nursing home administrators, who are required to test their residents and staff at regular intervals based on the status of their county.
This past Monday, October 5, I was pleasantly surprised to find a new update posted on CMS’ COVID-19 data page. I say “surprised” because I had been led to believe, both by past dataset updates and by reporting when the dataset was first published, that this source would be updated once every two weeks. And yet, here was a new update, with only one week’s delay (the last update before this was on Monday, September 28). CMS is also now posting weekly updates on an Archive page which goes back to August 19; some of these updates are older, while others were posted or edited in the past week.
I always appreciate more frequent data, even when the data provider in question is not particularly transparent about their update strategy. Frequent updates are particularly useful for testing data; the nursing home administrators monitoring testing in their counties will be able to see information that better reflects the level of COVID-19 risk around them.
I’ve updated my Tableau dashboard which visualizes these county-level data:
As you can see, the majority of the Northeast and much of the West Coast continues to be in the green (positivity rates under 5%), while areas in the South and Midwest are not faring so well. Twelve counties have extremely high positivity rates (over 30%), eleven of which are in Midwestern states. This table allows you to rank and sort the test positivity rates by state.
Also, a note on my methodology for this dashboard: in earlier iterations, I used state-level data from the COVID Tracking Project to calculate state test positivity rates for the same time period as the CMS has provided county-level rates. I then compared the county-level rates against state-level rates; this was the source of the “[x]% above state positivity rate” tooltips on the dashboard. After reading a new COVID Tracking Project blog post about the challenges of calculating and standardizing positivity rates, however, I realized that combining positivity rates from two different sources might misrepresent the COVID-19 status in those counties. So, I switched my method: the county-to-state comparisons are now based on averages of all the CMS-reported county-level positivity rates in each state.
Finally, out of curiosity (and to practice my Tableau skills), I compared the CMS-reported test positivity rates for the five counties of New York City to the city-level rate reported by the NYC Department of Health.
The positivity rates reported by the two sources follow the same general direction, but it’s interesting to see how the rates diverge when the five counties split up. Manhattan remaining far below 1% while Brooklyn surges up to 2%? Not surprising.
Meanwhile, CMS is cracking down on COVID-19 reporting from hospitals: NPR reported this week that hospitals which fail to report complete, daily data to HHS can lose money from Medicare and Medicaid, starting this coming January.
Entrepreneurial training for the COVID-19 Data Dispatch
This newsletter is about to step up its game in a big way. I am absolutely honored and thrilled to announce that I’ve been selected for the inaugural cohort of the Entrepreneurial Journalism Creators Program.
This program is a new 100-day course from the Craig Newmark Graduate School of Journalism at the City University of New York. Starting this coming week, I will learn how to better understand the needs of my audience (that’s you!) and develop a sustainable journalism project. The other 19 students in my cohort include journalists from around the world, working on projects ranging from local news to global health reporting. You can read more about the program and its participants here.
I’m incredibly grateful to all of the readers who have supported this project so far, whether you subscribed to my first issue or just found this newsletter last week. Thank you for your support and feedback on how I can best make COVID-19 data accessible. I’m excited to share what I learn in my course with you, and to grow the scope and resources offered by this project.
Featured data sources
As I promised in previous weeks, I’ve compiled all the data sources featured in this newsletter into a resource spreadsheet. The doc includes 56 sources, sorted by category (schools, testing, etc.) with descriptions and notes from past newsletters. I’ll keep adding to it in future weeks!
The Human Mortality Database: This database includes detailed population and mortality data for 41 countries. In response to the COVID-19 pandemic, the team behind the database has started compiling weekly death counts, which can be used for excess death calculations; they have compiled counts for 34 countries so far.
SARS-CoV-2 Superspreading Events: Superspreading events, or instances in which many people are infected with the novel coronavirus at once, have been identified as a major force behind the spread of COVID-19. This database includes over 1,400 superspreading events from around the world, with information on each event’s timing, location, inside/outside setting, and more.
COVID-19 Risk Levels Dashboard: A new map from the Harvard Global Health Institute and other public health institutions allows users to see the COVID-19 risk for every U.S. county. These risk levels are calculated based on daily cases per 100,000 population (7-day rolling average).
New York Times College and University COVID-19 counts: The NYT is now releasing the data behind its counts of COVID-19 cases reported on college and university campuses, which the paper has been collecting in surveys since July. The survey includes over 1,700 colleges. This initial data release only includes cumulative data as of September 8—and it does not include denominators. NYT reports that the data will be updated “approximately every two weeks.”
COVID source shout-out
I’m doing a shout-out instead of a callout this week, because sometimes even I tire of finding data issues to upon which I can focus my tirades.
Every few weeks, my mom forwards me an email from the Town Manager in my hometown, Glastonbury, Connecticut. This email comprises the Town Manager’s Weekly COVID-19 update, including data for the town, updates for the state, and the occasional public service announcements. The most recent email, sent on October 7, includes Halloween best practices, information on flu clinics, and absentee ballot resources.
After peering at endlessly complicated state dashboards during COVID Tracking Project shifts, it’s refreshing to see a COVID-19 update which presents data as simply as possible—no hovering or scrolling required. And yeah, they clearly made that chart in Microsoft Excel, but it does its job!
More recommended reading
My recent Stacker bylines
30 ways COVID-19 is impacting children around the world (updated 10/6)
News from the COVID Tracking Project
Bonus
A Deep Divide on COVID, and Masks, in Farm Country (Civil Eats)
The decline of local newsrooms could make it harder for us to detect the next disease outbreak (Columbia Journalism Review)
How Transparent Is Your College's COVID Dashboard? (Inside Higher Ed)
That’s all for today! I’ll be back next week with more data news.
If you’d like to share this newsletter further, you can do so here:
If you have any feedback for me—or if you want to ask me your COVID-19 data questions—you can send me an email (betsyladyzhets@gmail.com) or comment on this post directly:
Fill out my survey to share your topic suggestions and questions:
This newsletter is a labor of love for me each week. If you appreciate the news and resources, I’d appreciate a tip: