Faced with limited resources in a pandemic, Greece turned to machine-learning software to decide which sorts of travelers to test for COVID-19 as they arrived in the country.
The system in question used reinforcement learning, specifically multi-armed bandit algorithms, to identify which potentially infected, asymptomatic passengers were worth testing and putting into quarantine if necessary. It also was able to produce up-to-date statistics on infections for officials to analyze, such as early signs of the emergence of COVID-19 hot spots abroad, we’re told.
Nicknamed Eva, the software was put to use at all 40 of Greece’s entry points from August 6 to November 1 last year. Incoming travelers were asked to fill out a questionnaire detailing the country and region they were coming from as well as their age and gender. Based on these characteristics, Eva selected whether they should be tested for COVID-19 upon arrival. At its peak, Eva was apparently processing between roughly 30,000 and 55,000 forms a day, each form representing a household, and about 10 to 20 per cent of households were tested.
Crucially, the software would take the test results as feedback and learn from them to improve its accuracy, and would ensure it was not just getting good at identifying a few types of passenger – it would test types of travelers it rarely encountered to ensure it had a wide set of data. In other words, there was more to it than logic like: if departure location equals London then test.
The software was crafted to home in on high-risk travelers without relying on test figures provided by individual nations, which might under-report infections, suffer from biases, or lag behind the actual spread of the virus. Instead, Eva would use its own fresh, real-time data from people arriving in Greece, and try to keep infected folks away from the general population to help mitigate the pandemic.
There is a very interesting pattern that we observed and report in our study that shows that increases in the prevalence that we measure through our system are followed by a pickup in cases a few weeks later in the corresponding countries
“First, given current information, Eva seeks to maximize the number of infected asymptomatic travelers identified,” the US-Greece academic team behind the software explained in a paper on their work that was published this month in Nature. “Second, Eva strategically allocates some tests to traveler types for which it does not currently have precise estimates in order to better learn their prevalence.”
The code identified 1.85 times as many asymptomatic, infected travelers as random testing methods, “with up to 2-4 times as many during peak travel,” according to the team. “To achieve the same effectiveness as Eva, random testing would have required 85 per cent more tests at each point of entry,” their paper stated, regarding that first figure.
Eva also identified “1.25-1.45 times as many asymptomatic, infected travelers as testing policies that only utilize epidemiological metrics,” and it was able to reveal which countries were about to encounter an uptick in cases, too, it’s claimed.
That last part is important because, according to the paper, Eva’s recommendations led to 10 countries being grey-listed by Greece, which meant people from those nations were always required to be tested, which reduced non-essential travel from those places.
“There is a very interesting pattern that we observed and report in our study that shows that increases in the prevalence that we measure through our system are followed by a pickup in cases a few weeks later in the corresponding countries,” Kimon Drakopoulos and Vishal Gupta, assistant professors in the data sciences and operations department at the University of Southern California’s Marshall School of Business, who co-wrote the paper with colleagues, told The Register via email.
Ultimately, Eva was built to help the Greek authorities perform virus tests efficiently, in terms of ensuring that groups of people likely to have the virus were tested and those unlikely to have it were not. One way to cut the spread of COVID-19 in the country would be to test everyone at the border before entry, and block or quarantine those with coronavirus. Greece’s resources were constrained yet it couldn’t close its doors completely; tourism is the nation’s biggest industry, and it was thus relying on it.
“We had enough resources to test about 10 per cent of arrivals in the peak tourist season and 20 per cent in the off-peak tourist season when arrivals were lower,” the academic duo said.
Eva was terminated after November. “When the tourist season was over, the number of arriving international passengers became very low, and so there was very little benefit to permitting non-essential travel to the country,” Drakopoulos told The Reg.
“Hence, Greece decided to close the borders to non-essential travel and reassign all of the medical personnel and resources from Eva towards internal pandemic measures – testing local population, vaccinations, reopening schools, monitoring local lockdowns, and social distancing measures.”
The researchers declined to say how many people total were tested after being singled out by Eva, citing privacy reasons. The Register asked for the percentage of passengers selected by the system for testing who were confirmed to be infected with COVID-19. The accuracy rate for catching asymptomatic carriers of the bionasty wasn’t very high, though accuracy is tricky to evaluate as a metric of performance for Eva.
“In our setting, COVID-19 prevalence is generally low (e.g., ~2 in 1,000), and arrival rates differ substantively across countries. Combined, these features cause our testing data to be both imbalanced (few positive cases among those tested) and sparse (few arrivals from certain countries),” as the paper put it.
Gupta told El Reg that accuracy and Eva’s false positive and negative rates weren’t relevant in the study. “The project goals was not prediction, or guessing whether someone is [infectious] or not. Rather the goals were largely prescriptive: recommend how many and what types of passengers to test to both identify asymptomatic infections and maintain good estimates of COVID-19 prevalence across passenger types.”
To improve Eva’s performance at catching those asymptomatically infected with COVID-19, the academics would need more information from the passengers. That’s non-trivial, however, given rules around privacy and medical data.
Clearly, having access to more data would improve performance but would compromise people’s privacy
“Clearly, having access to more data would improve performance but would compromise people’s privacy,” Drakopoulos and Gupta told us.
The team plans to improve their open-source code so that other countries as well as companies, university campuses, and schools can deploy Eva during the pandemic.
“One area where we’d love to see these ideas applied is into the now ‘routine’ testing we currently see in US schools, large office buildings, etc,” they said.
“Leveraging a similar reinforcement learning approach as in Eva, it might be possible to make these systems more efficient and focus testing on high-risk individuals. The models would have to be slightly different, but it might be an interesting avenue for future work, especially in regions of the country where vaccination rates are low.” ®