COVID-19 screening data: challenges for ML algorithms

Screening images released by Recursion

In this post, we summarize the status of screening data available for training machine learning models for predicting antiviral properties against COVID-19. Since we started the AICures project in March, we spent inordinate amounts of time trying to find relevant screening data, specifically molecules with their associated antiviral SARS-CoV-2 score. We believed that once this data became available, we would be all set: the models can be trained to identify antivirals and their combinations, which can in turn be tested experimentally by our collaborators in Walter Reed and others. The reality turned out to be much more complex. In the last ten days, multiple groups have released their screening data. Of particular value are library level screenings which report activity scores for a large number of molecules, including both active and inactive compounds. To date, we are aware of two such releases: screening dataset from AMU [2] and image-based screen from Recursion [4]. In addition, several groups report their experimental testing of a few dozen compounds, preselected according to their expected antiviral activity. For instance, [6] preselected 75 compounds based on the SARS-CoV-2 protein-protein interaction network. Yet, other groups only released their positive hits [1]. In most cases, antiviral activity is measured by IC50, the concentration of drug required to reduce viral activity by half. This metric is inversely related to CC50, the concentration of drug that kills half of the host (human) cells. A useful drug must be both efficacious and safe, so ideal drug candidates have low IC50 and high CC50. Besides IC50 and CC50, some papers report alternate metrics, such as scores derived from a deep learning framework or cell viability against some baseline drug activity [4,6].

To our surprise, we found a stark disagreement between all these reported results [1,2,3,4,5,6]. Jointly, these papers report 200+ active compounds, however only 32 were identified as active by more than one study. In the figure above, we show the range of scores for these 32 compounds. So far, only a single compound – Remdesivir – is identified as active by all studies it appeared in [1,2,3,4]. Other top clinical candidates, Chloroquine and Hydroxychloroquine are identified as active by [2,3,5] and [2,3,6] respectively, but [4] reports that both are inactive. For 22 compounds that were assessed by two studies, their activity assessments are contradictory. (For a full comparison of these papers, check this spreadsheet.) What can contribute to such divergence? We asked our life science colleagues and they pointed out that such discrepancies in screening are expected due to differences in experimental testing. These include the type of cell used in the experiments, the strain of the virus, and the way assays were set up. Which of these screening results are indicative of the drug efficacy when given to a patient? Not clear. On the positive side, consensus between studies gives us some clues. For instance, Remdesivir, identified as active by 4 independent screens, has shown efficacy in trials and was recently approved by the FDA for treating COVID-19 patients. On the AI side, it suggests that we need to find a way to reconcile these divergent experimental results. It will be easier to separate the signal from the noise, as more screening data becomes available. Relation to Clinical Trials: One interesting question is how these experimental findings relate to compounds which are currently in clinical trials. To answer this question, we downloaded trial information from, which reports 272 COVID-19 studies involving 281 compounds. The graph below shows the number of clinical trials for the most frequently used 30 compounds, that collectively account for 155 trials. The following graph shows number in-vivo and in-vitro studies about these compounds in the context of COVID-19 (as reported in

As these graphs show, many drugs used in these trials have not been studied experimentally. Moreover, some of these drugs were shown as inactive [7,8,3]. Taken together, these two graphs show a clear discord between compounds tested experimentally and clinically. Does it mean that clinical trials involve drugs that are not sufficiently tested? Are screening results not predictive of compound efficacy in patients? Lots of questions, not many answers. About Data Sharing: When we started this work, we assumed that a single library level screening would be sufficient for our AI work. As the statistics above show, the matters are much more complicated -- different experimental settings for in-vitro testing inherently lead to divergent results. Instead of waiting for the perfect screening results (whatever that means), there is a lot of value for learning from multiple screens. To this end, sharing the data is paramount. We are grateful for all the groups who shared their screening data with the broad research community ( [2] , [4]). We are eagerly awaiting screening data from Gates-Wellcome Therapeutics Accelerator and NEIDL, federally and/or philanthropically funded initiatives which are yet to release their screening results. We also very much hope that groups releasing their active compounds [1] will also share the full screening results as they are highly informative for machine learning systems. We hope that other labs would also err on the side of over-sharing rather than holding back, especially at times like these. Similarly, we and others are releasing methods for all to use.

[1] Riva et al. A Large-scale Drug Repositioning Survey for SARS-CoV-2 Antivirals. (2020). [2] Touret et al. In vitro screening of a FDA approved chemical library reveals potential inhibitors of SARS-CoV-2 replication. (2020). [3] Jeon et al. Identification of antiviral drug candidates against SARS-CoV-2 from FDA-approved drugs. (2020). [4] Heiser et al. Identification of potential treatments for COVID-19 through artificial intelligence-enabled phonemic analysis of human cells infected with SARS-CoV-2. (2020). [5] Weston et al. FDA approved drugs with broad anti-coronaviral activity inhibit SARS-CoV-2 in vitro. (2020). [6] Gordon, D. E. et al. A SARS-CoV-2 protein interaction map reveals targets for drug repurposing. Nature (2020). [7] Corley, Michael Jay, et al. "Comparative in vitro transcriptomic analyses of COVID-19 candidate therapy hydroxychloroquine suggest limited immunomodulatory evidence of SARS-CoV-2 host response genes." bioRxiv (2020). [8] McGrane, Victoria “Massachusetts to launch first US trial of Japanese coronavirus drug”