Concerning the Widely-observed but perhaps Under-appreciated Variability in Results from Screens

An opinion piece by Doug Lauffenburger

Doug Lauffenburger, Ford Professor of Bioengineering, in his lab.

David Castro-Olmedo/MIT

High-throughput quantification of effects of treatments or perturbations (including chemical, biomolecular, and genetic) on biological cell behaviors (such as growth, death, differentiation, and metabolism) is in general highly problematic for aggregative comparisons or integrative computational modeling. This problem arises from a variety of reasons mainly related to the underlying cell biology involved. Some of these reasons are noted in the previous AICures post, and others have been studied with experimental depth by cell biologists such as Peter Sorger. These kinds of underlying cell biological causes of assay outcome variations include:

  • cell type(s) used

  • medium used

  • other aspects of culture context (e.g., culture-ware, such that adhesion protein adsorption might be different)

  • time-point(s) used, both for treatments and measurements

  • quantification metrics used

Although these may seem to be superficially technical, they in fact are related to cellular physiology in profoundly mechanistic ways that influence measurements of treatment / perturbation effects via their effects on cell regulatory processes. A broadly essential issue is context-dependence; the first three bullets above are explicitly related to this issue, and the latter two are implicitly related. All biological function is dependent on context (with the exception of chromosomal DNA sequence; this is why DNA sequence data is relatively so easy to compile but so concomitantly relatively limited in effectiveness for understanding or predicting biological function). My view is that the “outcome” data from treatment / perturbation assays, such as IC50s representing drug inhibition of cell growth, are inadequate for modeling when not associated with data characterizing at least some of the underlying cellular physiology – which could reflect context. This is especially problematic when then aspiring to predict clinical outcomes from assay models, since the contexts are substantively disparate between in vitro biological assay measurements and in vivo biological behavior. One among numerous examples of how context-dependence can alter prediction of a drug effect from desirable to undesirable can be found in an example study we performed to explain why an envisioned biopharma therapeutic for rheumatoid arthritis was predicted from cell culture experiments to be beneficial but in clinical tests turned out to be detrimental. This problem of context-dependence for ‘translation’ is also true for aspiring to predict human clinical trial effectiveness from animal studies to human clinical trials. Accordingly, one ought never expect that mere “standardization” will be a definitive solution to assay variation, because standardization typically connotes selection of a particular context for all assays aimed at a given test to use identically. The most reliable solution, from my perspective, is to construct systems models relating underlying cellular physiological data to outcomes, and then translating those systems models – instead of trying to translate the mere outcome observations. A perspectives article outlining this approach can be found here. This of course requires more intensive characterization of assay biological variables instead of simply outcomes, but I believe it is well worthwhile for generating actionable data. An example of this kind of study can be found here.