Greg Finak, CTO Ozette Technologies
You may have heard of the Duke research scandal (Wikipedia link) where some clinical trials proceeded based on fabricated data that ultimately harmed patients. If not, it’s a riveting story and worth a detailed read. The initial scientific claims were fascinating, that gene expression profiles of cancer cell lines (derived from the at the time cutting-edge high-dimensional microarray technologies) could be used to guide cancer therapies for human subjects. These claims turned out to be incorrect, as well as later findings that built upon them.
What’s interesting is that the origins of this scandal were a series of likely unintentional computational errors that seemingly snowballed and only turned into the apparent cover-up after they were brought to the authors’ attention.
The details of the investigative work to try and reproduce much of the original work were published by Kieth Baggerly and Kevin Coombes in 2009. Their efforts uncovered simple computational errors (i.e. off-by-one indexing, mislabeled samples) and their paper discussed their findings and their correspondence with the authors in trying to raise their concerns.
Around this time, I was also working with microarray data as well and had published some impactful work in a related area. I was fortunate enough to hear Baggerly lecture on the topic, which is where I first heard the term “Forensic Bioinformatics”. It left quite an impression and instilled in me a firm skepticism of lofty claims made without the requisite transparency.
I think the story is also an interesting case study of how compounded complexity can mask simple mistakes, only because in a complex system, no one individual can have a complete understanding of the whole. This compartmentalization is normal in modern science, which is by necessity an interdisciplinary affair. The real human risk is if we don’t understand something, it becomes easy to dismiss it when there is a problem, or more insidiously we may not even recognize it as a problem.
Of course, in practicing science we can’t hope to understand everything about every tool or machine we use and interact with on a daily basis. We rely on scientific instruments to correctly generate reliable data, even if we don’t understand the intricate engineering complexities of exactly how the instruments work. We rely on software to produce trustworthy analytical results, even if we did not write the software or develop the underlying methodologies. We don’t always understand, or need to understand in intricate detail, what happens to the data upstream of where we are looking, but we trust that there are those who do and that they have the tools necessary to verify their work.
In the human health space this is critical, and when it breaks down it has very real risks and consequences, especially if warnings are ignored or best practices not followed. Processes of quality assurance, engineering practices, system design, formal validation, and regulations are in place to ensure through practice and legal force that the systems we rely on, work as intended. These are the formal mechanisms that ensure data integrity, accountability, reliability, accuracy and reproducibility.
Yet, when research doesn’t touch human subjects, and even sometimes if it does but is purely for exploratory work, as is the case with much academic research software, then software systems are usually not validated. In fact up until relatively recently, academic Bioinformatics was quite the Wild West. Design and version control, code review, unit testing, discussion of software development practices–these things have only become widespread in the field over the past 10-15 years, driven in part by the story that played out above, a series of papers about the “scientific reproducibility crisis” some years ago, an influx of professionally trained software engineers into the domain, and concerted efforts for professional training in good software development practices.
These practices have improved the quality of research software, but still most academic research software in the industry today is not validated off the shelf in the strict regulatory sense. Instead, users who need to apply a piece of software or system for data analysis in human clinical trials must take on the burden of the software validation process themselves. It is a costly and time consuming but necessary endeavor. It also raises an interesting conundrum: how does an individual or organization verify that a software system that they did not write, is performing as intended? For “simple” systems and processes, ones that can be readily understood and clearly documented, expectations laid out, it should be relatively easy to validate and verify. But what if the system is complex? What if the user or organization doesn’t understand the inner workings well enough or have the expertise to verify the system or process? In such cases validation may not even be possible or viable for that organization. After all, how can we hope to validate something we don’t understand? This is one reason, I believe, that complex and ad-hoc computational methods often remain unvalidated and for “research use only”.
For such complex systems, the veracity and quality of outputs and results are taken largely at face value, on trust alone, with the user accepting that the approach underwent sufficient peer review from experts who do understand the system. It is, I believe, one of the reasons, manual analysis remains the gold standard for cytometry in a clinical trial setting. This is something we sought to change at Ozette, from the outset.
Modern cytometry is a fascinating technology. It can be remarkably high dimensional and high throughput thereby sharing some of the complexity risks of other high dimensional technologies of the past (such as microarrays, or modern scRNASeq).
It is an interesting example of a complex system because it is conceptually a very simple assay. Cells are stained with antibodies specific against certain proteins, conjugated to fluorophores that are then excited as they pass in front of a laser, single file, in the flow cytometer. The fluorophores emit light, which is filtered and captured at a series of detectors. Following electronic amplification and integration, the signal is recorded, cell by cell. The intensity of the signal is proportional to the amount of fluorophore on the cell, which is proportional to the amount of the specific protein on each cell. Each of these phenomena is well understood, by physicists, or biologists, or immunologists, and a trained lab technician can set up the instrument, ensure cell staining is appropriately titrated, run controls and samples, and analyze the data. On modern instruments, the capture software is responsible for performing the compensation/unmixing, which can be further adjusted in other software, and then the scientist can “gate” out the cells of interest–essentially a manual clustering step. Using their expert knowledge of immunology, they record the abundances of different cell populations and then pass that data to the statistical team who will perform prespecified analyses to determine the answer to scientific questions at hand for the trial. As I said, it is conceptually simple. Anyone working in the field would intuitively understand all the steps. But when one digs down into the details, there is an awful lot going on that few individuals can claim to fully understand.
Would we expect the lab technician, hired to process samples at scale, to understand the mathematical underpinnings of unmixing or compensation, the various data transformations, data file formats, or intricacies of downstream statistical modeling? Would we expect the statistician to understand the complexities of antibody affinity and avidity, how to appropriately titrate antibodies to set up an experiment, or calibrate a flow cytometry instrument? It’s not a reasonable expectation to be expert in all things, and we rely on each person in the chain of this complex system to understand their part and do it well.
Yet, in flow cytometry there are non-obvious knock-on effects between the different layers of complexity. That is to say, if something goes wrong with the instrument or its setup, if something is incorrect or inaccurate in the data processing layer, perhaps application of an suboptimal or inappropriate algorithm or method, (maybe because the data came from a different type of instrument than anticipated), or the titration wasn’t quite right, then these issues will manifest themselves in a readout that is not only inaccurate but it rarely obvious that it is inaccurate. The problem only becomes visible somewhere downstream, sometimes far downstream from the individual responsible for that piece of the system.
In flow cytometry these different layers of complexity are tightly coupled, there are strong dependencies between them, and errors have far reaching causal impact. The system is sensitive to small changes, and the individuals who ultimately see the final output and rely on it to make decisions don’t usually have access to all the information–data, intermediate decisions, and quality diagnostics–necessary to evaluate the quality and veracity of the results. As a consequence, decisions are not fully informed and sometimes even inadvertently misguided. That is to say, cytometry can easily deceive the user.
Poor quality data is a well known problem in the flow field and ISAC (International Society for Advancement of Cytometry), the professional scientific society overseeing much technical development, has devoted extensive resources to education and training to improve data quality in the field. This might all seem somewhat academic, but it has real impact. I have done methodological research for many years. When we debate computational methods, we aren’t just nerd-sniping. Methods matter, and have real impact on data quality and ultimately timely, accurate and informed decision-making.
At Ozette, we work with partners to help them make sense of their complex cytometry data. We have repeatedly seen this systemic complexity affect data quality. The data problems typically arise upstream in the data generation process coming from a third-party, and our team often surfaces these issues to our partners for the first time. Generally the response is appreciative, but no one likes receiving bad news, especially when that news is that they’ve just invested a lot of money in an experiment that will not give them the readout they need. Unfortunately, we’ve also seen some partners who chose to ignore the data quality issues we surfaced.
The demand for higher quality data across the industry is part of the motivation for establishing Ozette, the Ozette Lab, and bringing together experts in cytometry assay development, panel design, computation, statistics, and engineering, and taking a vertically integrated approach to cytometry data generation and analysis. We’ve revisited, revised, the entire computational workflow with fresh eyes, garnering input from experts to understand where current accepted best practices and conventional wisdom breaks down or fails, and identifying root causes of data quality failures, and building entirely new methods and tools to identify and solve problems where needed.
Ozette’s state-of-the-art assay and platform for spectral flow cytometry data utilizes data-driven computational analysis, quality monitoring, and gating to rapidly and robustly identify cell populations. The platform incorporates the quality assurance controls and regulatory compliance necessary for clinical trial use through formal validation of the assay and underlying algorithm. To our knowledge, Ozette provides the only validated computational gating solution (Endpoints™), which, when combined with our validated 48 color spectral assay, delivers deep phenotyping and immune profiling for clinical trials. We’ve integrated unmatched data observability and explorability, with the goal to ensure our partners have visibility and transparency into their complex cytometry data across the entire vertical of data consumer personas, from those individuals that run the assay, to those that analyze the data, to those that rely on the data to draw insights and conclusions, or make decisions about drug development programs.
By taking on the data quality component of data generation and analysis, rather than pushing it onto the user, we have built tooling that enables us to work with labs and rapidly identify and diagnose data quality issues before they become problematic or pervasive, and provide our partners with advice and resources to generate the highest quality data so that time and resources are not squandered. The impact is that our partners’ drug development programs are accelerated, moving forward faster from phase I to II to III, because data is reliable, rich, and informative in a way that it has heretofore not been before. The impact of ignoring the problems of complexity are multi-fold–impacting human health, investor, and patient finances. The Ozette platform brings transparency and understanding to anyone who needs to rely on flow data to make decisions, whether they are ‘close’ to the data or not.
In coming posts we’ll share information on technical challenges and problems we have solved along the way, including some novel perspectives on spectral unmixing, computational gating, cell annotation, and quality monitoring, and begin the narrative of how these differing pieces fit together.
The complexities inherent in modern scientific research are unavoidable. The quality of scientific insights rests on the quality of the data that is closely coupled to the quality and transparency of analytical methods brought to bear. The unintended consequences of overlooking these dependencies can be severe, impacting human health as well as financial and strategic outcomes. Ozette’s commitment to a vertically integrated approach to cytometry, coupled with a focus on transparency and data-driven solutions, offers a pathway to navigate these challenges, ultimately accelerating drug development and ensuring that decisions are based on reliable and robust insights.
