Recently, I have been spending some time analyzing real patient data. I’m preparing for a webcast I’ll be giving in which I will walk through the process of replicating the findings of Dr. Gholson Lyon‘s study on the novel disease diagnosis he named Ogden Syndrome.

Being so close to data that comes directly from clinical settings got me thinking about Gholson’s original paper and subsequent editorial in Nature.

The Ogden Syndrome paper was not only a great case study of using bioinformatic filtering to find a causal mutation with deadly consequences, Dr. Gholson Lyon also used it to make a statement.

When I read his editorial in Nature, I thought he was simply proposing we raise the bar on the standards of lab work and informatics done when sequencing patients for research projects.

It turns out he was saying something much more radical with deep implications on what it will take for NGS to really get traction in the clinic.

The Gap Between Research and Clinical Sequencing
A few months ago I was sitting in a split-session basement room at Hotel Kabuki, listening to molecular geneticists, clinical researchers and a representative from the FDA talk at The Clinical Genome Conference.

A debate had threaded through talks and questions to speakers all morning.

• How actionable is an exome or whole genome sequence of a patient?
• How reliable are research results in the clinical context?
• What should a patient be entitled to or even have the option to learn from those hundreds of variants a pathologist had to manually screen to determine if they were putative causal variants or secondary (often called incidental) findings?

But the temperature of the room went up ten degrees when Gholson started his presentation with a scathing reminder of the many failings of clinical medicine to put the patient first, and to keep our bureaucratic rigidities, perverse incentives, and rogue charlatans in check.

It did not take long to see what side of the debate he was on.

He repeatedly referred to the patients as customers, pleading that we move away from paternalistic medicine where the physician hands down information they deem relevant.

But more poignantly, he told the story of the family that approached Gholson with their son he would later diagnose in a research paper with Ogden Syndrome, which he named after the family’s home city in Utah.

He admits that he went into the situation a bit naively thinking he could lead a genetic research study to find a causal variant and report back the findings to the family.

It turns out he could not.

Pedigree of the proband’s family from the Ogden Syndrome paper. The blue represents individuals who were sequenced. The comprehensive study design lead to a very robust and reproducible analysis.

Although he drove out to Ogden, collected blood samples in the capacity of a clinical physician and facilitated the sequencing and subsequent diagnostic and functional validation of the causal variant, the sequencing itself was not done in a manner certified by the Clinical Laboratory Improvement Act (CLIA).

The personal angst and ethical and liability dilemma this put Gholson in is best understood by reading his own thoughts on the experience he posted on Genomes Unzipped. (He is continuing to work with the family and develop a CLIA certified carrier screening test for family members to use going forward.)

But his resolution that he lays out in the Nature editorial is not just about raising standards as I initially thought, but about making a much more demanding ultimatum to the research community. In summary:

It is unethical to perform research on living humans and not give them full access to the genomic data. To make this possible, we should do all research sequencing in CLIA labs so we can legally return results to consumers (patients).

But is a CLIA certified list of variants done with next-generation sequencing all that is needed to make a clinical grade genetic diagnosis?

See a tour of Illumina’s CLIA certified whole genome sequencing lab

Is Research with NGS Sequencing Reproducible?
One of the issues CLIA certification is designed to solve is the accuracy and consistency of clinical tests regardless of which facility it was performed in.

What does that mean for whole genome or whole exome sequencing?

It definitely means something to the actual lab that generates sequence data in terms of the procedures and fail-safes to reduce common problems like label mismatches on samples.

But it also means something to the bioinformatics performed to produce the deliverable of a clinical grade genome sequence.

This turns out to be no small barrier.

Illumina just got its whole genome sequencing lab and pipeline for whole genome sequencing CLIA certified and Complete Genomics is still working on one.

Because reproducibility is such an important aspect of this certification process, it may mean updating the bioinformatic tools and annotations less frequently.

But there is another thing to keep in mind.

The CLIA certification covers just the deliverable of a reproducible set of called, quality filtered variants. You can think of this as a patient’s “genome-type”: it should reproducibly report the variants present in the patient’s genome.

Those variants now need to be annotated, filtered, and interpreted by hand along with the patient’s record to arrive at a diagnosis.

While the variant list produced by a CLIA-certified lab is now capable of being returned to customers (patients), the ultimate utility of the data still rests on that interpretation done by the ordering researcher, genetic pathologist, or lab technician of a diagnostic lab.

23andme’s Exome Sequencing service returns high quality variant lists and aligned sequence data. Customers (patients) are free to crowd-source their own interpretation of their genomic data. Above picture is from the reported list of rare variants in genes implicated in Mendelian disease.

This final set of filters and manual variant inspection is often the crux of the research study or clinical diagnostic process and requires the most hands-on expert attention.

In fact, if all of this rigor goes into making the sequencing and secondary analysis robust and reproducible, it raises an interesting question about the remaining link in the diagnostic chain:

How likely are are two genetic researchers to come to the same candidate causal variant given the same input variant list and clinical patient data?

Next Generation Sequencing in the Clinic: Have We Arrived?
If you follow the daily blurbs of GenomeWeb, you’ll be familiar with the dance many large diagnostic and technology companies are making in trying to time the market of “sequencing in the clinic.” Still, by many individuals’ predictions, we are a couple years away from wide adoption.

But in all this posturing and planning, it’s easy to miss the very important fact: it’s already here.

Just in the past few weeks, I’ve had conversations with a couple directors of clinical labs that either already routinely provide exome or gene-panel sequencing and diagnostic reports as a test or plan to do so in the next six months.

And why not? Research papers showing successes in using sequencing as a “diagnosis of last resort” have been rolling off the press for the last couple years.

Alongside research on undiagnosed disorders, you have reports from oncologists, who by their nature are in a field that is always engaged in research. Clinical oncologists have been publishing success stories on using RNA-seq and DNA-seq tumor/normal paired analysis to discover actionable targets of chemotherapy.

Success Rates versus Success Stories
While the success stories have shown the way and possibly inflated our expectations, it is clear that there is still plenty of work to be done before NGS in the clinic will reach that place of mass adoption and consistent performance dubbed as the “plateau of productivity” in terms of the hype cycle of emerging technologies.

As a meaningful example, Dr. David Goldstein’s group at The Center for Human Genome Variation at Duke recently published on a pilot of exome sequencing of 12 patients who failed to arrive at diagnosis through traditional tests.

The group found a likely genetic diagnosis for six of the cases and a partial diagnosis of another out of the original 12.

The approach used was by no means novel: start with trios for each sample and focus on de Novo mutations, X-linked, and rare recessive variants in genes previously associated in Mendelian diseases.

Still, by Dr. Goldstein’s admission, the process of interpreting the exomes is “a little bit of an art.” The remaining six undiagnosed patients were not without interesting variants, but simply without highly confident variants that made sense with the phenotype of the patients.

As it turns out, patient 2 in the study, with the compound heterozygous mutations in NGLY1 that results in the accumulation of misfolded proteins, was the son of a friend of mine.

Matt Might, a professor at the University of Utah, wrote passionately about the “diagnostic odyssey” he had over three years that ended with the exome sequencing at Duke in his poignantly titled blog post Hunting Down My Son’s Killer.

The post went “viral”, and got over a half million views and reposts in the following weeks.

More importantly, it got Matt in contact with proteomics researchers and on lists for clinical trials to take the knowledge from the diagnostic and move forward towards the development of a treatment for his son.

Since I have known Matt for a while (he was one my thesis advisors), I called him up to hear about the diagnostic process from his perspective.

As usual, that perspective was enlightening, and its contrast to the published paper brought up another good point: our research papers reflect the final-cut version of success stories, rarely detailing all the road bumps and side-tracks taken to get there. Many interesting and potentially useful stories that resulted in dead-ends are never told.

All this points to the many challenges ahead to turn this artform of NGS interpretation into a robust clinical test. These challenges will touch every aspect of our informatic processing, public databases, and interpretation platforms.

It can seem daunting, which is probably why the hype cycle calls this phase the “Trough of Disillusionment”.

Personally, I think these challenges are definitely surmountable.

As a company, tackling some of these challenges is the core of our mission looking forward. And I count on our collaborations and efforts to contribute to the advancement of bringing genomic sciences into standard medical care.

And The Circle is Complete: Championing Reproducibility
While the hypothetical question I posed earlier might have a nuanced answer as there are certainly many ways for two genomic interpretations to diverge with the same input, we can turn to a staple of scientific advancement in a fast-changing world: reproducibility.

This is precisely why Gholson shared the sequencing data from the Ogden study with me.

Although he had already done functional validation that confirms the causal variant and in his own paper used three different bioinformatic pipelines to arrive at the same result, he also champions reproducibility and more open collaboration as a few of the many ways in which the community can advance the progress of NGS in the clinic.

There are still many challenges to overcome for NGS to be ready for mass adoption in the clinic.

Gholson outlines some of these challenges in a recent article in Genomic Medicine. Core to addressing those challenges will be in making our bioinformatic tools and interpretation process more robust and transparent, where we can expect a very high level of reproducibility.

So in my upcoming webcast this Wednesday (register to attend it live), I’ll demonstrate in detail the reproducibility of the Ogden Syndrome finding.

As an example of how robust the Ogden Syndrome analysis is, I will use a completely orthogonal set of bioinformatic steps and secondary analysis methods than what was used in the paper, and unambiguously arrive at the same causal variant that was published and used in the final diagnosis.

NGS is certainly destined for the clinic. Its value and benefit to clinical practice have been demonstrated. Certainly this is an exciting time to be part of the genomics community.