Another lesson in genomics experimental design and avoiding batch effects

Twitter has been abuzz with Orna Man and Yoav Gilad’s (re)analysis of the data from a recent PNAS paper: “Comparison of the transcriptional landscapes between human and mouse tissues”.

The PNAS paper concluded that the gene expression profiles of different tissues within the same species were more similar than the same tissue across different species. For example, their analysis showed that the gene expression profile of a mouse spleen was more similar to that of a mouse heart than it was to a human spleen. This didn’t seem to mesh with well previous research, so it didn’t make much sense. Well, that is until Man and Gilad decided to probe it a bit further.

Once Man and Gilad got their hands on the data (hooray for data transparency!) they noticed something that quickly raised their eyebrows: the human and mouse tissues were sequenced in separate lanes and on separate runs! In other words, the inter-species comparison they were testing was completely confounded with a well-known batch effect: sequencing run (and lane).

Sequencing lane (a batch effect) was almost completely confounded with species in the PNAS study. From @Y_Gilad

Next, they controlled for the batch effect of sequencing run (and lane). Unsurprisingly they found that gene expression profiles clustered by tissue first, and then by species (see figure below).

After controlling for the batch effects, gene expression clusters by tissue first, not species. From @Y_Gilad

So, what gives? Does this mean that there results of the PNAS paper are completely bogus? Not necessarily. When Man and Gilad removed the batch effects they could have also removed some inter-species effects. Unfortunately, there’s no way to disentangle the two given the poor experimental design. The only way to do so would be to redo the experiment and balance the tissues and species across each sequencing lane (or, even better, just multiplex them all in the same pool and spread it across multiple lanes). We’ll just have to wait and see if someone else, or the authors, decide to generate new data from a more thoughtfully designed experiment.

Retraction watch!
There have been a lot of calls for the authors to retract their original paper given the flawed experimental design. I’m not going to weigh in on this, but I will direct you to a thoughtful blog post from Arjun Raj about dealing with “retraction in the age of computation”.

Lin S, Lin Y, Nery JR, Urich MA, Breschi A, Davis CA, Dobin A, Zaleski C, Beer MA, Chapman WC, Gingeras TR, Ecker JR & Snyder MP (2014) Comparison of the transcriptional landscapes between human and mouse tissues. Proc. Natl. Acad. Sci. 111, 201413624.

Man O & Gilad Y (2015). I’m calling your experimental design out on Twitter. Twitter.


About Noah Snyder-Mackler

I'm a postdoctoral fellow in the department of Evolutionary Anthropology at Duke University. Broadly, I study non-human primate genetics and genomics. More specifically, I'm interested in the interaction between behavior, genotype, and gene expression in response to social stress.
This entry was posted in bioinformatics, genomics, next generation sequencing, RNAseq and tagged , , . Bookmark the permalink.