Researchers at The University of Manchester in the U.K. say fixing deficiencies in method reporting of animal experiment results could save hundreds of thousands of pounds and stop commissioning of clinical trials that have no realistic hope of successful result.

A research team led by Dr. Sheena Cruickshank of the UoM Faculty of Life Sciences, and Professor Andy Brass from the School of Computer Science analyzed 58 papers on research into inflammatory bowel disease published between 2000 and 2014. They discovered a wide variety in reporting methodology, and that vital data from experiments were missing, meaning that the studies couldn’t be accurately reproduced in animal or human models, rendering them scientifically useless.

The researchers found that in several instances, gender of the animal experimented on wasn’t recorded — a factor that can have significant bearing on results since female mice have stronger immune response than males. How the animals were housed will also impact on the results in experiments about the gut.

Dr. Cruickshank notes that our immune systems are bombarded with potential infectious threats that they must recognize and respond to by killing or containing the threat. However, the immune system can be harmful too to us too when it isn’t properly controlled, as is seen in allergies or autoimmune diseases like rheumatoid arthritis and Inflammatory Bowel Disease.

Research at Dr. Cruickshank’s lab focuses on how immune responses are initiated. Events that happen early in an immune response determine whether someone will be able to get rid of an infection or not, or whether they get a chronic long-lasting infection or even an autoimmune disease like Inflammatory Bowel Disease. Her research team investigates how immune cells and microbes “talk to each other,” and how this communication shapes immune response in infection, inflammation (e.g., inflammatory bowel disease) and delayed wound-healing.

She notes that worldwide, diseases of the gastrointestinal tract are among the most common human afflictions, but despite this, relatively little is understood about gut immunology — specifically how immune responses are switched on and controlled. A major investigative theme of the Cruickshank lab is understanding and defining the initiation and control of immune responses in mucosal sites such as the gut and more recently the skin. In particular, they focus on the mechanisms of crosstalk between immune cells and epithelial cells, and the research group have developed systems for the isolation and culture of primary intestinal epithelial cells and immune cells.

Dr. Cruickshank says she was shocked at the lack of information provided in papers, noting in a UoM release: “What our research has uncovered is that this lack of data makes it difficult to validate the experiment and the result. Crucially this is having an impact of the reproducibility of experiments, both in the animal model and when transferred to human trials.”

The problem first became clear to the researchers when they began building a knowledge base to help pull together research on inflammatory bowel disease (colitis). Working cooperatively with computer scientists, it became clear to the biologists that study data were proving incomprehensible to colleagues from different disciplines.

Professor Andy Brass observes: “So much research is now being carried out across disciplines so its vital that experiments can be understood by as many scientists as possible. Relying on the reader to make assumptions based on their own experience will only lead to errors, but this is what many papers are asking people to do when they come to replicate the research.”

To address the issue, the research team developed a critical checklist outlining what information should be included. The checklist covers nine areas ranging from general information about the animals, their housing condition, genetics, how colitis is induced, experiment design and monitoring. The checklist is included in a research paper due to be published in the journal Inflammatory Bowel Diseases.

The paper, entitled: “Quality of methods reporting in animal models of colitis“ is coauthored by Bramhall M, Flrez-Vargas O, Stevens R, Brass A, and Cruickshank S., who note that current understanding of the onset of inflammatory bowel diseases (IBD) relies heavily on data derived from animal models of colitis, but the omission of key information concerning methodologies used in research makes the interpretation of some studies difficult or impossible.

The investigators assessed current quality of methods reporting in four animal models of colitis that are used to inform clinical research into IBD: dextran sulfate sodium (DSS), IL-10-/-, CD45RBhigh T cell transfer and 2,4,6-trinitrobenzene sulfonic acid (TNBS). They performed a systematic review based on PRISMA (preferred reporting items for systematic reviews and meta-analyses) guidelines. They used a PubMed search (2000-2014) to obtain publications that used a microarray to describe gene expression in colitic tissue. Methods reporting quality was scored against a checklist of essential and desirable criteria.

In all, 58 papers were identified and included in this review, only one of which reported all essential criteria on their checklist. They found that animal age, gender, housing conditions and mortality/morbidity were all typically poorly reported.

The researchers conclude that this repeated failure to include all essential criteria is cause for concern, since it can have huge impact on the quality and replicability of published colitis experiments. They recommend adoption of their checklist as a base requirement for publication in order to improve quality, comparability and standardization of colitis studies and will make interpretation and translation of data to human disease more reliable.

Dr Cruickshank explains: “Our checklist sounds like fairly basic information that should be in all papers. But over the past few years journals have asked for more and more abbreviated methods so information has stopped being included. Instead papers are focussed on the results and discussion and sometimes you have to go back to a paper from the sixties to find the last time a particular method was accurately recorded.”

While the researchers have been critical of method reporting, they hasten to emphasize that their criticism doesn’t mean the research isn’t valid or accurate. The experiments may well be sound, but the lack of detail in how they’re reported makes that judgement more difficult to make and reproducibility much more difficult.

For example, Dr Cruickshank cites a test case for the Reproducibility Initiative published in the journal PLOS One in December 2014. The team wanted to reproduce an experiment which had found peptide hormones made naturally by cattle to be efficient killers of the parasite Leishmania major. However, when they followed the reported methodology they had to up the dose by ten times to get the same result. But what had actually happened was that the original paper had not precisely described the molecules involved.

Moving forward, the Manchester team believes adoption of their checklist as a requirement for publication will improve quality, comparability and standardization of studies on inflammatory bowel disease, and will make the interpretation and translation of data to human disease more reliable and ultimately contribute to making clinical trials more successful.

The paper “Quality of Methods Reporting in Animal Models of Colitis” will be published in the journal Inflammatory Bowel Diseases.

Sources:
University of Manchester
Inflammatory Bowel Diseases