At Retraction Watch, there’s a story of a paper about ethics training retracted due to IRB human subjects protocol problems. We tend to think of human subjects research as involving things like drug trials, but a lot of it is things like this:
This was an IRB-approved paper-pencil study investigating how certain features of ethics case studies influence knowledge and application of case study principles to new ethical scenarios.
In other words, if I’m understanding the post and the excerpts, the investigators were studying what works and what doesn’t work for how to teach higher levels in Bloom’s taxonomy in the field of teaching about ethics. So what went wrong?
Several administrative issues influenced our Institutional Review Board’s decision to not allow the data from this study to be used for research purposes. One of these had to do with the fact that some of the course instructors were not listed as key study personnel and they handed out and collected the study materials and informed consent forms. Even though they did not have any other involvement in the study, we recognized this oversight. Additionally, we implemented two minor changes to study materials, including dropping two items and renumbering 8 items, and did not obtain re-approval for these changes. Lastly, through this review process, we became aware that roughly half of the informed consent forms (ICFs) were not on file. Although we kept a clear record of who consented and who did not through the use of a training checklist, we recognized this was a data storage lapse. We worked with our IRB to fix these problems and have better processes in place to prevent similar issues from occurring in the future. Although the senior editor for the journal did not think that these issues warranted retraction of our paper, our university’s decision that we could not share the data publicly influenced our decision to voluntarily retract the article.
I understand the basis for the IRB/Human subjects rules, but this case illustrates a problem for all of us as educators. Every time I teach any course, I am doing an experiment in what works. It’s usually an uncontrolled experiment with unconscionably bad record-keeping, but it’s still an experiment and my students are subjects. As there is an increased emphasis on doing more rigorous assessment of student learning outcomes, we will try to be more rigorous about gathering and analyzing data from assessment instruments, which can be done without IRB approval as long as it’s not for publication. Consider how different that is from a drug trial!
As I understand it, doing a proper human subjects protocol for assessment of teaching requires outlining ahead of time all the interactions the assessor will have with the students/subjects. Deviation from that protocol (analogous to changing an FDA approved drug manufacturing protocol) is a no-no – as in the case above. This makes it almost impossible for the actual instructors to also do the assessment… and creates a barrier to dissemination of interesting STEM teaching methods being tried by scientists who don’t have the right collaborators for assessment.