Hypothesis annotation tool — dissertation research

Context

In 2020, the English Department at Indiana University began using Hypothesis in all online freshman reading and writing courses. I led an end-to-end generative research study with two connected lines of inquiry: what kinds of learning conversations were students actually having through annotations, and how were instructors — as the users designing those experiences — actually using the tool?

Most research and design around collaborative annotation had focused on students. Instructors were shaping how students used the software through assignment design, instructions, and facilitation choices, but their perspective as users had been largely overlooked. Going in, it wasn’t obvious that instructors were a distinct user group worth studying in their own right. The clearest thing the research did was establish that they were, and that their needs didn’t show up anywhere in student annotation data.

The work was part of my dissertation, but the research questions and methods mapped closely to product research: a clearly defined user group, generative inquiry into their workflows, and findings shared directly with the Hypothesis team.

What I did

I structured the research around four focus group sessions over four months. Instructors shared their screens and walked through how they were using Hypothesis — closer to contextual inquiry than a traditional academic interview. The format was deliberate: focus groups felt familiar to instructors as a professional development setting, and conversations between participants surfaced nuances that one-on-one interviews might not.

Across the study I collected and analyzed 1,500+ student annotations from 50+ course documents, conducted 7 sessions with 6 instructors, gathered 25 survey responses, and coded over 7 hours of focus group data using thematic and content analysis.

With that volume of data, synthesis was the hard part, and the study’s slower academic pace gave me room to do it properly rather than settling on the first themes that felt right. I worked iteratively: draft an initial reading of what the data was saying, take it to colleagues to pressure-test, revise, and often go back for another round. I also brought preliminary findings back to the focus group participants themselves (member checking), which led to minor but real revisions where my interpretation didn’t quite match their intent. The part that took the most iteration was the instructor themes: getting past what instructors said they did with the tool to an accurate account of the job they were actually trying to get done.

I shared early findings at iAnnotate, a conference hosted by Hypothesis, and shared my findings with Hypothesis showing some of the specific pain points instructors reported. The research generated three articles on social annotation and strategies for instructors implementing it in online courses.

Outcome

The research surfaced two categories of findings:

Product improvement feedback — instructors wanted a clearer way to see when students had submitted annotations, and the ability to create smaller student groups. Both became product changes: Hypothesis later added Canvas group support and improved annotation submission visibility.

Implementation insights — instructors used annotations in notably different ways. Some used them to prep for class discussions, spotting misconceptions before students came in. Others used them as pre-reading accountability. Some valued threaded conversation; others valued deep engagement with specific passages. Understanding that variety helped the university calibrate instructor onboarding after the study.

Reflection

The methods here — focus groups structured like contextual inquiry, direct stakeholder briefings, findings translated into product recommendations — are the same ones I use in industry research. What the dissertation context added was the pace: slower by institutional necessity, which forced me to develop patience with long-horizon data and to make claims I could genuinely defend rather than conclusions that felt right early.

Most research on collaborative annotation was studying what students did with it — their behavior, their learning. But instructors had a different relationship with the tool entirely: they were making deliberate choices about how to deploy it, and they had specific jobs they were trying to accomplish (e.g., Jobs to Be Done) that didn’t show up in student annotation data at all. Understanding their goals was a different research question.

I find that pattern almost everywhere now. There’s usually a layer of users whose job is to make the product work for someone else — instructors, managers, admins — and studying the downstream user’s behavior doesn’t tell you much about what those people actually need. The Hypothesis study was where I first had to ask that question directly.