Collaborative Annotation Technology in Online University Courses

Overview

In Fall of 2020, the English Department at Indiana University began using the collaborative annotation software, Hypothesis, in all the online freshman reading and writing courses. This implementation provided an opportunity to study how students and teachers were using this technology towards learning.

I led an end-to-end generative research study exploring how university instructors used the Hypothesis collaborative annotation tool in their classroom.

Problem and Research Questions

Collaborative annotation software is not new in higher education, but most of the design and research has been focused on the students as the users. University instructors also use the software and often design how the students use collaborative annotation through instructions and assignments requirements.

I sought understand the user perspective of the instructors. I asked:

How did the English Department university instructors use the collaborative annotation technology, Hypothesis, in their courses?
How did instructors use the technology features?
What problems arose while using the technology?

Challenges

This project was part of my dissertation research and I had to move at a much slower pace than I would have liked due to educational institution requirements and administrative hurdles.

To speed up the impact of the research for Hypothesis, I shared out early results at a conference hosted by Hypothesis called iAnnotate. I also met with organization leaders at Hypothesis to share some of the specific issues that instructors faced.

Methods

I chose to use a focus group approach for collecting user insights and I structured the focus groups similar to a contextual inquiry. There were four focus group sessions that took place over the course of 4 months. The instructors shared their screen and showed how they were using collaborative annotation and how they were asking their students to use it.

I deliberately chose to use focus groups for a few reasons.

First, I wanted the sessions to feel comfortable for the users (instructors) and the focus group format is similar to professional development meetings that they were already familiar with.
Second, conversations among users can lead to naunced and rich insights because users will ask each other questions or build off of each other's responses in ways that may not happen in in-depth interviews.
Finally, focus groups can save time and money by combining multiple individual interviews into one session.

What I Did

As the lead researcher on this project, my responsibilities included:

Proposing study design and methods to the institutional review board to ensure participant data privacy
Designing research protocols such as consent documents, interview and focus group protocols, data collection timeline, etc.
Managing study budget and participant compensation
Facilitating 7 focus group and user testing sessions with 6 instructors who were using collaborative annotation
Collecting and managing research data (50+ course documents, 1500+ annotations, 25 survey responses, over 7 hours of focus group data)
Transcribing focus group data
Analyzing data using qualitative coding (using thematic analysis and content analysis)
Using data collection and analysis software: Microsoft Excel, MaxQDA, Zoom, and Google Forms
Conducting data audits with research team to ensure data analysis is consistent
Presenting results to stakeholders
Writing up research reports

Qualitative coding in MaxQDA

Collecting user testing data via Zoom

Key Insights

Product improvement feedback for software designers:

Instructors wanted a better way to see when students submitted annotations
Instructors wanted ways to create smaller groups of students

Product implementation insights to understand how university instructors used the software:

Instructors often used the annotations to prepare for class. Seeing their students annotations before class helped them identify misconceptions and areas from the annotated text that students missed.
Some instructors used the annotation tool to make sure students read a text before class while other instructors saw the annotations as resources that could be used by students to complete future assignments such as essays.
Some instructors valued that the annotations encouraged students to engage deeply with the text, while other instructors valued that students could have threaded conversations about specific passages in the text.

My Impact

The results of my research on collaborative annotation have led to better implementation of the technology at the university. For example, there were minor adjustments to the training given to instructors using Hypothesis after understanding what different instructors valued about the technology.

My research contributions also provided Hypothesis with additional insights for product improvements for university instructors, including adding the ability to create smaller groups and providing a more accurate submission date for student annotations.

Google Sites

Report abuse