ChromaScribe - AI-based Qualitative Data Analysis Tool
RECENT ACHIEVEMENT
Puranik, A., Chen, E., Peiris, R. L., & Kong, H.-K. (2025). Not a Collaborator or a Supervisor, but an Assistant: Striking the Balance Between Efficiency and Ownership in AI-incorporated Qualitative Data Analysis. https://doi.org/10.48550/arXiv.2509.18297
UX Research Study
(Funded by NSF)
Overview
Qualitative research is a multifaceted process involving data collection, transcription, organization, coding, and thematic analysis. As qualitative data analysis (QDA) tools gain popularity, artificial intelligence (AI) is increasingly integrated to automate aspects of this workflow. This study investigates researchers’ QDA practices, their experiences with existing tools, and their perspectives on AI involvement in qualitative research, bias reduction, usage of multimodal data and their preferences among human-initiated coding, AI-initiated coding and traditional coding.
I conducted in-depth interviews with 16 qualitative researchers, via Zoom. This also involved thematic coding, data analysis and usability tests for our AI-based QDA tool - ChromaScribe. Lastly, I recommend 3 design features for AI-based QDA tools, to improve trust in AI, transparency in theme generation and improved team collaboration.
Timeline
1.5 years
My Role
UX Researcher
Usability Testing
Team
1 UX Researcher
1 UX Designer
1 Backend Developer
1 Frontend developer
2 RIT HCI Professors
Tools
Atlas.ti
Zoom
Otter.ai
Qualtrics
Research Questions
RQ 1.a. - How do researchers perceive the effectiveness of current qualitative coding tools and the impact on their qualitative analysis practices?
RQ 1.b. - How effectively does ChromaScribe align with participants' desired features and address limitations commonly identified in existing QDA tools?
RQ 2 - How do researchers perceive human-AI collaboration in qualitative analysis and its effectiveness in mitigating bias?
RQ 3 - To what extent do researchers utilize multimodal data in their qualitative data analysis?
Research Insights
Trust in AI-assisted coding
Participants appreciated AI’s efficiency but expressed skepticism due to its lack of contextual understanding and transparency. Trust in AI was conditional, with many seeking explanations behind AI-generated codes before relying on them.
Ownership of codes
Researchers preferred traditional and AI-initiated coding methods where they retained final control. They emphasized that coding is an interpretive act central to their identity as researchers, and delegating this to AI felt like a loss of ownership.
Collaboration as a key to reduce bias
Most participants agreed that bias is best mitigated through collaboration—either with other researchers or AI tools used as supportive “second eyes.” However, they stressed that AI alone is not sufficient for bias reduction without human judgment.
Limited usage of multimodal data
Most participants agreed that bias is best mitigated through collaboration—either with other researchers or AI tools used as supportive “second eyes.” However, they stressed that AI alone is not sufficient for bias reduction without human judgment.
Challenges in current QDA tools
Participants cited steep learning curves, lack of collaboration features, and concerns over data privacy in existing tools. They often resorted to general-purpose tools like Google Docs or Miro for their flexibility, despite lacking advanced analytical support.
How did I do it?
Participant recruitment
We conducted a formative qualitative study using semi-structured interviews to explore researchers' perceptions of AI-assisted qualitative data analysis. 16 participants were recruited through university mailing lists, LinkedIn, and flyers. All participants had at least one year of experience in qualitative research and prior use of QDA tools.
Data collection: Interviews and tool testing
The study involved three phases: a pre-study survey (demographics and experience), a 90 minute interview, and a post-study survey. During the interview, participants were introduced to our AI-based prototype ChromaScribe, watched a demo video, explored the tool hands-on, and completed four structured tasks to assess usability and feasibility.
Prototype description: ChromaScribe
ChromaScribe supports AI-generated thematic coding, audio-transcript sync, pitch visualization and demographic-based filters. Participants engaged with features like search, filtering, and color-coded data visualizations to simulate real-world QDA workflows.
Interview setup and Recording
Interviews were conducted over Zoom (14) and in-person (2), with recordings captured via Zoom Cloud or iOS Voice Memos. All recordings were transcribed using Zoom and Otter.ai and manually cleaned in Google Docs to ensure accuracy before analysis.
Data analysis
Transcripts were coded using Atlas.ti. Thematic analysis was conducted through inductive coding. One researcher developed the initial codebook and another coded a subset independently. Emerging themes included trust, ownership, collaboration, tool limitations, and multimodal data use.
Consent
The study was approved by our university’s Institutional Review Board (IRB). All participants gave informed consent and received a $30 gift card for their participation.
Participants' feedback on ChromaScribe
Participants found AI-generated codes helpful for jumpstarting their analysis. Still, they desired more transparency on how the themes were generated to feel confident using them.
The color-coded visualizations helped users easily spot patterns across the transcript.
Participants appreciated the tool’s search and filter functionalities, which helped them quickly locate specific themes or participant segments.
Opportunity
How can we design AI-based QDA tools to improve trust in AI, increased transparency in automated theme generation and enhance team collaboration?
Design recommendations for AI-based QDA tools
Explainable AI Coding with Interactive Justifications
To build trust, AI-generated codes should be accompanied by clear, interactive justifications. For example, when a user hovers over a code, the tool should highlight the exact transcript excerpts that influenced that code and display a short rationale, helping researchers understand why the AI made a particular decision.
Human-in-the-Loop Coding Validation System
To support researcher ownership and interpretive control, the tool should adopt a human-in-the-loop model where AI suggestions remain editable and must be explicitly confirmed by the user. A dual-pane interface—one showing AI suggestions, the other for human codes—can allow easy comparison and validation before codes are finalized.
Collaborative Coding Interface with Role-Based Access
Participants expressed a strong need for teamwork in analysis. The tool should offer real-time collaboration features such as role-based access (e.g., reviewer, coder), comment threads on codes, and live updates of coding progress. This not only enhances collaboration but also supports bias mitigation by incorporating diverse perspectives into the analysis process.
Key Takeaways
Human oversight is irreplaceable
While participants acknowledged AI’s value in streamlining repetitive tasks, they emphasized the importance of retaining control over the final analysis to preserve accuracy, context, and a sense of ownership.
Trust in AI depends on transparency and control
Participants were more willing to use AI when they could see how codes were generated and had the ability to validate or modify them. Future tools must prioritize explainability and human-in-the-loop design.
Collaboration—not automation—reduces bias
Reducing bias was less about replacing humans with AI and more about supporting collaborative workflows. Participants preferred using AI as a secondary “check,” especially when human collaborators were unavailable.
Multimodal data holds untapped potential
Despite acknowledging the value of audio and video data for richer analysis, most participants defaulted to text due to lack of supportive tools. Future QDA systems should simplify multimodal integration to improve context and reduce interpretive gaps.
What did I learn?
Follow-up questions are where the gold is
The most revealing insights came not from the initial questions, but from thoughtful follow-ups. I learned how to listen actively, sense hesitation or excitement, and then dig deeper to uncover the “why” behind participants’ behaviors and opinions.
Piloting helps refine everything
Conducting pilot interviews taught me how essential it is to test everything—from timing to clarity of tasks to the technical setup. This step helped me make the sessions smoother and more meaningful for participants.
Analyzing qualitative data requires patience
Conducting pilot interviews taught me how essential it is to test everything—from timing to clarity of tasks to the technical setup. This step helped me make the sessions smoother and more meaningful for participants.
Participant recruitment takes strategy
Coding 16 interviews line-by-line made me realize how labor-intensive but rewarding qualitative analysis can be. I learned to create a reliable codebook, navigate subjectivity in interpretation, and distill rich, messy data into meaningful patterns and themes.