ERM Tutor-AI logoMaster's thesis · Educational Technology

ERM Tutor-AI

An AI tutor for the Empirical Research Methods course that answers from the course's own materials, shows students where each answer comes from, and talks like a tutor, not a search engine.

Try the live app View code My portfolio
Photo of Amrina
Amrina
Saarland University · Supervisors: Dominik Thüs, Dr. Sarah Malone · LinkedIn
Screenshot of the ERM Tutor-AI chat interface

Results at a glance

20 questions from the weekly ERM assignments, answered by ERM Tutor-AI and by ChatGPT-4o, rated by two course instructors on a 7-point rubric.

6.75 / 7
Mean rubric score for ERM Tutor-AI (ChatGPT-4o: 6.00)
−29%
Shorter answers: 198 vs 280 words on average
p < .001
Effect of the model in a two-way ANOVA; no effect of the instructor

What it does

Large language models can sound confident and still be wrong. In a methods course, a wrong explanation of a p-value or an ANOVA harms learning, so the tutor is built around grounding and transparency.

Grounded in the courseAnswers come first from 12 ERM lecture units, from experimental design to ANOVA, regression and bias in research.
Shows its sourceEvery answer says whether it comes from the course materials (and which unit) or from general research-methods knowledge.
📘 Course Materials: 7. T-test and ANNOVA test
Behaves like a tutorA short empathetic reflection, the explanation, and a follow-up question that keeps the student thinking, in one natural reply.
Stays on topic, stays shortOff-topic questions are politely declined, and each answer shows its word count to keep replies focused.

How it works

Retrieval-Augmented Generation: retrieve the relevant course passages first, then let the model answer from them.

Flowchart: course PDFs are chunked and embedded into a vector database; a question is embedded, relevant chunks are retrieved and passed with the question to the LLM, which returns the answer.
  1. Index the courseLecture PDFs are parsed with Markitdown, split into chunks and embedded with sentence-transformers into ChromaDB.
  2. RetrieveFor each question, the 4 most relevant and diverse chunks are selected (maximal marginal relevance).
  3. GenerateGPT-4o answers with a tutor prompt: course materials first, general ERM knowledge second, decline if off-topic.
  4. PresentA Streamlit chat shows the answer, its source and its length.
PythonStreamlitLangChainChromaDBsentence-transformersOpenAI GPT-4oMarkitdown

Designed in four iterations

A design-based research approach: each version was reviewed and refined for usability, visual identity and pedagogical orientation.

Version 1 interface
1 · Minimal Q&A prototype
Version 2 interface
2 · Clearer structure, refined prompts
Version 3 interface
3 · Branding and welcome message
Version 4 interface
4 · Final, user-centred interface

Evaluation

Rubric adapted from Lee (2024): accuracy, relevance, excessive or insufficient information, user perception, and tutoring behaviour. 40 ratings per system.

Mean rubric score (1–7)
ERM Tutor-AI
6.75 ± 0.44
ChatGPT-4o
6.00 ± 0.76
Mean answer length (words)
ERM Tutor-AI
198
ChatGPT-4o
280

Two-way ANOVA: model F(1, 76) ≈ 32.8, p < .001; no significant effect of instructor or interaction.

Limitations and next steps