Re-thinking Assessment in the Age of Generative AI
This project develops and tests the first evidence-centred assessment framework for student writing in higher education that explicitly accounts for human–AI collaboration with tools like ChatGPT
Focus Domains
Information Technology & Law
Two contrasting disciplines spanning STEM and humanities writing.
Methodology
Lab & Field Studies
Controlled lab experiments and real-classroom deployments across IT and Law.
🎯 Project Goals & Aims
Our project aims to fill a critical gap in knowledge by developing and testing a novel assessment framework that accounts for the interactions between learners and increasingly sophisticated tools like generative AI.
Core Research Aims
- Develop, refine, and evaluate an evidence-centred framework for assessing GAI-enhanced writing that accounts for complex interactions between learners and tools.
- Deploy human-centred assessment systems for two distinct genres of writing (IT and Law).
- Investigate the affordances and constraints of GAI-enhanced writing practices compared to traditional practices.
- Derive recommendations for coherent assessment of GAI-enhanced writing and the effective use of GAI for writing support.
Expected Benefits & Outcomes
- Deliver the first feasible, valid, and reliable means for the evidence-centred assessment of GAI-enhanced writing.
- Expand theoretical understanding of authentic assessment practices, the cognitive and metacognitive dimensions of writing, and how AI supports or inhibits these dimensions.
- Provide a refinable blueprint for assessment of GAI-enhanced writing in other domains (e.g., programming).
- Generate evidence-based guidance for universities on coherent assessment policies.
⏰ The Challenge of GAI in Assessment
AI Ubiquity vs. Traditional Practices
Generative AI tools are becoming ubiquitous and will be integrated into widely used office software, making their use commonplace. Traditional "pen-and-paper" assessments separate learners from these tools, measuring skills that may not be socially valuable or authentic.
The Evidence Collection Gap
Present assessment designs struggle to account for the interdependence between humans and tools. Existing GAI interactions often occur in separate interfaces, meaning learners can use GAI without providing assessment evidence.
Collaborative Writing vs. Individual Analysis
Writing with GAI is a collaborative, not individual, act. Current methods often emphasise the final product rather than the process, which endangers assessment validity in collaborative contexts and misses opportunities for formative feedback.
💡 Solution: Evidence-Centred Design (ECD) Framework
We will develop an evidence-centred framework that supports student learning through teacher-provided feedback by leveraging our expertise in learning analytics and collaborative interaction modelling.
Student Model
What claims are we making?
Describes a set of claims about participants' learning. Structured around the SR-WMS typology (Self-Regulation in Writing from Multiple Sources), it maps metacognitive, semantic, and rhetorical features of writing and considers which features apply to the author versus the GAI.
Task Model
What activities & environment are used?
Describes the activities participants do and the environment where they take place. We design and implement a GAI-enhanced writing environment (building on the existing Casenotes Writing Tool, CWT) that includes rich text editing, source organisation, and a GAI-powered chatbot.
Evidence Model
What analyses connect data to claims?
Describes the analyses used to relate the data participants produce to claims in the student model. It uses Learning Analytics (trace data, keystrokes, mouse moves) and Epistemic Network Analysis (ENA) to model collaborative processes and provide feedback representations to educators.
🛠️ Methodology & Work Packages
The 36-month roadmap involves four interconnected Work Packages (WPs) that iteratively develop the framework, system, and data analytic techniques.
Project Timeline & Structure (Fig. 1)
CDx: Co-design sessions; LSx: Comparative Lab Studies; FS: Field Study; DEV: Development; LAD: Learning Analytics Development.
WP1: Co-design
Focus: Assessment & feedback design. Three co-design sessions with educators and students (n ≈ 16 per session) will prototype student, task, and feedback models.
WP2: Lab Studies
Focus: Controlled comparisons. Two lab studies (LS1, LS2) with treatment (GAI) and control (non-GAI) conditions (n ≈ 128 each) will compare written product quality and writing processes.
WP3: Field Study
Focus: Real-classroom deployment. The final GAI-enhanced system will be deployed in IT and Law courses (n ≈ 160) to test the ecological validity of the framework.
WP4: Learning Analytics
Focus: Analytics backbone. This includes GAI fine-tuning for IT/Law, training classifiers for SR-WMS processes, automated scoring, and ENA-based models of writing processes.
🧑🔬 Project Team & Impact
World-Class Team & Capability
The team brings productive collaborations across learning analytics, self-regulated learning, assessment, educational technology, and AI.
- Chief Investigators: Prof. Gašević, Dr. Swiecki, Dr. Raković, Dr. Tsai, Dr. Rong, and Assoc. Prof. Nagtzaam.
- International PIs: Prof. Jelena Jovanović and Prof. Sanna Järvelä.
- Institutional context: Active collaboration in the Centre for Learning Analytics at Monash (CoLAM), the largest centre of its kind in the world.
- Oversight: An international Advisory Board including experts in evidence-centred design and quantitative ethnography.
Policy & Societal Impact
Our work aligns with Australia's national priorities and delivers tangible benefits to the education sector.
- National priorities: Contributes to the DISR Artificial Intelligence Action Plan and the Digital Economy Strategy by developing trustworthy methods for assessing human–AI interactions.
- Industry: Project software technologies will be open source, providing blueprints for the EdTech industry to develop next-generation assessment products.
- Policy: Creates and validates at-scale evidence to sharpen and renew higher education policy regarding GAI use.
- Dissemination: Results will be shared through top journals and conferences, and via the Media Centre for Education Research Australia (MCERA).