Learning Analytics and Game-based Assessment in Serious Games

Every decision a player makes in a serious game leaves a record. Metrics like “# of attempts before a solution is found” or “# of hints requested” both describe how that player is thinking, and well-designed games turn those records into evidence of learning. Game-based assessment uses that evidence to estimate what a learner knows without pausing play for a quiz, and learning analytics gives instructors and designers a view across every player at once. The evidence for this approach is growing through expanding research like the study described below from Florida State University, where a stealth assessment in Physics Playground produced competency estimates that tracked students' scores on an external physics test. Read on for four interesting examples that demonstrate how your next serious game can measure learning as players play.

A Decade of Research Maps the Field of Game-based Assessment

Researchers Fan Su and Di Zou reviewed 21 empirical studies of game-based assessment published in SSCI journals over the past decade, reporting their findings in Knowledge Management & E-Learning. Evidence-centered design was the most frequently cited framework, while nearly half of the studies named no theoretical framework at all. Across the studies, games captured process data such as time on task, attempts, restarts, and decision sequences, and the authors recommend balancing these in-game measures with external assessments. A named assessment framework gives your team a defensible way to connect each logged action to the competency it measures.

Stealth Assessment Measures Physics Learning Without Interrupting Play

Valerie Shute and colleagues at Florida State University studied 263 students in grades 9-11 who played Physics Playground for about four hours, publishing their results in the Journal of Computer Assisted Learning. A Bayesian network updated estimates of each student's physics understanding in real time from gameplay logs, and those estimates correlated significantly with students' scores on an external physics posttest. Students who played improved significantly from pretest to posttest while the control group showed no change, and players rated their enjoyment at 3.90 on a five-point scale. Measurement and enjoyment held up together, which shows that rigorous assessment can live inside a game that students want to keep playing.

Mission HydroSci Builds Analytics Into the Design From the Start

Wenyi Lu and colleagues at the University of Missouri designed a learning analytics system into Mission HydroSci, a serious game that teaches middle school water science, and reported their work in the Journal of Learning Analytics. The team generated around 60 in-game features from the gameplay telemetry of 373 students and identified several statistically significant features as the foundation for a predictive model. The system's purpose is to give instructors timely signals about student progress during the unit. Planning your telemetry alongside your mechanics ensures the game records the evidence instructors need from the first day of play.

Articoding Shows How Analytics Can Guide Game Development

Antonio Calvo-Morata and colleagues at Complutense University of Madrid tracked 134 high school students playing Articoding, a block programming game for ages 12-16, and published the case study in the journal Computers. The game logged 98,704 interaction traces through a serious games extension of the xAPI standard, and simple flags for repeated attempts and inactivity identified students who struggled with variables. The same data showed that players rarely used hints and that two data type levels demanded unusual effort, which led the team to plan revisions to the hint system and level sequence. Analytics serve the design team as much as the instructor, since every playtest becomes a source of evidence for the next build.

~

Game-based assessment and learning analytics require a game to capture the right evidence, and which is generally defined during the design phase alongside your core mechanics. We covered the design side of embedded assessment in How Educational Game Designers Test Players Without Breaking Immersion, and the integration side in Behold, the LTI Standard: LMS Integration for EdTech Developers. Together with today's research, they describe a complete path from a player's first action to a data point in your learning management system. Ready to build a serious game that measures what your learners know as they play? Let's talk.

‍