Human Routine Modelling and Routine Change Detection

May 2018

Yangdi Xu, Dima Damen


Routine, as a term, is used by the general public to refer to the common, regular or standard course of activity patterns. It is tightly linked to temporal regularity. For example, routine morning coffee and evening coffee both contain the same act of making coffee. However, since they regularly happen at a different times of day, they make two different routines. This research focuses on modelling human routine from visual input, as well as detecting not only the presence of routine changes, but also which activity patterns have been changed, in a fully unsupervised manner

Motivated by the researches related to smart home living and elderly care, this work focuses on investigating the indoor routines of a single person using unsupervised Dynamic Bayesian Network. Routine change detection is accordingly modelled as a pairwise model selection problem.

We propose to model the routine using a Dynamic Bayesian Network (DBN). Our DBN avoids using heuristic thresholds and incorporates routine-independent priors to improve performance.



(2018) Y Xu and D Damen. Human Routine Change Detection using Bayesian Modelling. International Conference on Patter Recognition (ICPR). Pre-print

(2017) Y Xu, D Bull, D Damen. Unsupervised Long-Term Routine Modelling using Dynamic Bayesian Networks. IEEE Int Conf on Digital Image Computing Technologies and Applications (DICTA). Preprint

(2015) Y Xu, D Bull, D Damen. Unsupervised Daily Routine Modelling from a Depth Sensor using Bottom-Up and Top-Down Hierarchies. Asian Conference on Pattern Recognition (ACPR).


The dataset that is viable for routine modelling needs to have the following properties:

Office Kitchen:

The office kitchen dataset contains five sets of recordings. the video sequences captured three single individuals in an office kitchen and lasted for 6 days using a single RGB-D prime sense. The camera is set at the entrance of the kitchen. For routine change detection (ICPR2018) we use recordings 1, 2 and 5 captured by the same individual in 2014, 2016 and 2017, with 15 months gap between the pair of recordings

Link: coming soon!

Residential Kitchen:

In the publication above, we also captured a dataset from a multi-sensored house for one week's period. Due to privacy constraints this dataset cannot be recorded. This residential recording is part of the SPHERE dataset. More details on the SPHERE project is available here