Human Routine Modelling and Routine Change Detection

May 2018

Yangdi Xu, Dima Damen

Routine, as a term, is used by the general public to refer to the common, regular or standard course of activity patterns. It is tightly linked to temporal regularity. For example, routine morning coffee and evening coffee both contain the same act of making coffee. However, since they regularly happen at a different times of day, they make two different routines. This research focuses on modelling human routine from visual input, as well as detecting not only the presence of routine changes, but also which activity patterns have been changed, in a fully unsupervised manner

Motivated by the researches related to smart home living and elderly care, this work focuses on investigating the indoor routines of a single person using unsupervised Dynamic Bayesian Network. Routine change detection is accordingly modelled as a pairwise model selection problem.

We propose to model the routine using a Dynamic Bayesian Network (DBN). Our DBN avoids using heuristic thresholds and incorporates routine-independent priors to improve performance.

Publications:

(2018) Y Xu and D Damen. Human Routine Change Detection using Bayesian Modelling. International Conference on Patter Recognition (ICPR). Pre-print

(2017) Y Xu, D Bull, D Damen. Unsupervised Long-Term Routine Modelling using Dynamic Bayesian Networks. IEEE Int Conf on Digital Image Computing Technologies and Applications (DICTA). Preprint

(2015) Y Xu, D Bull, D Damen. Unsupervised Daily Routine Modelling from a Depth Sensor using Bottom-Up and Top-Down Hierarchies. Asian Conference on Pattern Recognition (ACPR).

Dataset:

The dataset that is viable for routine modelling needs to have the following properties:

• The recorded activities needs to appear frequently to be routines. There are a lot of the dataset that is focus on specific action or activity for the recognition purpose, and their lack of repetitiveness makes them inappropriate for our experiment.
• The data should be a "long-term" recording. The definition of "long-term" here is not about the length of each individual recorded sequence, it is rather about the range of time that the dataset can cover. Since activities can only be considered as routine if they happen frequently in period of weeks or even months. Therefore, the datasets that only focus on activity instances do not meet the requirement.
• The recording should be `unscripted' which means that the participant is free to act according to his own behaviour habit.

Office Kitchen:

The office kitchen dataset contains five sets of recordings. the video sequences captured three single individuals in an office kitchen and lasted for 6 days using a single RGB-D prime sense. The camera is set at the entrance of the kitchen. For routine change detection (ICPR2018) we use recordings 1, 2 and 5 captured by the same individual in 2014, 2016 and 2017, with 15 months gap between the pair of recordings

Link: coming soon!

Residential Kitchen:

In the publication above, we also captured a dataset from a multi-sensored house for one week's period. Due to privacy constraints this dataset cannot be recorded. This residential recording is part of the SPHERE dataset. More details on the SPHERE project is available here