Deep-learning method can predict daily human activities

Researchers develop method enabling computers to 'see' and understand what humans do in a typical day

Researchers from the School of Interactive Computing and the Institute for Robotics and Intelligent Machines have developed a technique that teaches computers to 'see' and understand what humans do in a typical day.

The technique gathered more than 40,000 pictures taken every 30 to 60 seconds over a six month period by a wearable camera and predicted with 83% accuracy what activity that person was doing.

Researchers taught the computer to categorise images across 19 activity classes. The test subject wearing the camera could review and annotate the photos at the end of each day to ensure that they were correctly categorised.

The group believes they have gathered the largest annotated dataset of first-person images to demonstrate that deep-learning can understand human behaviour and the habits of a specific person.

“This work is about developing a better way to understand people's activities, and building systems that can recognise people's activities at a finely-grained level of detail,” said Edison Thomaz, co-author and graduate research assistant in the School of Interactive Computing.

Thomaz added: “Activity tracking devices like the Fitbit can tell how many steps you take per day, but imagine being able to track all of your activities – not just physical activities like walking and running. This work is moving toward full activity intelligence. At a technical level, we are showing that it's becoming possible for computer vision techniques alone to be used for this.”

The ability to see and recognise human activities has implications in a number of areas – from developing improved personal assistant applications like Siri to helping researchers explain links between health and behaviour. In the future, the technology has the potential to provide industrial and manufacturing benefits for workers such as increased productivity.

Daniel Casto, a Ph.D. candidate in Computer Science and a lead researcher on the project, said: "Methods such as the one we have developed could enable workers in industrial or manufacturing settings to better understand the activities they are performing every day, how much time they dedicate to each, and an overall visualisation of the work that they are performing. For instance, a new worker could look at the workflow of a more experienced worker to get a better understanding of how to effectively use their time and ways in which it could be improved.

“I will note that these scenarios are speculative since we have not conducted any work specific to the manufacturing and industrial area – nonetheless, the ability to automatically recognise how we use our time and to understand our daily schedule is likely to uncover ways in which we can be more productive.”