SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding

Abstract

Human-object interaction (HOI), gaze patterns, and their anticipation are intricately linked, providing valuable insights into cognitive processes, intentions, and behavior. However, most existing models handle gaze and actions separately, missing both their interdependence and the advantages of a unified solution. This paper presents SAGE (Synchronized Action and GazE), a unified end-to-end trainable framework that integrates simultaneous recognition and anticipation of both HOI and human gaze. SAGE uses a transformer-based architecture and incorporates gaze into spatiotemporal attention mechanisms to jointly predict current and future human actions and gaze behavior. We study this bidirectional relationship in both egocentric and exocentric settings. To support comprehensive gaze-action analysis in exocentric video, we also establish the ExoCook benchmark. Experiments on VidHOI, EGTEA Gaze+, and ExoCook show consistently strong performance, often surpassing specialized state-of-the-art models designed for individual tasks.

Resources

Citation

@inproceedings{kuang2026sage,
  title     = {SAGE: A Synchronized Action and Gaze Estimation Framework for Comprehensive Human Behavior Analysis},
  author    = {Kuang, Chenyi and Agarwal, Nakul},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}