Video-based Contrastive Learning on Decision Trees: From Action Recognition to Autism Diagnosis

Mindi Ruan, Xiangxu Yu, Na Zhang, Chuanbo Hu, Shuo Wang, Xin Li

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

How can we teach a computer to recognize 10,000 different actions? Deep learning has evolved from supervised and unsupervised to self-supervised approaches. In this paper, we present a new contrastive learning-based framework for decision tree-based classification of actions, including human-human interactions (HHI) and human-object interactions (HOI). The key idea is to translate the original multi-class action recognition into a series of binary classification tasks on a pre-constructed decision tree. Under the new framework of contrastive learning, we present the design of an interaction adjacent matrix (IAM) with skeleton graphs as the backbone for modeling various action-related attributes such as periodicity and symmetry. Through the construction of various pretext tasks, we obtain a series of binary classification nodes on the decision tree that can be combined to support higher-level recognition tasks. Experimental justification for the potential of our approach in real-world applications ranges from interaction recognition to symmetry detection. In particular, we have demonstrated the promising performance of video-based autism spectrum disorder (ASD) diagnosis on the CalTech interview video database.

Original languageEnglish
Title of host publicationMMSys 2023 - Proceedings of the 14th ACM Multimedia Systems Conference
PublisherAssociation for Computing Machinery, Inc
Pages289-300
Number of pages12
ISBN (Electronic)9798400701481
DOIs
StatePublished - Jun 7 2023
Event14th ACM Multimedia Systems Conference, MMSys 2023 - Vancouver, Canada
Duration: Jun 7 2023Jun 10 2023

Publication series

NameMMSys 2023 - Proceedings of the 14th ACM Multimedia Systems Conference

Conference

Conference14th ACM Multimedia Systems Conference, MMSys 2023
Country/TerritoryCanada
CityVancouver
Period06/7/2306/10/23

Keywords

  • autism diagnosis
  • decision trees
  • graph convolutional network
  • interaction adjacency matrix
  • interaction modeling
  • skeleton graphs

Fingerprint

Dive into the research topics of 'Video-based Contrastive Learning on Decision Trees: From Action Recognition to Autism Diagnosis'. Together they form a unique fingerprint.

Cite this