SLVideo: A Sign Language Video Moment Retrieval Framework

ACM Multimedia 2024
Gonçalo Vinagre Martins, Afonso Quinaz, Carla Viegas, Sofia Cavaco, João Magalhães,
Universidade NOVA de Lisboa
Paper arXiv Code Demo

Abstract

Sign Language Recognition has been studied and developed throughout the years to help the deaf and hard-of-hearing people in their day-to-day lives. These technologies leverage manual sign recognition algorithms, however, most of them lack the recognition of facial expressions, which are also an essential part of Sign Language as they allow the speaker to add expressiveness to their dialogue or even change the meaning of certain manual signs.

SLVideo is a video moment retrieval software for Sign Language videos with a focus on both hands and facial signs. The system extracts embedding representations for the hand and face signs from video frames to capture the language signs in full. This will then allow the user to search for a specific sign language video segment with text queries, or to search by similar sign language videos. To test this system, a collection of five hours of annotated Sign Language videos is used as the dataset, and the initial results are promising in a zero-shot setting.

SLVideo is shown to not only address the problem of searching sign language videos, but also supports a Sign Language thesaurus with a search by similarity technique.

Model Architecture

Model Architecture

Examples

All these examples were produced using the clip-ViT-B-32 model


Searches

Query "Primeiro" with Frames Embeddings

Example Search
Example Search
Example Search

Query "Esconder" with Average Frames Embeddings

Example Search
Example Search
Example Search

Query "Rir" with Best Frame Embedding

Example Search
Example Search
Example Search

Query "Ter" with Annotations Embeddings

Example Search
Example Search
Example Search

Thesaurus

Example Thesaurus
Example Thesaurus
Example Thesaurus
Example Annotation Edition

Annotations Edition

Example Annotation Edition
Example Annotation Edition
Example Annotation Edition
Example Annotation Edition

Results

Automatic Evaluation

Results for searching for the words "Primeiro" (first), "Rir" (laugh) and "Ter" (have) using the three techniques of frame embedding-based search, the three methods of processing the extracted frames and the annotations embeddings search. The metric used in these results is Recall, measuring the proportion of relevant video segments that were retrieved.

BibTeX

@misc{martins2024slvideosignlanguagevideo,
      title={SLVideo: A Sign Language Video Moment Retrieval Framework}, 
      author={Gonçalo Vinagre Martins and Afonso Quinaz and Carla Viegas and Sofia Cavaco and João Magalhães},
      year={2024},
      eprint={2407.15668},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.15668}, 
}