Sign Language Translation System

Abstract

Sign language translation systems play a crucial role in eliminating communication barriers between Deaf and Hard-of-Hearing (DHH) individuals and those who can hear. These systems are complex and consist of multiple components. Among them, the automatic translation of spoken language into sign language, known as Sign Language Production (SLP), holds the potential to revolutionize sign language communication applications. Contrary to its importance and necessity, research in SLP still lacks depth, with only a few models publicly available with insufficient evaluations and comparisons. Consequently, the comprehension of SLP experiments remains obscure, with few new studies emerging in this domain. This project endeavors to investigate the efficacy of existing public SLP methods on American Sign Language (ASL). Specifically, the experiment involved training and evaluating three distinct approaches (Regressive Training with Progressive Transformers, Adversarial Training with Progressive Transformers, and Non-Autoregressive Transformers with Gaussian Space) using the How2Sign dataset, one of the most comprehensive datasets comprising instructional videos in American Sign Language. Back-translation evaluation metrics were employed to assess the performance of these methods in translating discrete spoken language sentences into continuous 3D sign pose sequences. The results indicate that, for the complex data involved in translating from spoken language to sign language in SLP, Non-Autoregressive Transformers with Gaussian Space (NSLP-G) outperform other methods, accurately capturing both manual and non-manual features with minimal errors. Additionally, with the Progressive Transformers model, the effectiveness of adversarial compared to sole regressive training in translate from text to sign language field is observed. Within the scope of this thesis project, a Minimum Viable Product (MVP) was developed to test real-time text-to-sign language translation. The project’s outcomes can provide valuable insights for future researchers, guiding them towards viable approaches in exploring this field or considering practical applications. The report also outlines the limitations of this project and proposes future work that could be utilized to further develop and improve sign language production models.