StepNet: Spatial-temporal Part-aware Network for Sign Language Recognition

Authors: Xiaolong Shen, Zhedong Zheng, Yi Yang

Published in ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2024

Recommended citation: Xiaolong Shen, Zhedong Zheng, Yi Yang, "StepNet: Spatial-temporal Part-aware Network for Sign Language Recognition." ACM TOMM, 2024. DOI: 10.1145/3656046
Download PDF: https://zdzheng.xyz/files/2024/TOMM-Xiaolong-sign.pdf

Abstract: The goal of sign language recognition (SLR) is to help those who are hard of hearing or deaf communicate with other people more conveniently. Most existing works formulate the SLR task as a classification problem, which is limited to a fixed vocabulary and lacks the ability to handle out-of-vocabulary signs. In this article, we address the continuous sign language recognition (CSLR) task and propose a spatial-temporal part-aware network, termed StepNet, which extracts part-level features and captures the sequential dependencies of sign language. In particular, we first adopt a pretrained pose estimator to obtain the human body keypoints, which provide strong prior knowledge about the body structure and part locations. Then, a part-aware transformer encoder is designed to model the spatial relations among different parts, and a temporal transformer decoder is introduced to capture the long-term temporal dependencies. Moreover, we propose a novel part-aware attention mechanism that adaptively selects the informative parts at each time step, suppressing the interference of unrelated parts. Extensive experiments on two large-scale CSLR benchmarks, i.e., RWTH-PHOENIX-Weather 2014 and RWTH-PHOENIX-Weather 2014T, demonstrate that StepNet achieves state-of-the-art performance while maintaining high efficiency.

@article{shen2024stepnet,
author = "Shen, Xiaolong and Zheng, Zhedong and Yang, Yi",
title = "StepNet: Spatial-temporal Part-aware Network for Sign Language Recognition",
abstract = "The goal of sign language recognition (SLR) is to help those who are hard of hearing or deaf communicate with other people more conveniently. Most existing works formulate the SLR task as a classification problem, which is limited to a fixed vocabulary and lacks the ability to handle out-of-vocabulary signs. In this article, we address the continuous sign language recognition (CSLR) task and propose a spatial-temporal part-aware network, termed StepNet, which extracts part-level features and captures the sequential dependencies of sign language. In particular, we first adopt a pretrained pose estimator to obtain the human body keypoints, which provide strong prior knowledge about the body structure and part locations. Then, a part-aware transformer encoder is designed to model the spatial relations among different parts, and a temporal transformer decoder is introduced to capture the long-term temporal dependencies. Moreover, we propose a novel part-aware attention mechanism that adaptively selects the informative parts at each time step, suppressing the interference of unrelated parts. Extensive experiments on two large-scale CSLR benchmarks, i.e., RWTH-PHOENIX-Weather 2014 and RWTH-PHOENIX-Weather 2014T, demonstrate that StepNet achieves state-of-the-art performance while maintaining high efficiency.",
journal = "ACM TOMM",
url = "https://zdzheng.xyz/files/2024/TOMM-Xiaolong-sign.pdf",
doi = "10.1145/3656046",
year = "2024" }