Spatio-Temporal Transformer Block with a Skeletal Temporal Transformer for Human Activity Recognition

Most of the recent methods for Skeleton-Based Human Action Recognition rely on Graph Neural Networks (GCNs) based on the depiction of human skeleton joint points by graphs, where the bone joints are represented as nodes. GCNbased methods often underestimate the interrelation between body parts that are not adjacent. Recent efforts rely on an efficient combination of GCN and a transformer for accurately modeling the interrelationships between body parts. However, the fixed graph structure of GCNs limits the efficacy of the method to extract valuable features. To address this limitation, we replace the GCN module from a transformer model where GCN was used as a feature extractor, with a spatio-temporal transformer block (ST-TR), to make use of the pixel and channel attention mechanisms to extract spatio temporal features. The proposed ST-TR block, due to the unified spatial and temporal structure, reduces the computational redundancy, thus providing a better GFLOPs measure compared to the existing methods based on transformer and GCNs. Experiments on NTU-60 and NTU-120 datasets show the efficacy of our method compared to the state-of-the-art, both according to accuracy and the GFLOPs measures. The code for the proposed method will be available after acceptance.