Multi-Modal Description of Public Safety Events Using Surveillance and Social Media

A public safety event is a danger and urgent event that need early detection, quick response, and accuracy recover. The efficient method for responding to a happening public safety event is to collect and describe the related data. Besides the surveillance cameras from the physical space, the social media data can also be used to collect and describe the related data of a public safety event. In this work, the proposed method focuses on the step for describing public safety events. Given a public safety event, videos from the surveillance cameras and social messages from social sensors are collected. The multi-modal information including texts, images, videos, and spatial-temporal data is mined to give a description precisely and concisely. First, the social sensors are associated to surveillance cameras by the spatial and temporal information. In the second stage, the social messages are associated to surveillance cameras by the semantic information. In the third stage, the social messages are associated to surveillance cameras by the visual feature. Besides the text, social sensors may upload images or videos. Finally, the multi-modal description step is given based on the three different associations. The experiments on the real data demonstrate the superiority of the proposed framework. Case studies on the real public safety event show the proposed model has good performance and high effectiveness.