Exploiting Multiple Social Media Sources and Multiple Modalities for Severe Weather Management: The Case Study of the Medicane Ianos

The current paper explored aspects of social media datasets when those were treated for disaster management (DM). The aspects

were accumulated into three basic components: I. simultaneous processing of multiple social media sources. ii. Processing of

multiple modalities iii. Organizing and visualizing output effectively. The case study used was the Medicane Ianos, occurred

during 2020 in the Central and East Mediterranean. Data were collected from 4 social media sources: Instagram, Flickr, X and

YouTube (YT). After scraping the data were merged per modality, resulting to a dataset of 7,058 text strings, 2,949 photos and

168,150 video frames, extracted from 752 YT and Instagram videos.

Sequentially, the processing included four binary classifications of the text strings, in which the effectiveness of LSTM-RNNs

and Transformers was assessed: I. Medicane identification, II. Consequences, III. Disaster Management (DM) Info, IV. Weather.

Mapping the text strings was the third part of the process, and included location entity recognition (LER), geocoding, the use of

conventional geoparsing methods and geographic information systems (GIS) analysis.

Next, photos related to the Medicane Ianos were identified from the 2,949 posted photos and 168,150 video frames. Three deep

learning models: a VGG19, a ResNet101 and an EfficientNetB0 were fine-tuned on Deucalion dataset v1.0 (binary classification)

for that purpose. Moreover a useful RSVI simple index was invented for measuring the related content in the posted videos.

Among the main findings of the research, was that LSTM-RNN was more effective in the majority of the text classification tasks.

DM info was mostly extracted from X texts and less from Flickr captions. Moreover the ResNet101 model with train accuracy 0.95

and validation accuracy 0.93 performed better in both identifying images and video frames. Finally, according to the invented

RSVI index, Instagram videos had greater accumulation of medicane related content, while the analyzed YT videos had more

related volume.