Twitter Data Collection and Extraction: A Method and a New Dataset, the UTD-MI

This paper proposes a Twitter Data collection method using some Python libraries to automatically read and process the raw collected data "Tweets" from the Twitter streaming API, where the returned Tweets are encoded in the lightweight JSON data-format. Using a Python script, the JSON data is parsed into a Python object and transformed into a Pandas DataFrame to help extract the required data fields. Lastly, the Pandas DataFrame is exported to a CSV file for future data analysis tasks, such as the use by some Machine Learning Algorithms. Furthermore, the proposed method was used to automatically collect Tweets and initiate a novel Twitter dataset. The result is a new corpus of Universities-Based Tweets called "UTD-MI" dataset, incorporating real-time Tweets about some selected Universities in Michigan. In addition, some statistical information and visual representations of a sample from the collected corpus is demonstrated.