Performance Comparison Of Large Language Models İn Disaster Related Two-stage Classification Of Tweets Written İn Turkish;

Abstract

Natural disasters are very frequent in Turkiye, therefore it is quite vital to tackle the problems aroused after these disasters. This study proposes a system to reduce the losses caused by the natural disasters and provides a comparison method for the efficient selection of the system components. A database is formed from the tweet samples posted in the aftermath of the previous natural disasters and these tweets are classified in two stages using prompt engineering and large language models. In the first stage, the classification is done based on disaster type such as “earthquake”, “fire” or “flood”, then the tweets in these disaster types are classified for needs such as “search and rescue”, “equipment and food” in the second stage. In order to find the best model for aforementioned classifications, ChatGPT-3.5, fine-tuned ChatGPT-3.5 and ChatGPT-4 are selected and tested. Fine-tuned ChatGPT-3.5 with enhanced prompting is found to have the highest performance with 98.4% average success score for disaster classification. The success rate of the fine-tuned model for classification of needs is calculated as 95.6% in average. This study is expected not only to contribute to the Turkish language processing research area but also to support rescue organisations as well. © 2024 IEEE.