Belhan C.Fikirdanis D.Cimen O.Pasinli P.Akgun Z.Yayci Z.O.Türkan, Mehmet2023-06-162023-06-1620219.78E+12https://doi.org/10.1109/ASYU52992.2021.9599016https://hdl.handle.net/20.500.14365/3510IEEE SMC Society;IEEE Turkey Section2021 Innovations in Intelligent Systems and Applications Conference, ASYU 2021 -- 6 October 2021 through 8 October 2021 -- 174400Lip reading, described as extracting speech data from the observable deeds in the face, particularly the jaws, lips, tongue and teeth, is a very challenging task. It is indeed a beneficial skill that helps people to comprehend and interpret the content of other people's speech, when it is not sufficient to recognize either audio or expression. Even experts require a certain level of experience and need an understanding of visual expressions to interpret spoken words. However, this may not be efficient enough for the process. Nowadays, lip sequences can be converted into expressive words and phrases with the aid of computers. Thus, the usage of neural networks (NNs) is increased rapidly in this field. The main contribution of this study is to use Short-Time Fourier Transformed (STFT) audio data as an extra image input and employing 3D Convolutional NNs (CNNs) for feature extraction. This generates features considering the change in consecutive frames and makes use of visual and auditory data together with the attributes from the image. After testing several experimental scenarios, it turns out to be the proposed method has a strong promise for further development in this research domain. © 2021 IEEE.eninfo:eu-repo/semantics/closedAccess3D convolutional neural networkaudio-visual speech recognitionautomatic speech recognitionLip readingshort-time Fourier TransformConvolutionSpeechSpeech recognition3d convolutional neural networkAudiovisual speech recognitionAutomatic speech recognitionConvolutional neural networkFourierLip readingNeural-networksShort time Fourier transformsSpeech dataSpoken wordsConvolutional neural networksConference Object10.1109/ASYU52992.2021.95990162-s2.0-85123175238