Multimodal Fusion Algorithm and Reinforcement Learning-Based Dialogue System in Human-Machine Interaction

Peneliti: Hanif Fakhrurroja, Carmadi Machbub, Ary Setijadi Prihatmanto, Ayu Purwarianti

This study develops a human-machine interaction system method. It involves several stages, including a multimodal activation system, methods for recognizing speech modalities, gestures, face detection and skeleton tracking, a multimodal fusion strategy, human intention understanding and an Indonesian dialogue system, and a method for developing machine knowledge and appropriate responses. This research contributes to a more user-friendly and natural human-machine interaction system using a multimodal fusion-based system. The average accuracy rates of multimodal activation, dialogue system testing using Indonesian gesture recognition interaction, and multimodal fusion were 87.42%, 92.11%, 93.46%, and 93%, respectively. The user satisfaction rate for the developed multimodal recognition-based human-machine interaction system was 95%. According to 76.2% of users, this interaction system felt natural, while 79.4% agreed that the machine responded well to their wishes.