STEI - Sekolah Teknik Elektro dan Informatika

Enter your keyword

ChatGPT Evaluation (3rd in a series of articles about ChatGPT & AI)

ChatGPT Evaluation (3rd in a series of articles about ChatGPT & AI)

ChatGPT Evaluation (3rd in a series of articles about ChatGPT and AI)

Dear Mr./Ms. GB, regarding the intelligence demonstrated by ChatGPT in various fields that have been hot topics in WAG GB recently, it seems that a fairly comprehensive and quantitative evaluation of ChatGPT is needed, so that we can understand its strengths and weaknesses, and can use it more wisely and carefully, although this is also not easy. I believe that through the refinement of the AI algorithm (language model) and retraining with a more complete and up-to-date dataset, ChatGPT will become even smarter in the future. The following paper, released several months after ChatGPT was launched, is an attempt for the first time to conduct a comprehensive assessment of ChatGPT that touches on aspects of reasoning, hallucinations, and its interactive nature in dialogue. Yesterday I attended a webinar about this paper by one of its co-authors, an Indonesian student. 

https://arxiv.org/pdf/2302.04023.pdf

The paper titled “A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity” evaluates ChatGPT comprehensively and quantitatively using 21 datasets covering several different natural language processing tasks, namely multitask, multilingual, and multimodal aspects. The multitask aspects of NLP include text summarization, language translation (from English to other languages ​​and from other languages ​​to English), sentiment analysis, question and answer, misinformation detection, task-oriented dialogue, etc., with performance varying from  4.1 to 93.3 on a scale of one hundred according to the respective metrics for this task. This paper also found that ChatGPT outperforms other large-scale language models based on zero-shot learning (the ability of a language model to perform tasks for which it is not explicitly trained). ChatGPT has advantages in understanding non-Latin languages ​​(e.g., Korean, Japanese, Chinese) compared to its ability to generate sentences in those languages. ChatGPT is able to generate multimodal content (images) from user instructions through (intermediate) coding steps, for example with HTML canvas. On average, ChatGPT has an accuracy rate of 64.33% in ten (10) types of reasoning tasks, such as logical reasoning (deductive, inductive, abductive), non-textual reasoning (temporal, spatial, mathematical), common sense reasoning, logical reasoning, and causal reasoning. Thus, in terms of reasoning, ChatGPT is considered not yet good enough. In this context, ChatGPT has better deductive reasoning capabilities than inductive ones. Like other large-scale language models, ChatGPT has the weakness of sometimes (quite often?) hallucinating, meaning it provides new facts that are incorrect because it does not have access to an external knowledge base. Furthermore, this paper concludes that ChatGPT has the ability to collaborate with humans/users in turn-based dialogue interactions with an improvement of 8% for text summarization and 2% for language translation. Details of the quantitative evaluation and examples of the dialogues used can be seen in detail in this paper. 

Bandung, 18 Februari, 2023

Bambang Riyanto

  

Post Views: 897

Riwayat Hidup Ringkas Prof. Bambang Riyanto Trilaksono Bambang Riyanto Trilaksono dilahirkan di Banyuwangi, 15 November 1962. Mendapat gelar sarjana di bidang Teknik Elektro, ITB, pada Tahun 1986, dan gelar Master dan Doktor berturut-turut pada Tahun 1991 dan 1994 dalam bidang Teknik Elektro dari Waseda University, Jepang. Saat ini yang bersangkutan adalah Guru Besar di Sekolah Teknik Elektro dan Informatika, ITB, dan Direktur Advanced Robotics Lab. Yang bersangkutan adalah salah satu founder dari Pusat Artificial Intelligence, ITB. Dari Tahun 2015 sd 2020 yang bersangkutan mendapat tugas sebagai Wakil Rektor bidang Riset, Inovasi dan Kemitraan, ITB. Bambang adalah research fellows dari University of New South Wales, Australia, dari Tahun 2012 sampai sekarang. Yang bersangkutan mempublikasikan lebih dari 50 artikel di jurnal internasional terindeks Scopus. Yang bersangkutan adalah mantan ketua dewan redaksi beberapa jurnal ITB yang terindeks di Scopus, dan anggota dewan redaksi Journal of Intelligent Unmanned Systems (Emerald), International Journal of Electrical Eng. and Informatics, Journal of ICT Research and Applications dan Indonesia Internetworking Journal. Bambang adalah founder dari beberapa start-ups, salah satunya dalam bidang AI yaitu Riset Kecerdasan Buatan (www.riset.ai) yang mengkhususkan produk-produknya dalam computer vision dengan menggunakan Artificial Intelligence, dan sudah digunakan di beberapa industri. Yang bersangkutan adalah anggota Tim Strategi Nasional Artificial Intelligence, dan memimpin Tim Taskforce Nasional Artificial Intelligence untuk Deteksi Covid dibawah koordinasi Ristek/BPPT. Bidang peminatan risetnya Artificial Intelligence, Robotics & Control. Yang bersangkutan mendapat penghargaan Toray Science & Technology Award pada Tahun 2004 untuk risetnya mengenai Artificial Intelligence pada peredaman bising aktif, dan pada Tahun 2019 mendapatkan penghargaan sebagai peneliti berprestasi di ITB. Yang bersangkutan merupakan anggota Akademi Ilmu Pengetahuan Indonesia (AIPI) dan IEEE (Institute of Electrical and Electronics Engineer). Yang bersangkutan adalah anggota pendiri Korika (Kolaborasi riset dan inovasi industri Kecerdasan Artifisial). Bambang juga berpengalaman terlibat di industri, khususnya industri dirgantara. Saat ini yang bersangkutan memimpin tim riset dan inovasi pengembangan autonomous vehicle/tram dengan menggunakan Artificial Intelligence, bekerjasama dengan PT INKA dan riset.ai, dengan pendanaan dari Rispro, LPDP.