{"id":23477,"date":"2024-12-16T01:45:23","date_gmt":"2024-12-15T18:45:23","guid":{"rendered":"https:\/\/stei.itb.ac.id\/?page_id=23477"},"modified":"2024-12-16T01:49:38","modified_gmt":"2024-12-15T18:49:38","slug":"poster12","status":"publish","type":"page","link":"https:\/\/stei.itb.ac.id\/en\/poster12\/","title":{"rendered":"Hardware Accelerator Design of Deep Reinforcement Learning  for Continuous Control in Mobile Robots"},"content":{"rendered":"<div class=\"wpb-content-wrapper\"><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid kepala vc_custom_1734236300248 vc_row-has-fill\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\"><div class=\"container\" ><div class=\"vc_row wpb_row vc_inner vc_row-fluid\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\"><div class=\"vc_btn3-container vc_btn3-inline vc_do_btn\" ><button class=\"vc_general vc_btn3 vc_btn3-size-lg vc_btn3-shape-rounded vc_btn3-style-modern vc_btn3-icon-left vc_btn3-color-white\" onclick=\"history.back()\"><i class=\"vc_btn3-icon fas fa-home\"><\/i> Kembali ke Beranda<\/button><\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid vc_custom_1734170418007\"><div class=\"wpb_column vc_column_container vc_col-sm-3\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<p><strong>Nana Sutisna, S.T., M.T., Ph.D.<\/strong><br \/>\nSchool of Electrical Engineering and Informatics<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><div class=\"wpb_column vc_column_container vc_col-sm-3\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<p><strong>Dr. Eng. Infall Syafalni, S.T., M.Eng.<\/strong><br \/>\nSchool of Electrical Engineering and Informatics<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><div class=\"wpb_column vc_column_container vc_col-sm-3\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<p><strong>Nur Ahmadi, S.T., M.Eng., Ph.D.<\/strong><br \/>\nSchool of Electrical Engineering and Informatics<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><div class=\"wpb_column vc_column_container vc_col-sm-3\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<p><strong>Prof. Trio Adiono, S.T., M.T., Ph.D.<\/strong><br \/>\nSchool of Electrical Engineering and Informatics<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid vc_custom_1734168902943\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<h5><strong>Abstract<\/strong><\/h5>\n<p>Currently, the development of artificial intelligence (AI) is increasing rapidly. With the increasing system complexity such as dynamic environments and large state-action space (continuous state and actions), as well as the requirement for high computing speed, conventional Reinforcement Learning techniques such as Q-Learning face limitations in terms of accuracy and computing speed since the Q-value update process is performed based on comparing the Q-values in every state. To overcome this problem, the Deep Reinforcement Learning (DRL) technique is adopted to perform Q-value calculation and policy optimization. DRL implementation based on the Deep Q-Network (DQN) algorithm has several major issues, specifically: (1) large memory access, (2) different data flows for the Q-value update process and policy optimization, and (3) the need for a flexible DQN architecture to increase the efficiency of resource utilization and computing workload. In this research, a dedicated hardware design (accelerator) has been developed to obtain efficient and optimum performance. To achieve this target, a system is designed with a special hardware architecture (Application Specific Integrated Circuits\/ASIC) by leveraging system-on-chip (SoC) architecture. In addition, the HW\/SW co-design methodology can also be used to allow flexibility and reconfigurability. Furthermore, these features can be used in design space exploration to obtain optimum system parameters quickly. Hardware optimization for low-complexity design is considered by involving several LSI design techniques such as exploiting shared hardware, employing approximate computing, and optimizing the number of layers. These optimizations result in an efficient design and allow the system can be used in edge computing systems, such as mobile robots.<\/p>\n<p><strong>Keyword<\/strong>: Reinforcement Learning, Deep Q-Network, Hardware Accelerator, System on Chip, FPGA<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid vc_custom_1734198597576 vc_row-o-content-bottom vc_row-flex\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<h5><strong>Introduction<\/strong><\/h5>\n<p>Reinforcement Learning (RL) has been applied in various real-world application. However, the implementations face challenges due to the large size of state spaces and complex reward formulas.<\/p>\n<p>To address these challenges, deep Reinforcement Learning (DRL) which integrates RL and deep learning, has gained attraction since it can handle the continuous states and actions. In DRL, a neural network models the agent\u2019s decision-making process, mapping environmental states to actions, and further learns an optimal policy that maximizes expected cumulative rewards. This adaptability allows DRL to excel in dynamic and complex environments.<\/p>\n<p>Implementing DQN in real-world applications, specifically for edge computing targets has main challenges due to resource limitations. The complexity of DQN architectures, particularly in handling neural networks, leads to increased chip area usage while power savings remain modest. Currently, DQN accelerators for edge computing are in their early stages, focusing on more straightforward solutions and resulting in an inefficient performance of accelerator design. To overcome such problems, in this research, we propose a lightweight design capable of training and inference on a compact Processing Element (PE) as well as a scalable design with support of configurable parameters, including the learning rate, discount factor, the number of input and hidden layers, and approximation parameters for the sigmoid activation function. The designed system furthermore is implemented in SoC architecture and employs HW\/SW Co-design to enhance scalability.<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid vc_custom_1734236752660 vc_row-o-content-top vc_row-flex\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<h5><strong>Research Method<\/strong><\/h5>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid vc_custom_1734236752660 vc_row-o-content-top vc_row-flex\"><div class=\"wpb_column vc_column_container vc_col-sm-3\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div  class=\"wpb_single_image wpb_content_element vc_align_center wpb_content_element vc_custom_1734288128913\">\n\t\t\n\t\t<figure class=\"wpb_wrapper vc_figure\">\n\t\t\t<div class=\"vc_single_image-wrapper   vc_box_border_grey\"><img loading=\"lazy\" decoding=\"async\" width=\"384\" height=\"374\" src=\"https:\/\/stei.itb.ac.id\/wp-content\/uploads\/12-2.jpg\" class=\"vc_single_image-img attachment-full\" alt=\"\" title=\"12\" \/><\/div><figcaption class=\"vc_figure-caption\">(1) Deep Q-Network (DQN) Model Design<\/figcaption>\n\t\t<\/figure>\n\t<\/div>\n<\/div><\/div><\/div><div class=\"wpb_column vc_column_container vc_col-sm-3\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div  class=\"wpb_single_image wpb_content_element vc_align_center wpb_content_element vc_custom_1734288152865\">\n\t\t\n\t\t<figure class=\"wpb_wrapper vc_figure\">\n\t\t\t<div class=\"vc_single_image-wrapper   vc_box_border_grey\"><img loading=\"lazy\" decoding=\"async\" width=\"495\" height=\"488\" src=\"https:\/\/stei.itb.ac.id\/wp-content\/uploads\/12b.jpg\" class=\"vc_single_image-img attachment-full\" alt=\"\" title=\"12b\" \/><\/div><figcaption class=\"vc_figure-caption\">(2) SoC Architecture Design<\/figcaption>\n\t\t<\/figure>\n\t<\/div>\n<\/div><\/div><\/div><div class=\"wpb_column vc_column_container vc_col-sm-3\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div  class=\"wpb_single_image wpb_content_element vc_align_center wpb_content_element vc_custom_1734288190941\">\n\t\t\n\t\t<figure class=\"wpb_wrapper vc_figure\">\n\t\t\t<div class=\"vc_single_image-wrapper   vc_box_border_grey\"><img loading=\"lazy\" decoding=\"async\" width=\"612\" height=\"540\" src=\"https:\/\/stei.itb.ac.id\/wp-content\/uploads\/12e.jpg\" class=\"vc_single_image-img attachment-full\" alt=\"\" title=\"12e\" \/><\/div><figcaption class=\"vc_figure-caption\">(3) Architecture Details<\/figcaption>\n\t\t<\/figure>\n\t<\/div>\n<\/div><\/div><\/div><div class=\"wpb_column vc_column_container vc_col-sm-3\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div  class=\"wpb_single_image wpb_content_element vc_align_center wpb_content_element vc_custom_1734288345850\">\n\t\t\n\t\t<figure class=\"wpb_wrapper vc_figure\">\n\t\t\t<div class=\"vc_single_image-wrapper   vc_box_border_grey\"><img loading=\"lazy\" decoding=\"async\" width=\"616\" height=\"212\" src=\"https:\/\/stei.itb.ac.id\/wp-content\/uploads\/Screenshot-2024-12-16-014454.png\" class=\"vc_single_image-img attachment-full\" alt=\"\" title=\"Screenshot 2024-12-16 014454\" \/><\/div><figcaption class=\"vc_figure-caption\">(4) Implementation  and Deployment Scenario<\/figcaption>\n\t\t<\/figure>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid vc_custom_1734236752660 vc_row-o-content-top vc_row-flex\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<p>The proposed architecture has several features, including :<br \/>\n\u2022 Low-complexity hardware by enabling training and inference on a single configurable PE, utilizing shared resources, reducing the number of layers, employing Stochastic Gradient Descent (SGD) and including approximate computing techniques.<br \/>\n\u2022 Highly Scalable design by allowing the configuration of neural network parameters and execution of policy generation, environment, and reward functions performed by software on the Processing System (PS). This can support diverse DQN design parameters and larger environments<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid vc_custom_1734169842459\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<h5><strong>Discussion &amp; Result<\/strong><\/h5>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid\"><div class=\"wpb_column vc_column_container vc_col-sm-4\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div  class=\"wpb_single_image wpb_content_element vc_align_center wpb_content_element vc_custom_1734288460492\">\n\t\t\n\t\t<figure class=\"wpb_wrapper vc_figure\">\n\t\t\t<div class=\"vc_single_image-wrapper   vc_box_border_grey\"><img loading=\"lazy\" decoding=\"async\" width=\"406\" height=\"201\" src=\"https:\/\/stei.itb.ac.id\/wp-content\/uploads\/12c.jpg\" class=\"vc_single_image-img attachment-full\" alt=\"\" title=\"12c\" \/><\/div><figcaption class=\"vc_figure-caption\">(1) System Parameters<\/figcaption>\n\t\t<\/figure>\n\t<\/div>\n<\/div><\/div><\/div><div class=\"wpb_column vc_column_container vc_col-sm-4\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div  class=\"wpb_single_image wpb_content_element vc_align_center wpb_content_element vc_custom_1734288481853\">\n\t\t\n\t\t<figure class=\"wpb_wrapper vc_figure\">\n\t\t\t<div class=\"vc_single_image-wrapper   vc_box_border_grey\"><img loading=\"lazy\" decoding=\"async\" width=\"437\" height=\"321\" src=\"https:\/\/stei.itb.ac.id\/wp-content\/uploads\/12d.jpg\" class=\"vc_single_image-img attachment-full\" alt=\"\" title=\"12d\" \/><\/div><figcaption class=\"vc_figure-caption\">(2) DQN Algorithm Performance<\/figcaption>\n\t\t<\/figure>\n\t<\/div>\n<\/div><\/div><\/div><div class=\"wpb_column vc_column_container vc_col-sm-4\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div  class=\"wpb_single_image wpb_content_element vc_align_center wpb_content_element vc_custom_1734288508341\">\n\t\t\n\t\t<figure class=\"wpb_wrapper vc_figure\">\n\t\t\t<div class=\"vc_single_image-wrapper   vc_box_border_grey\"><img loading=\"lazy\" decoding=\"async\" width=\"603\" height=\"252\" src=\"https:\/\/stei.itb.ac.id\/wp-content\/uploads\/12i.jpg\" class=\"vc_single_image-img attachment-full\" alt=\"\" title=\"12i\" \/><\/div><figcaption class=\"vc_figure-caption\">(3) Performance comparison with state-of-the art design<\/figcaption>\n\t\t<\/figure>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid vc_custom_1734169842459\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<p>\u2022 Shared processing modules minimize chip size and power consumption, achieving an area efficiency of 44.25 IPS\/1LUT and 14.65 ItPS\/LUT<br \/>\n\u2022 The accelerator can achieve a speedup of 671x from the i7 CPU and 1106x from embedded hardware in inference mode, and a speedup of 287x from the i7 CPU and 314x from embedded hardware in training mode.<br \/>\n\u2022 The processing times are 1.43 \u03bcs for inference and 4.31 for training and inference, respectively.<br \/>\n\u2022 The designed DQN accelerator consumes power at just 8 mW at operating clock frequencies of 70 MHz.<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div><div class=\"fullwidth\" ><div class=\"vc_row wpb_row vc_row-fluid vc_custom_1734194817030\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<h5><strong>Conclusion<\/strong><\/h5>\n<p>\u2022 In this work, we have developed an AI accelerator architecture for Deep Q-Network (DQN) that supports the inference\/training process using the same processing element.<br \/>\n\u2022 The proposed DQN accelerator has succeeded in handling a large and dynamic number of states using SoC architecture.<br \/>\n\u2022 Real-time performance can be obtained by achieving an acceleration ratio up to 671 times for inference and 287 for training mode, and at the same time maintaining lower power consumptions for about 8 mW.<br \/>\n\u2022 The designed system has the potential to be deployed in edge platforms and real-world applications, such as path planning\/smart navigation in mobile robots.<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<\/div><\/div><\/div><\/div><\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"Kembali ke Beranda Nana Sutisna, S.T., M.T., Ph.D. Sekolah Teknik Elektro dan Informatika Dr. Eng. Infall Syafalni, S.T., M.Eng. Sekolah Teknik Elektro dan Informatika Nur Ahmadi, S.T., M.Eng., Ph.D. Sekolah [...]","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-23477","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/stei.itb.ac.id\/en\/wp-json\/wp\/v2\/pages\/23477","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stei.itb.ac.id\/en\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/stei.itb.ac.id\/en\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/stei.itb.ac.id\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stei.itb.ac.id\/en\/wp-json\/wp\/v2\/comments?post=23477"}],"version-history":[{"count":5,"href":"https:\/\/stei.itb.ac.id\/en\/wp-json\/wp\/v2\/pages\/23477\/revisions"}],"predecessor-version":[{"id":23489,"href":"https:\/\/stei.itb.ac.id\/en\/wp-json\/wp\/v2\/pages\/23477\/revisions\/23489"}],"wp:attachment":[{"href":"https:\/\/stei.itb.ac.id\/en\/wp-json\/wp\/v2\/media?parent=23477"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}