Font Size

HOME > No.30, Sep. 2022 > Feature Story : In Pursuit of Robots that Act Autonomously to Support Human Activities

In Pursuit of Robots that Act Autonomously to Support Human Activities

Jun Miura

Professor Jun Miura (center) and his lab members.
Professor Jun Miura (center) and his lab members.

Professor Jun Miura seeks to create service robots that recognize their surrounding circumstances, allowing them to act autonomously to support people's lives. They are not humanoid robots that walk on two legs, whose development was led by Japan, nor are they communication robots that talk with a charming voice and have charming facial expressions. The robot that his research team has so far developed moves on wheels and is not designed for looks. This is the result of prioritizing the function of autonomous operation in the complexity of the real world. Professor Miura was interviewed about his research approach and the application of the robot in the agricultural sector.

Interview and report by Madoka Tainaka

Combining technologies to improve the performance of the robot

"I believe that it is the eyes that are the most important part of a robot. In other words, the ability to view (recognize) the surrounding environment. Based on this, the robot should employ its brain to think (plan) about what action to take and move (control itself). These three steps are the most basic functions that allow the robot to perform different tasks in an environment where humans live," says Professor Jun Miura.

In the past, Japan led the world in industrial robots and produced the world’s first humanoid and communication robots. Japan was always ahead of the game in this field. Regrettably however, it must be said that Japan's robot researchers are now outperformed by overseas competitors. One explanation is that Japan has focused on hardware for control whereas the West, in particular, developed robots with a focus on the brain of the robot. Specifically, they focused on software for recognition and planning, or artificial intelligence (AI). In this sense, the professor's research emphasizes recognition. It is cutting-edge research with the goal of increasing the intelligence of robots.

However, he says that even recognition, which is positioned as the first step, has yet to reach a sufficiently practical level. "Today, autonomous cars are reaching the commercialization phase. Robots are now capable of moving freely. However, it is still difficult to ensure their sound operation in an unknown environment. While recognition accuracy surpasses 90% in the research phase, we will still feel hesitant to immediately introduce autonomous cars in the real world. We will have no choice but to work steadily combining multiple technologies to improve the entire system with a view toward achieving that remaining 10%."

As a consequence, the robots he designs rely on more than a single eye. In addition to the RGB camera, he is combining multiple sensors to determine the positional relationship between objects and people from multiple perspectives. These sensors include a far infrared thermometer for measuring the temperature, a laser distance sensor (with light detection and ranging (LiDAR) technology) for measuring distances and shapes, and a GPS system for accurately identifying position. In addition, machine learning and deep learning are used to discover target patterns and features of objects from a large amount of data, and multiple recognition programs are automatically created and utilized in a combination with them. If the robot still lacks the information it needs to make a decision, it will act autonomously to actively obtain the information it requires.

A robot which is designed to ask humans when it is unable to easily make a decision.

"Several existing technologies are used together to interpret a complicated physical space. What I think is particularly significant is an approach called sensor fusion. This means making a comprehensive judgment based on a massive amount of collected data. We are combining this with deep learning and other learning algorithms and active sensing, which is the proactive collection of needed information, to improve the recognition performance of the entire system and to create a robot that operates soundly."

Developing robots that move autonomously

Professor Miura uses the research approach above to develop service robots equipped with sensors for recognizing the environment surrounding them so that they can autonomously move to escort and accompany human users. For example, the robots can go shopping together with elderly people. They can guide them to destinations, carry luggage and act as wheelchairs to bring them back home.
"Robots like these need to accurately recognize the circumstances and operate autonomously. For this purpose, we operate robots under different assumed conditions for a long time to solve problems one by one."

For instance, his team developed a system that integrated three-dimensional LiDAR technology with machine learning to distinguish humans from other objects. In the development process, they paid attention to human traits.
"Take the sensing of a town using LiDAR technology for example. Utility poles look like bars. In contrast, the human silhouette is characterized by a rather small head, wide shoulders and tapering towards the legs. We then sliced the human data horizontally in the direction of height, extracted the feature values in each cross-section, and used a machine learning program to automatically investigate what kind of feature values can be combined to distinguish between people and non-humans. By applying this learning program, we were able to discriminate people from LiDAR information in real time."

They also invented a robot designed to recognize objects using deep learning in a room with chairs, a desk, a television set, apples and other items and to pick up and retrieve the item chosen by the user after measuring the distance to the item using LiDAR technology. They also succeeded in creating a robot with multiple sensors that could follow a specific person without losing them.

"As I said earlier, the key to our research is sensor fusion. The accuracy of identification of a specific person can be massively increased by combining multiple factors such as height, color of clothing and the way they walk instead of focusing on a single factor. That enables the robot to follow the person without missing them in a situation where the image of the person is not very discernible due to backlighting."

Furthermore, he and his fellow researchers are working to develop a robot that leverages sensor fusion to draw maps. It is a robot that autonomously collects data using LiDAR technology to draw a map of the route that it has traveled. It is effective for the exploration of unknown places.

"Another advantage of creating a map is that the current whereabouts of the robot can be visually indicated on the map when it is traveling through a place it has passed through before. We have already created a map of the university campus using three-dimensional LiDAR technology. In the future, we will use this technology to create a robot that can move more freely."

Introducing robotics to the agricultural sector prior to full-scale application in society

Professor Miura's various research achievements are starting to be applied in the agricultural sector. For example, he is using deep learning to analyze information collected by robots in horticultural facilities, and is working to predict the cultivation environment and yield based on this information.

A robot which completely automates the processes from the sorting to the binding of perilla leaves.

He explains, "One robot has already been commercialized. It completely automates the processes from the sorting to the binding of perilla leaves. It was released by Sinfonia Technology Co., Ltd. this past spring. It incorporates an image recognition system based on deep learning to determine the size of leaves and whether they conform to standards. It achieves a level of accuracy that is comparable with highly skilled workers." However he is also willing to admit that "their applications in society are still limited."

To address this issue, he is taking another approach. This is the development of robots that can collaborate with humans.
"One example is a robot we have developed which is designed to ask humans when it is unable to easily make a decision. When it is ordered to fetch a cup from another room, the robot observes the shapes, colors and materials of multiple cups and autonomously figures out a question to get an appropriate answer. If there is more than one cup with a similar color and shape, it will ask the person what material is desired to ensure that it will carry the right cup to them."

In a sense, it may be seen as a way of using humans as a sensor for collecting data. "We will use anything we can use in order to somehow create robots that will work in society." His earnest dream is gradually coming closer to reality.

Reporter's Note

Jun Miura loved railways as a child and chose to study mechanical engineering in university. This was at a time when robot research was expanding. Gradually, he became interested in intelligent robots. As a graduate student, he studied computer science.

After getting a job, I began fully studying image recognition under the guidance of Prof. Yoshiaki Shirai at Osaka University. At Carnegie Mellon University, I was hugely inspired by Prof. Takeo Kanade and Prof. Katsushi Ikeuchi. They are all pioneers in computer vision.Then, I realized that robots would be meaningless if they did not move," reminisces the professor. He continues, saying that while moving robots involves far more difficult problems that must be resolved they can do more and support people's lives. We hope to see reliable robots like these become a reality.

Share this story

Researcher Profile

Jun Miura

Jun Miura

Jun Miura received Dr. Eng. degree in Information Engineering from the University of Tokyo in 1989. From 1989 to 2007, he had been with Department of Mechanical Engineering, Osaka University, first as a Research Associate and later as an Associate Professor. In 2007, he became a Professor of Department of Computer Science and Engineering, Toyohashi University of Technology.

Reporter Profile

Madoka Tainaka

Madoka Tainaka
Editor and writer. Former committee member on the Ministry of Education, Culture, Sports, Science and Technology Council for Science and Technology, Information Science Technology Committee and editor at NII Today, a publication from the National Institute of Informatics. She interviews researchers at universities and businesses, produces content for executives, and also plans, edits, and writes books.