In Pursuit of Robots that Act Autonomously to Support Human Activities

Combining technologies to improve the performance of the robot

"I believe that it is the eyes that are the most important part of a robot. In other words, the ability to view (recognize) the surrounding environment. Based on this, the robot should employ its brain to think (plan) about what action to take and move (control itself). These three steps are the most basic functions that allow the robot to perform different tasks in an environment where humans live," says Professor Jun Miura.

In the past, Japan led the world in industrial robots and produced the world’s first humanoid and communication robots. Japan was always ahead of the game in this field. Regrettably however, it must be said that Japan's robot researchers are now outperformed by overseas competitors. One explanation is that Japan has focused on hardware for control whereas the West, in particular, developed robots with a focus on the brain of the robot. Specifically, they focused on software for recognition and planning, or artificial intelligence (AI). In this sense, the professor's research emphasizes recognition. It is cutting-edge research with the goal of increasing the intelligence of robots.

However, he says that even recognition, which is positioned as the first step, has yet to reach a sufficiently practical level. "Today, autonomous cars are reaching the commercialization phase. Robots are now capable of moving freely. However, it is still difficult to ensure their sound operation in an unknown environment. While recognition accuracy surpasses 90% in the research phase, we will still feel hesitant to immediately introduce autonomous cars in the real world. We will have no choice but to work steadily combining multiple technologies to improve the entire system with a view toward achieving that remaining 10%."

As a consequence, the robots he designs rely on more than a single eye. In addition to the RGB camera, he is combining multiple sensors to determine the positional relationship between objects and people from multiple perspectives. These sensors include a far infrared thermometer for measuring the temperature, a laser distance sensor (with light detection and ranging (LiDAR) technology) for measuring distances and shapes, and a GPS system for accurately identifying position. In addition, machine learning and deep learning are used to discover target patterns and features of objects from a large amount of data, and multiple recognition programs are automatically created and utilized in a combination with them. If the robot still lacks the information it needs to make a decision, it will act autonomously to actively obtain the information it requires.

Developing robots that move autonomously

Professor Miura uses the research approach above to develop service robots equipped with sensors for recognizing the environment surrounding them so that they can autonomously move to escort and accompany human users. For example, the robots can go shopping together with elderly people. They can guide them to destinations, carry luggage and act as wheelchairs to bring them back home.
"Robots like these need to accurately recognize the circumstances and operate autonomously. For this purpose, we operate robots under different assumed conditions for a long time to solve problems one by one."

For instance, his team developed a system that integrated three-dimensional LiDAR technology with machine learning to distinguish humans from other objects. In the development process, they paid attention to human traits.
"Take the sensing of a town using LiDAR technology for example. Utility poles look like bars. In contrast, the human silhouette is characterized by a rather small head, wide shoulders and tapering towards the legs. We then sliced the human data horizontally in the direction of height, extracted the feature values in each cross-section, and used a machine learning program to automatically investigate what kind of feature values can be combined to distinguish between people and non-humans. By applying this learning program, we were able to discriminate people from LiDAR information in real time."

They also invented a robot designed to recognize objects using deep learning in a room with chairs, a desk, a television set, apples and other items and to pick up and retrieve the item chosen by the user after measuring the distance to the item using LiDAR technology. They also succeeded in creating a robot with multiple sensors that could follow a specific person without losing them.

"As I said earlier, the key to our research is sensor fusion. The accuracy of identification of a specific person can be massively increased by combining multiple factors such as height, color of clothing and the way they walk instead of focusing on a single factor. That enables the robot to follow the person without missing them in a situation where the image of the person is not very discernible due to backlighting."

Furthermore, he and his fellow researchers are working to develop a robot that leverages sensor fusion to draw maps. It is a robot that autonomously collects data using LiDAR technology to draw a map of the route that it has traveled. It is effective for the exploration of unknown places.

"Another advantage of creating a map is that the current whereabouts of the robot can be visually indicated on the map when it is traveling through a place it has passed through before. We have already created a map of the university campus using three-dimensional LiDAR technology. In the future, we will use this technology to create a robot that can move more freely."

Introducing robotics to the agricultural sector prior to full-scale application in society

Professor Miura's various research achievements are starting to be applied in the agricultural sector. For example, he is using deep learning to analyze information collected by robots in horticultural facilities, and is working to predict the cultivation environment and yield based on this information.

Reporter's Note

Jun Miura loved railways as a child and chose to study mechanical engineering in university. This was at a time when robot research was expanding. Gradually, he became interested in intelligent robots. As a graduate student, he studied computer science.

After getting a job, I began fully studying image recognition under the guidance of Prof. Yoshiaki Shirai at Osaka University. At Carnegie Mellon University, I was hugely inspired by Prof. Takeo Kanade and Prof. Katsushi Ikeuchi. They are all pioneers in computer vision.Then, I realized that robots would be meaningless if they did not move," reminisces the professor. He continues, saying that while moving robots involves far more difficult problems that must be resolved they can do more and support people's lives. We hope to see reliable robots like these become a reality.