Developing Visual Information Processing Technology through Human Exploration

Exploring human brains and functions for coexisting with machines

Prof. Nakauchi became interested in computers when he was a middle school student and the computer boom was just beginning. He entered a technical college (KOSEN) hoping to research information engineering, but eventually turned his attention to biology. This was because he was driven by an interest in testing the limitations of machines and finding out how they differ from humans as living beings. For Prof. Nakauchi the key to these questions was to focus on learning about human beings.

Prof. Nakauchi is still inspired by these questions. The current debate in this field focuses on the possibilities thrown up by the third AI boom that the singularity (a hypothetical technological singularity when AI supersedes the human brain) may be within reach. In this context, Prof. Nakauchi believes that researchers should be focusing on the differences between machines and humans and in particular how machines and humans may communicate with each other.

"Computers are able to compute much more data than the human brain at much faster speeds, but generic AIs that supersede human ability are not likely to be developed for some time. On the other hand, humans are gradually adapting to a society that utilizes a plethora of machines. As one example, the Internet now serves as an external memory device for humans, meaning that we no longer need to remember maps or unfamiliar words. This causes the brain and society to change with it.

The trend towards ever greater human-machine coexistence appears unstoppable. In order to prepare for such a future, my goal is to uncover how to achieve optimal human-machine relations by conducting an in-depth study of humans."

How do humans distinguish between "textures"?

Broadly speaking, Prof. Nakauchi's research can be divided into two categories. The first is the basic groundwork of "scientifically studying visual perception" with the aim of uncovering visual function. The second involves consolidating the findings obtained through this basic research and seeking technological applications, in other words "applying visual perception as technology".

Recently, in particular, this basic research has produced a string of ground-breaking discoveries, the results of which have been published in international journals1. One such study, relating to "distinguishing between textures" using visual perception, uncovered what humans use to distinguish between "reflective materials" such as mirrors and polished metals which produce a reflective effect through light reflecting on the surface and "transparent materials" such as glass and ice through which light passes and refracts.

According to Prof. Nakauchi, "humans have good eyes and can discriminate slight differences between various materials. For example, humans can tell that a desk which looks like wood is actually made of plywood, or whether or not leather is real. The same applies to glass and metals, with humans being able to distinguish between these materials even if they have complicated shapes. However, light reflects only on the surface of metal materials and is partially absorbed by and scattered in glass materials, meaning that these materials have no color of their own and merely distort and reflect whatever is around them. Up until recently, it wasn't clear what humans used to distinguish between the two."

Prof. Nakauchi and his team generated images using computer graphics that looked like either metal or glass and had participants say which material they thought the picture looked like. The team noted that the participants found it easier to discern the material when the object rotated rather than when they viewed a still image. Through controlling the shape and movement components of the objects to see how differently people perceived them, they achieved truly fascinating results.

"We found that, with metals, light reflection moves in the same direction as the rotation of the object, but glass reflects transmitted light and has some component in which light reflection moves in a direction opposite to rotation of the object. We also discovered that information on movement is not just used for distinguishing between materials but also gives materials that real feeling of being metallically shiny or transparent."

Interestingly enough, Prof. Nakauchi stated that humans tend to treat translucent materials such as marble or jade as important and expensive. This is why many bottles used for expensive cosmetic products are translucent.

"If we can further investigate what humans use as clues to distinguish between textures, such findings could be useful for future product development," suggests Prof. Nagauchi regarding possible outcomes of his study.

Humans subconsciously determine "face-likeness"

Studies concerning the face are also of great interest. People not only have the ability to determine that a person is still the same person despite a change in expression, hair style or clothing but can also determine "face-likeness" in car bonnets and paintings made up of objects, such as Arcimboldo’s portraits using fruit and vegetables. Because we need to be able to recognize faces for communication within society, humans are believed to be very good at discerning faces.

In order to investigate the visual information processing that takes place when a human looks at a face-like object, Prof. Nakauchi and his team measured brain activity and brainwaves when a subject was shown an object that looked like a face and studied the stage at which face-likeness recognition occurs2. All studies used black and white images which were pictures of human faces, as well as cars, bugs, paintings by Arcimboldo, and other objects. Participants were also shown upside-down images.

"The results of the brainwave measurement indicated that face-likeness is first processed at a very early stage of around 100ms, a face is detected at around 170ms, and expressions and individuals are identified at 250ms, meaning that facial recognition occurs in three stages. In addition, when a face is rotated, facial recognition slows to the same speed as when a face-like portrait is viewed, which tells us that processing patterns vary depending on how much an object looks like a face."

In the initial stage of the first 100ms after viewing an object the processing appears to be instinctive rather than being consciously controlled. In other words, humans almost instantaneously take note of face-likeness, even before they are aware of what the object is.

Prof. Nakauchi explains that, "We believe that face-likeness recognition is most likely an inborn function, which can be seen in how a baby needs to recognize the face of its mother to survive."

Does inspiration occur before we are aware of it?

Prof. Nakauchi also studies the "science of inspiration," which is a subject unique to his team. Through studying the time at which inspiration occurs by measuring pupils, which are said to represent a person’s mental state, the team made the remarkable finding in tests that the participants’ pupils began to dilate before they were consciously aware of particular thoughts3.

"We showed participants 6-second videos of black and white dots that moved and changed density to depict various images, like a coffee cup. The video was stopped at 1.5 seconds and participants were asked whether or not they recognized the object. Then, the video was resumed and played until the end and participants were asked the same question again.

The people who answered "no" when asked the question at the 1.5 second mark provided us very interesting results. Those who answered "no" then but went on to figure out the object after watching the whole video presented pupil dilation before the one second mark. In other words, their brain had begun the process of piecing together memories even though they were not aware of this. Meanwhile, the participants who failed to recognize the object even after viewing the entire video presented little pupil dilation."

People often experience times where they can picture someone’s face but their name remains on the tip of the tongue. This kind of phenomenon may be one instance of knowing the answer but not being conscious of that knowledge.

Prof. Nakauchi says that, "applying these findings may make it possible in the future to measure how much people understand by measuring their pupils or to support memory retrieval by externally controlling pupil dilation." Future applied research is also expected to yield exciting results.

Reporter's Note

Now in his 50s, Prof. Nakauchi wants to focus on the themes that really touch his heart for the remainder of his career as a researcher. Wanting to ask the simple but big questions that have largely been neglected by science, Prof. Nakauchi now turns his attention to studying how perceptions such as preference, novelty and intimacy, which affect human behavior and decision making, are generated. "Studying aesthetics falls under this category. Why do we feel that pieces of art by famous artists are good? How do we spot a fake? It’s quite a mystery, isn’t it?

Also, since I was in a brass band as a student and am now a member of a choir for my local opera, I’m also interested in finding out the state of the brain when humans are in synch or have fostered a sense of unity when playing music or a team sport together. Figuring this out should definitely prove to be useful for coexisting with robots as well."

Evidently, Prof. Nakauchi still has a lot of interesting topics to sink his teeth into along his way to finding out what it means to be human.