Font Size

HOME > No.29, May 2022 > Walking in a 360° video with foot vibrations for seated observers

Walking in a 360° video with foot vibrations for seated observers

Walking sensations and ground-material perceptions are enhanced by scene-congruent vibrations Junya Nakamura
Junya Nakamura

Walking is a basic human activity that improves human physical and mental health. However, physical disabilities and the COVID-19 pandemic have discouraged people from walking outside. To remove these limitations, a research team at Toyohashi University of Technology in collaboration with researchers at the University of Tokyo, has developed a virtual walking system for seated observers with 360° video and scene-congruent foot vibrations. The timing of foot vibrations was automatically calculated from the 360° video. They found that rhythmical scene-congruent vibrations improved the sensation of walking and the perception of ground materials. This system can convert various public and private 360° videos into realistic virtual walking experiences. This study was published in IEEE Access on 20th December, 2021.

Virtual reality (VR) can provide people with walking disabilities or people during the COVID-19 pandemic with realistic walking experiences. However, most virtual walking systems use custom content that is made for a specific system.

Researchers at Toyohashi University of Technology and the University of Tokyo have proposed a virtual walking system that converts 360° videos into virtual walking experiences and provides scene-congruent vibrations on the feet at the appropriate time.

The virtual walking system.
The virtual walking system.

In the experiments, congruent vibration patterns were prepared for four ground scenes. The modulation of walking-related sensations and the perception of ground materials using congruent and incongruent vibrations was verified using psychological measurements. They found that rhythmic foot vibration improved the sensations of self-movement, walking, leg action, and telepresence irrespective of scene–vibration congruency. Moreover, congruent vibrations were better than incongruent vibrations for walking-related sensations and telepresence in indoor corridors and snowy ground scenes. The perception of ground materials was enhanced by scene-congruent vibrations, whereas it was confused by scene-incongruent vibrations.

Junya Nakamura, a graduate student and the first author of the article, said, "I came up with the idea of using existing 360° videos for our virtual walking system, which could provide a variety of travel experiences to people at home. So, I applied visual SLAM (simultaneous localization and mapping) to videos to estimate the motion trajectories of the viewpoint in the movies, and generated appropriately timed foot vibrations." Professor Michiteru Kitazaki explained, "Another important point of this study is that the congruency of foot vibrations with the ground in scenes improves the perception of ground materials and walking sensations. We can feel soft snowy ground or the hard floor of a corridor. However, we used only four common scenarios and four prepared vibrations in the experiment. We should develop a method for generating congruent vibration patterns from any movie in the future."

Movie demonstrating virtual walking. Scene-congruent and scene-incongruent foot vibrations (represented as sounds in the movie) were applied to the heels and forefeet of users watching 360° videos.

The developed system can convert various public and private 360° videos into realistic virtual walking experiences and provide virtual travel to various locations around the world. It could also contribute to the improvement of the mental health and well-being of people even people with walking disabilities and people who are unable to walk due to social limitations.

This research was supported in part by JST ERATO (JPMJER1701) to MK, JSPS KAKENHI JP18H04118 to YI, and JP20H04489 to MK.


Nakamura, J., Matsuda, Y., Amemiya, T., Ikei, Y., and Kitazaki, M. (2021). Virtual Walking with Omnidirectional Movies and Foot Vibrations: Scene-congruent Vibrations Enhance Walking-related Sensations and Ground Material Perceptions. IEEE Access. DOI: 10.1109/ACCESS.2021.3136557.



中村 純也

歩行は、人の心身の健康を増進させることが期待できます。しかし、身体的な障害やCOVID-19の影響などによって、外を歩くことが困難な場合があります。豊橋技術科学大学の研究チームは、東京大学の研究者と共同で、360度動画とシーンに合わせた足裏への振動を利用した、座っている人のためのバーチャル歩行システムを開発しました。足裏振動のタイミングは360度動画(全天周動画)から自動的に計算されます。実験の結果、リズミカルでシーンに調和した振動は、歩行感覚や地面の材質感を向上させることがわかりました。本システムを用いると、様々な360度動画をリアルなバーチャル歩行体験に変換することができます。本研究は、2021年12月20日にIEEE Access誌に掲載されました。




筆頭著者で大学院生の中村純也氏は、「もし既存の360度動画をバーチャル歩行体験システムに利用することができれば、自宅で様々な旅行体験ができると思いつきました。そこで、自動走行などに利用されているvisual SLAM(Simultaneous Localization and Mapping)を動画に適用して、動画内の視点の動きの軌跡を推定し、適切なタイミングで足の振動を発生させる方法を開発しました。」と話してくれました。北崎充晃教授によると、この研究のもう一つの重要なポイントは、足裏振動がシーンの中で地面と一致することで、地面の質感知覚のみならず歩行感覚も向上することだそうです。これにより、雪のように柔らかい地面や廊下の硬い床などを歩いている感じがします。しかし、実験では一般的な4つのシナリオしか使用しておらず、振動も既存のパターンを利用しています。今後はあらゆる動画から調和した振動パターンを自動生成する方法を開発すべきであるとのことです。


本研究の一部はJST ERATO JPMJER1701(稲見自在化身体プロジェクト)および科研費(JP18H04118、JP20H04489)の補助を受けて実施されました。

Share this story

Researcher Profile

Junya Nakamura
Name Junya Nakamura
Affiliation Department of Computer Science and Engineering
Title Doctor course student
Fields of Research Psychology/ tactile perception/ virtual reality/ virtual walking/ visual perception