Live performance in MR (Mixed Reality)
"ANYCOLOR/niji-sanji New Technology Demonstration Event ver. 2025.01” was an MR-style live event that could be experienced using Meta Quest3 and smartphones, and featured a talk and live music performance by ‘Hayato Kagami,’ a member of niji-sanji.
Press release of “ANYCOLOR/niji-sanji New Technology Demonstration Event ver.2025.01
ANYCOLOR株式会社のプレスリリース(2024年12月16日 12時00分)にじさんじ所属VTuberが出演するMR(複合現実)実証実験の限定イベント「ANYCOLOR/にじさんじ 新技術実証実験イベント ver.2025.01」開催決定!
prtimes.jp
In the past, music live performances by VTubers were mainly experienced by watching images displayed on LED monitors at the venue, but for this event, we developed a system that allows the audience to watch the performance in real time from their own perspective using a head-mounted display or smartphone. The aim was to provide a
new sensation of live entertainment.
System Development
The system for this event was designed to allow VTubers, presenters, and audience members to communicate with each other through low latency data transmission, and to provide a more realistic live experience through the venue's speakers, rather than through earphones, as in a normal live performance. This enables a more realistic live experience.
In addition, in order to realize an experience in which VTubers are more real in front of the audience using MR technology, two systems were developed: a transmission system specialized for VTuber live performances and an MR synthesis system.
In order to enable real-time mutual communication, production signals such as the movements and lighting of the VTubers need to be transmitted to multiple head-mounted displays and smartphones with low latency. For this reason, we have developed a new transmission system that enables MR live in real time based on our original data compression method specialized for VTuber live performances, which we have been jointly developing with STU Inc.
In addition, since compositing highly accurate images in real time has been a challenge, we developed new functions based on the basic functions of Meta Quest3 and developed an MR synthesis system dedicated to VTubers that links physical space, sound, and lighting.
In addition, general venues are not designed to wirelessly transmit and receive data to a large number of terminals, which has been a barrier to providing MR content in real time. To ensure stable transmission of large volumes of data, we redesigned the venue's network infrastructure, selected the network equipment most suitable for real-time MR live performance, and reviewed the data flow, enabling stable data transmission while reducing the burden on operators.
Future Outlook
The development of the real-time MR live system has made it possible to realize an experience that brings VTubers closer to the audience, and to plan and provide content with a greater degree of freedom than ever before.
The technology demonstrated in this event can be applied not only to live performances, but also to various other places such as games, exhibitions, theme parks, and shopping mall event spaces.
We will continue to take on new challenges to deliver “magical, new experiences. We will continue to work on new challenges to deliver “magical, new experiences” to our customers.