VR streaming: live green-screen video and web input inside VR headsets
ScalaCode built a platform that streams green-screen video and a live camera feed into VR headsets over WebRTC. It adds two-way audio and video and a pointer that lets people on the web interact inside the VR scene. The work ran in two phases, from a technical assessment and proof of concept to the full feature set.
A media client bringing live video into VR
The client wanted live video inside VR. That meant streaming a green-screen feed and a Raspberry Pi camera to headsets, adding two-way audio and video, letting web users upload data that appears in VR, and keeping the platform on the latest Meta SDK.
What made live VR streaming hard
Green screen in real time
The green-screen environment had to stream into VR in real time without breaks.
Many audio and video channels
Video and audio over several channels at once had to perform well at scale.
From 2D to 3D
Clicks made on a flat web video had to become actions inside a 3D VR space.
Keeping up with the Meta SDK
The platform had to stay compatible with the latest Meta SDK.
Consistent quality
Video quality had to hold up across many different devices.
Unity, WebRTC and a Raspberry Pi camera feeding VR headsets
Unity renders the scene, WebRTC carries the streams, and a Node.js backend runs the APIs.
- 01
Green screen integration
We rendered the green-screen feed in Unity and streamed it to the headsets over WebRTC, after testing the green-screen setup and the WebRTC server logic.
- 02
Raspberry Pi camera
A Raspberry Pi streams its camera to the WebRTC server, and the green-screen and camera streams are merged into one output for VR.
- 03
Audio and video
Audio and video calls run on a single channel or on separate ones. R&D on call quality, and moving to the Meta SDK, improved performance.
- 04
Data upload into VR
APIs take data uploaded from the web, store it securely in the cloud and render it inside VR, where users can interact with it.
- 05
Platform updates
We moved the platform to the latest Meta SDK and fixed the bugs that stood in the way of the new features.
- 06
Interactive VR pointer
Input from the web-streamed video reaches VR in real time, and a 3D pointer maps each 2D action into the VR space.
Live video and web input, inside the headset
Headset users see the green-screen and camera feeds merged in real time, talk with others over two-way audio and video, and work with data uploaded from the web. When a web user points at something in the video, the pointer shows up in the right place in VR.
proof of concept to full build
streams to VR headsets
What VR streaming platform runs on
Unity, WebRTC, Meta SDK
Node.js, custom APIs
Raspberry Pi
Secure cloud storage
Plan a build like VR streaming platform for your business
- 30 minutes with the ScalaCode team
- What is realistic, what it costs and how long it takes
- You decide afterwards whether to build with us
