With the release of consumer-grade virtual reality headsets like the Oculus Rift and HTC Vive, VR is finally becoming a mainstream technology. But how does it work? This article will explain the key software and hardware components that make up a VR system.
What is virtual reality and how does VR work?
Virtual reality (VR) aims to give the user an immersive experience of a virtual simulated environment. The simulated world may be much like the real world, or completely different. VR systems typically employ hardware to track the user’s position and movement, as well as displays near the eyes to display a simulated 3D environment.
Virtual reality applications include entertainment (video games are very popular), education (such as medical, vocational, or first responder training) and business (e.g. virtual meeting rooms). AR (Augmented Reality) and MR (Mixed Reality) are related but unique types of technology from VR. They are sometimes collectively called extended reality or XR, but the vocabulary is changing fast due to how new the industry is.
Currently, most VR systems use VR headsets to create realistic imagery, sounds and sometimes touch or smell to simulate a user being located in a virtual environment. Special rooms with environments projected on screens surrounding the users are another possibility but are a less common and less scalable approach.
With VR equipment simulating a VR world, the VR users can explore and interact with their surroundings as if they were really there.
When you put on a VR headset, the simulation effect is created by an HMD or head-mounted display with small screens directly ahead of your eyes. Virtual reality generally includes both audio and video-simulated elements, but can also incorporate other types of feedback such as force-sensitive haptic gloves, vests, or body suits.
Key hardware components of a VR system
There are various ways to implement VR, but certain techniques are more common and widely accessible as of the early 2020’s, so we’ll focus on some of the most popular current technologies. For example, it’s possible to configure an entire room with screens on every wall to create a virtual environment. But for most settings, a headset worn by the user is a much easier way to achieve the immersive experience that VR aims for.
VR headsets (a.k.a. Head Mounted Displays or HMDs)
A head-mounted device completely immerses the user in a virtual world by covering their eyes. A virtual reality headset can be worn by a user and is typically strapped to the user’s face by wrapping around their head. Typical VR headsets consist of several interconnected components that we will cover individually below.
Display panels for each eye
There are typically two small display panels with either an OLED or LCD screen that has been designed for close up viewing. The resolution of these screens is much higher than that of a typical smartphone or television, which is required to provide the level of immersion required for VR. The current crop of VR headsets also use what is known as ‘stereoscopic 3D’, which means that each eye sees a slightly different image. This is what gives VR its sense of depth and immersion.
Head tracking
Another important component of VR is head tracking. This is what allows you to look around the virtual world and have that movement replicated in the headset. Head tracking is usually accomplished with some combination of an accelerometer, gyroscope, and magnetometer. These tracking sensors monitor the rotational movement of your head and translate it into movement in the virtual world.
Hand controllers or other input devices
VR headsets also often include handheld VR controllers or another form of input device to allow you to interact with the virtual world. This could be something as simple as a button on the headset itself or a dedicated hand controller that you hold in each hand. The HTC Vive, for example, comes with two motion controllers that let you interact with the virtual world in a variety of ways.
Other common input device options include:
- Handheld VR controllers with motion tracking
- Haptic gloves that track movements of your hand including fingers
- Treadmills that allow you to walk in any direction while tracking the amount of movement and replicating it in the virtual environment
Audio (stereo speakers or headphones)
The audio in VR is also crucial for creating an immersive experience. Most VR headsets on the market today use binaural audio, which means that there are two separate audio channels feeding into each ear. This helps to create a sense of directional sound, so you can hear things coming from different directions just like in real life.
CPU and GPU
Like other computers, VR systems require a CPU and graphics processing unit to render the virtual world. But the requirements for VR are much higher than for traditional gaming or other applications. The CPU and GPU need to be powerful enough to render two high-resolution images at a minimum of 60 frames per second (fps). To avoid motion sickness, the system needs to be able to understand a user’s head movement, and then adjust the displayed virtual environment without a significant lag. Achieving this type of refresh rate requires significant processing power, so the most realistic VR environments can usually only be achieved on high end devices.
Tethered vs. standalone VR hardware
VR systems can be either standalone or tethered to a computer. Standalone VR systems like the Oculus Quest are self-contained and don’t require a separate PC or smartphone. Tethered VR systems, on the other hand, need to be connected to a powerful computer or gaming console in order to function.
The main advantage of standalone VR is that it’s more portable and doesn’t require any additional hardware. The downside is that standalone VR systems are generally not as powerful as tethered systems and can’t provide the same level of immersion.
Key software components of a VR system
While the hardware is certainly important, it’s the software that really makes VR tick. The software is what takes the input from the various hardware components and translates it into a realistic and immersive virtual world.
Rendering engines
A rendering engine is a piece of software that is responsible for generating the 2D or 3D images that are displayed to the user. This is usually done using a technique known as ‘rasterization’, which converts the 3D models and world data into a series of 2D images (or ‘frames’).
The frames are then sent to the display device, such as a VR headset, at a rate of around 60 frames per second. This gives the illusion of movement and creates a realistic and immersive VR experience.
There are a number of different rendering engines available, each with its own strengths and weaknesses. Some of the more popular ones include Unreal Engine 4, Unity 3D, and CryEngine.
Making simulated people look realistic
One of the most difficult software challenges in VR is known as the ‘uncanny valley’. This is the phenomenon whereby a computer-generated character looks and acts almost like a real human, but not quite. This can often create a feeling of unease or even revulsion in the user.
To combat this, VR developers often use a technique known as ‘facial mapping’. This involves taking a 3D scan of a real person’s face and then mapping it onto a computer-generated character. This gives the character extremely realistic facial expressions and helps to avoid the uncanny valley effect.
One company that is doing some cutting-edge work in this area is 8i. Their technology allows them to capture the 3D likeness of real people and then render them in high fidelity VR. This gives users the ability to interact with virtual characters that look and feel like real people.
Worldbuilding
Another important aspect of VR software is known as ‘worldbuilding’. This is the process of creating a virtual world that is believable and immersive. This often involves using real-world data to create an accurate and realistic virtual environment.
For example, if you were creating a VR experience set in New York City, you could use real world data to create an accurate 3D model of the city. This would include things like the layout of the streets, the height of the buildings, and so on.
Once the basic structure of the world has been created, it can then be populated with virtual objects and characters. This can be anything from furniture in a room to trees in a forest.
The worldbuilding process is often iterative, meaning that it is constantly being tweaked and improved as the VR experience is developed further.
How VR headsets create the immersive virtual environment
Building on the components above, VR headsets create an immersive environment by doing the following:
- Blocking the user’s view of the real world
- Replacing the user’s view of the physical world view with a software generated view of a simulated environment. The simulated view is delivered by high resolution displays placed near the user’s eyes, with a slightly different view shown to each eye to mimic the different positions of eyes in the physical world.
- Tracking the user’s head movement via a special tracking system, and adjusting the simulated environment to correspond to the user’s head angle and position
- Using headphones or speakers to provide an immersive stereo sound environment
- Using other devices to capture inputs from the user’s body movement. This could be anything from a simple pointing device, to a full body tracking suit. These VR input conto
Different types of VR headsets available today
If you’re interested in trying out VR, there are a few different ways to get started. The Oculus Rift and HTC Vive are both available for purchase right now. If you don’t want to spend the money on a high-end headset, there are a number of cheaper options available, like the Samsung Gear VR and Google Daydream. Finally, if you have a PlayStation 4, you can buy the Sony PlayStation VR headset to use with it.
There are a number of different software platforms that developers can use to create VR experiences. The two most popular ones right now are Facebook’s Oculus Rift and the HTC Vive. These platforms offer developers access to high-powered computers and a variety of different input devices that can be used to create VR experiences.
Other popular VR platforms include the Samsung Gear VR, Google Daydream, and Sony PlayStation VR. These platforms are all powered by mobile phones or dedicated gaming consoles. While they don’t offer the same power as a PC-based platform like the Oculus Rift or HTC Vive, they are much more affordable and easier to set up.
No matter which platform you choose, VR is an incredibly exciting technology that is sure to change the way we interact with computers and the world around us
How positional and rotational tracking works in VR systems
Positional and rotational tracking for VR are typically achieved by one or more of the following. VR systems typically use a combination of these tracking technologies to provide the best possible tracking accuracy.
Inertial Measurement Units (IMUs)
These are small, self-contained tracking devices that use a combination of accelerometers and gyroscopes to track movement. IMUs are typically located in the headset, controllers, and any other tracked objects in the VR environment.
Magnetic trackers
These use magnetic sensors to track the position and orientation of objects in relation to a central reference point. Magnetic trackers are often used in combination with IMUs, as they can provide more accurate tracking data than IMUs alone.
Optical trackers
These use cameras to track the position and orientation of objects relative to a central reference point. Optical trackers are often used in combination with IMUs and magnetic trackers to provide the most accurate tracking data possible.
The benefits and potential applications of virtual reality technology
Virtual reality technology is still in its infancy, but it has already shown a lot of promise. The most obvious benefit of VR is the ability to provide users with an immersive experience that can transport them to another place. This has a wide range of potential applications, from video games and movies to education and training.
VR also has the potential to be used for more than just entertainment. For example, it could be used for education and training purposes. VR training could be used to provide students with a realistic and immersive learning experience. It could also be used to train soldiers or first responders in realistic virtual environments.
There is so much potential for how VR can be used that we’ve created a separate article focused on practical VR applications.
Conclusions and How to Learn More
VR technology is developing quickly. Many aspects of the VR devices that we see today were built on innovations that were first implemented in the mobile phone industry. Now that VR headsets are becoming more popular and widespread, VR software developers will get a lot more data to work with that will help them iterate and improve VR content even faster.
If you want to learn more about the technical aspects of VR, understanding a bit about how human vision works is a good first step. You may want to brush up on how the human brain combines signals from each eye and blends it to create a combined view of the environment. The human eye is pretty amazing, but to make virtual reality work well you also need to track the eye and brain into treating virtual worlds and digital objects as if they were real. There is a lot of really interesting science to unpack here and we wish you the best on your journey.
