HoloLens, an AR product from MicroSoft, features virtual objects pinned and placed in a spatial environment, even allowing users to color in augmented reality environments. In simple terms, the HoloLens consists of the following four main components: 1. The holographic Lens technology, or holographic display or output technology; 2. Sensor technology, which is used to obtain input information; 3, the Holographic Processing Unit: HPU, the former is similar to the Holographic computing CPU and GPU we use; 4. Interactive Control System.

1. Holographic Lens Technology

Microsoft claims that The HoloLens is beyond the screen and pixels. It has a high-definition perspective hologram and contains three layers of base color glass, for example, red, green and blue, each with a micron-level corrugated notch. A light engine causes light particles to bounce millions of times into these lenses, where they touch these glass lenses and enter the eye at an Angle, creating an image of intensity and color in the retina. Microsoft's patent US9759917 discloses an interactive head-mounted eyepiece that displays content and images on the optical components of the eyepiece. And the eyepiece contains projection optics, which project an image onto a perspective/translucent lens through which the user can also see objects in the real world around them. The optical element also includes an RGB LED module that USES the region sequential color technology to produce full-color images. The image is reflected into a two-sided waveguide for total reflection, allowing the user to view the image in the lens area.

Figure 1. Display optical element for headpiece

Figure 2 provides a good summary of all the HoloLens key components, with a holographic lens, A radio frequency Antenna, open Multimedia Application Processor, holographic projector, ear buds, radio transceiver.

Figure 2. Schematic diagram of key components of HoloLens

2. Sensor Technology

Sensor technology is the backbone of all AR or VR systems. It enables head tracking, eye tracking, depth perception, and environment mapping to enhance the user's immersive experience. HoloLens Comes with some advanced sensors, as well as an inertial Measurement Unit: IMU. These sensors include a pair of environment-understanding cameras placed on each side (i.e., two cameras on each side), a Depth camera, an Ambient Light Camera, and a 2MP photo/HD video camera.

The two Environment-Understanding cameras on each side of the sensor pole can be used both to track head movement and as a 3D time-of-flight camera. A depth camera in the middle can both track gestures and help combine holograms with objects. In addition to these sensors, it was equipped with four microphones (microphones) in order to better capture the user's instructions and identify them from the noise of the surrounding environment. The other is a common sensor used in AR helmets. It is a gyroscope, Magnetometer, and accelerometer inertial Measurement Unit. These cutting-edge sensor arrays constantly capture internal and external information about the user. See Figure 3 below

Figure 3. Sensor array for helmet display

Microsoft released a patent called "Pose Tracking an Augmented reality Device" US9495801, which disclosed that an AR device is equipped with multiple sensors to obtain Pose information. It filters out trace errors from the attitude information, then renders a virtual image and displays it. The helmet display in Figure 4 includes a transparent display system that projects holographic objects into the user's eyes, as well as various sensors to track the user's eye and head movements.

FIG. 4 is a schematic diagram of helmet display equipment

3. Holographic Processing Unit

Microsoft's HoloLens enables these advanced sensors and holographic processing units (HPU) to work together. The sensor continuously produces data for HPU processing. HoloLens runs on an X86 based Windows 10 operating system. Microsoft claims that HPU can perform trillions of calculations per second, making the algorithm 200 times faster than pure software, and requiring relatively low power consumption (less than 10W) to process posture information and sense the surrounding environment. Microsoft aims to select custom processing units to optimize device performance, latency, flexibility, and power supply. Thus, Microsoft has used For HoloLens Cadence's Tensilica DSP (formerly a Silicon Valley company with the business of Semiconductor Intellectual Property Core, now owned by Cadence Design Systems), and Cadence has reserved all the rights and permits for Microsoft to Design the chip.

Because Microsoft and Cadence have licensing agreements, the patent assigned to Cadence, US8539399, discloses a novel approach to computer implementation by designing a microprocessor with user-defined interfaces that allow users to transfer state or control information between the processor and the external device/processor.

Figure 5 shows the processor configured with a user-defined interface. The user can connect the input and output interfaces through a processor pipeline execution path. The input and output interfaces read and write data through an external agent.

Figure 5 holographic Processor Unit (HPU)

4. Interactive control system

Microsoft conceived and implemented HoloLens in order to create a more natural and immersive experience for users. HoloLens responds quickly to eye gaze, gestures, sounds, and the user's surroundings. A built-in sensor allows the user to control the cursor on the screen with the orientation of their eyes, tracking head movements as the cursor moves accordingly.

Microsoft has disclosed a patent related to tracking eyes US9759913. This patent describes a transparent waveguide for tracking eye movement, which includes an input coupler and an output coupler. The input coupler has multiple curved lines to define the Radially varying pitch. When the eyes are exposed to infrared light, infrared light will reflect from the eyes into the input coupler. An input light enters a waveguide in an input coupler where it travels through total reflection and comes out of the output coupler.

Figure 6. View of a planar waveguide for an example

The patent US20160080874, named "Gaze Based Audio Direction", is a technology to assist users in listening to sound in noisy environment by tracking the direction of eye Gaze. This patent addresses the problem of having more than one source of sound in the environment and allows the user to focus only on the target speaker through the direction of gaze of the helmet display device, as shown in Figure 7.

Figure 7, an example of voice-assisted use

The sensor not only tracks the direction of eye gaze but also recognizes the user's gestures to control any app, select and zoom, and drag the hologram. A patent called Gesture Based Region Identification for Holograms, US9116666, reveals a technique for selecting virtual images in a region by Gesture. The user can select the area by hand by forming a closed ring to scale, filter, and so on. Users take photos of their surroundings with HMD, and then merge the virtual images with the surroundings, thus giving the user an augmented reality experience.

Cortana can also assist HoloLens users with tasks such as navigating, selecting, opening and controlling apps using voice commands. Four microphones make it easy to communicate in a noisy environment. It also has a built-in speaker that allows users to hear precise sounds without headphones. In addition to these features, Microsoft allows other third-party technologies to enhance HoloLens performance and enrich the user experience, such as the holographic Portal proprietary US20170169610.

While the first generation of HoloLens has a decent display, it's bulky; Although the volume of the second generation is lighter, the dispersion is serious and the Angle of view is small. Therefore, it still needs a long way to go to make a good AR product.