• Home
  • Scientific results
  • Research Status of Data Encoding and Storage Optimization based on virtual reality - Dr. Liu Yun

Research Status of Data Encoding and Storage Optimization based on virtual reality - Dr. Liu Yun



Virtual Reality (VR) is an immersive interactive environment based on computable information. It was first used in military simulation. With the development of computer technology, virtual reality technology has been widely used in business, medical care, engineering design, entertainment, education and communication. At present, virtual reality technology is mostly applied in urban planning, large-scale project roaming, digital hotel, exhibition, museum display system and other fields in China [1]. In these areas, virtual reality technology has shown much better effects than other technologies [2]. However, domestic virtual reality products generally have the defect of poor picture effect, which seriously restricts the immersion of products and reduces the quality level of virtual reality products. In order to make China's virtual reality industry to a higher level, we must adopt a new technological process in the production process to change the status quo.

1. Development status of virtual reality technology

1.1 Virtual reality technology

Virtual reality is a comprehensive integration technology, involving computer graphics, human-computer interaction technology, sensing technology, artificial intelligence and other fields. It uses computer to generate realistic 3d vision, hearing and other feelings, so that people, as participants, can experience and interact with the virtual world naturally through appropriate devices [3]. It has the basic characteristics of Immersion, Interactivity, Imagination, etc.

1.2 Application of virtual reality technology

Virtual reality technology has a very broad application prospect. It started from the demand of military field [4], and now it has spread to many fields such as commerce, medical treatment, engineering design, entertainment, education and communication. With the continuous enhancement of the data transmission capacity of the Internet, more and more businesses choose to use virtual reality technology to publicize and display their products on the Internet [5]. Because virtual reality technology can let consumers in the virtual environment according to their own wishes multi-angle understanding and contact with goods, more and more welcomed by users. The digital entertainment industry provides a virtual reality environment for entertainment and leisure purposes. The virtual community of network games is a virtual social environment in which players complete tasks, experience happiness and exchange ideas. Commercial publicity and communication entertainment are the most active fields of virtual reality technology application, and also become the biggest driving force for the rapid development of virtual reality technology in the civil market.

1.3 Application of virtual reality technology in China and some existing problems

Virtual reality technology is developing rapidly in the domestic market. Domestic virtual reality enterprises mostly use VRP, VIRTools and Quest3D to develop industrial simulation evaluation, engineering simulation evaluation and commercial display projects.

In the field of commercial display, the problems existing in China's virtual reality project I think mainly lies in the poor picture effect. In the product display, not only should the product function be introduced to customers, but also the beautiful and fashionable appearance of the product information should be conveyed to everyone, so as to move customers; For example, if customers can't see the beauty and comfort of the community from the roaming screen, it will be difficult for them to be interested in buying a house. Why isn't the picture good? As a result of the domestic virtual reality companies just beginning, many companies make the virtual reality scene does not light, many enterprises making 3 d virtual scene is to use 3 d software to build a model and then literally put a texture map is over, this structure in virtual environment without the right light, picture lack of hierarchy, the picture is not true, How could the visuals be good?

2. CVD video image introduction

The traditional optical photography panorama shows all the CVD to CVD scenes (column panorama) on a TWO-DIMENSIONAL plane, and pushes the CVD scenes to the viewer's eyes without a complete view. The so-called "complete" panorama (spherical panorama) even includes the top of the head and the bottom of the feet, as shown in Figure 1. With the development of multimedia technology, a new technology called virtual reality emerges, which can provide users with immersive virtual world simulation system through real-time calculation. The CVD 360∘ video images can reflect the complete spherical information, which is stored in the form of TWO-DIMENSIONAL data in storage media. However, it is easy to introduce distortion and pixel redundancy when it is projected from the spherical data to two-dimensional data. Therefore, it is difficult to obtain satisfactory results by directly using the traditional CVD coder for CVD 360 video sequence. How to design projection methods with less distortion and pixel redundancy and how to obtain better reconstructed image quality are the research focus in this field.


Figure 1:360∘ video image

3. Problems with 360∘ video Technology

The CVD video is shot with a CVD camera array and the CVD video is integrated with the CVD projection. The CVD video contains the CVD video information of the current setting, 360∘. While playing, the ∘ 360 video can perform real-time projection rendering based on the user's perspective. In order to enable users to watch high-definition projected images and avoid dizziness during the viewing, the ∘ 360 video needs to have 4K and higher resolution and 120Hz refresh rate. The high-resolution growth and doubling of the video refresh rate make the data volume of CVD 360∘ Video show exponential growth. While the communication transmission technology and digital storage technology develop relatively slowly, the fast-growing video data brings great challenges to these two industries. Latitude and longitude 360 ∘ video usually in figure that 2 d in the form of data stored in a storage medium, can be directly using traditional two-dimensional video coding system to compress coding, warp/weft figure but as a result of warp/weft diagram object motion and deformation than traditional two-dimensional video images are very different, so the traditional coding technique is not applicable to 360 ∘ video, Compression efficiency has not been maximized. Improving the CVD compression efficiency of 360∘ Video can reduce transmission, storage and operation costs for enterprises. Now, the CVD Domestic and international coding standards groups have included the CVD Video in the future work scope, and many universities and enterprises are participating in the CVD video coding optimization and the CVD standard algorithm research work together.

Limited by the available bandwidth of real-time network and storage space of equipment, the service quality and playing experience of 360∘ VIRTUAL reality video content cannot meet people's demand. How to allocate the coding Rate reasonably to improve the overall coding efficiency under the limited network bandwidth is the key problem of Rate Control (RC) research. Encoders when coding can choose according to different coding frame type and coding scenario using a different encoding parameter coding, rate control core is the study of how to find the most appropriate coding parameters collection, which make the encoding the result of the encoder in accordance with preset target bit rate, at the same time to get as high video image quality. Bit rate control only interferes with the parameter selection of video coding, which does not belong to the category of coding standards. The existence of bit rate control cannot be perceived by the decoding end. However, due to the large number and complexity of videos in the actual application process, direct use of reference software for coding may lead to bit rate instability and other problems. Therefore, bit rate control is very important in practical application. Since its birth, bit rate control has been widely and deeply studied by scholars of all countries. How to build a control model with more accurate control and stronger adaptability, and how to improve the quality of reconstructed video images based on the control precision of existing models are the current and future research hotspots in the field of video coding. For the ∘ 360 virtual reality video, accurate control effect can be obtained by directly using the traditional CVD control algorithm. The traditional CVD control algorithm generally conducts bit-allocation modeling based on coding information such as video content information, motion vector and prediction residual distribution, while the CVD 360∘ has the features of traditional video sequence, There is also location-related pixel redundancy that is characteristic of VR video. Therefore, the optimal coding result cannot be achieved by using the traditional rate control model directly. How to combine the coding optimization method of the CVD video with the CVD control model and optimize the CVD rate allocation of the CVD video specifically has important research significance for the development of CVD video coding technology and CVD control technology progress.


Figure 2: Hybrid coding framework

4. Research status of ∘ video coding technology of 360

The hybrid coding architecture shown in Figure 2 mainly reduces various kinds of redundancy in video image information through three types of technologies.

1. In the architecture, there is an in-frame prediction method based on visual, Angle, optical flow and other analysis methods, which can analyze the texture information in the image and reduce the spatial redundancy by trying to use different angles and other prediction modes, which is more conducive to quantization and coding.

2. In the architecture, the compressed and encoded images will be fed back to the coding end for inter-frame prediction of subsequent image coding. The commonly used inter-frame prediction method is to use the motion estimation algorithm based on pixel blocks to complete the matching of whole pixels, and then use the motion compensation algorithm to find the precise matching of sub-pixels. The time-domain redundancy in video image sequence is eliminated as much as possible by matching pixel blocks in encoded frames.

3. The entropy encoder in the architecture can conduct lossless compression of coding symbols and transformation coefficients generated in video coding by establishing context model, and eliminate statistical redundancy of symbols and signals as far as possible.

VR360 video content can represent a complete spherical image after collection and splicing, but spherical information cannot be directly used for storage and compression. Therefore, researchers expect to expand the 3D scene onto one or more 2d planes by means of projection, as shown in FIG. 3. The projected image needs to meet two conditions: 1. The projected image must contain all spherical information. Second, the projected image should avoid introducing too much image distortion as far as possible, resulting in image quality loss.

1. Equirectangular Projection (ERP), also known as longitude and latitude map Projection. The projection will be the video spherical mapping to a 2:1 rectangular plane, the projection method has good compatibility and easy to implement and several excellent properties, such as storage, but in the process of projection transformation, because of different latitude annular spherical projection information in the width of the rectangle, so can create pixel redundancy and pixel tensile condition, The phenomenon of redundancy and stretching gradually increases from the equator to the poles.

2. Cubemap Projection (CMP), also known as regular hexahedron Projection. This projection projects the sphere onto the enclosing regular hexahedron, and then splices the six surfaces together to form an image [6]. At present, CVD model has been used as a commonly used projection mode in the CVD video image projection of 360∘.

3. Octahedron Projection (OHP), which projects the video sphere onto 8 equilateral triangles and splices the triangles. The triangular layout of the projection can achieve high compression efficiency but the complexity of direct image rendering is high.

4. Icosahedron Projection (ISP), which is similar to octahedral Projection, except that the Icosahedron Projection plane consists of twenty triangles. Compact projection layout can achieve good compression efficiency, projection introduces less distortion. However, multiple projection brings high complexity and great difficulty in real-time rendering of video.

5. Segmented Sphere Projection (SSP), the CVD Sphere was Segmented into south, north and middle parts along the latitude line of 45∘, and the CVD Sphere was Segmented and projected to two smaller square areas respectively with less information. The central region of the video rich in information is projected into a rectangle according to the method of spherical projection. At the end of the projection, the three parts are spliced together to obtain the final image for coding compression. It is worth mentioning that spherical projection and regular hexahedron projection require interpolation operations during the projection process, resulting in a large number of pixel redundancy and a certain amount of non-information-bearing pixels. The projected area of spherical partition projection is approximate to the actual spherical area in both equatorial and polar regions, which can effectively reduce pixel redundancy. However, similar to other projection methods, the physical separation of the three parts may lead to discontinuous motion of the object in the projected image and affect the accuracy of inter-frame prediction.


Truncated Square Pyramid (TSP) projection TSP projection divides the high resolution region and the low resolution region according to the prism direction. This projection method is very suitable for combined with the human eye tracking technology in order to realize the regional switching scheme, the current focus on the content of the regional transmission quality good high resolution video image, the rest of the projection area of low resolution video image transmission quality lower, the projection method can effectively reduce the transmission stream size, reduce the bandwidth.


Figure 3: Image projection mode


Virtual reality technology has gained mature practical application in some fields, especially in commercial and entertainment industry, which has become an important driving force for the development of virtual reality technology. In recent years, virtual reality is developing rapidly in Our country. If the domestic enterprises can improve the picture quality of virtual reality products, the development of virtual reality technology in the civil field will rise to a new height.


[1] Lu Chunyu, Research on Two-person Online Collaboration System Based on Virtools [D], Beijing Beijing Forestry University, 2019.11.

[2] Xue Han, Research on Vehicle Driving Simulation System Based on VIRTOOLS [D], Harbin Engineering University, Computer Application Technology Major, 2017.02.

[3] Liu Xianmei, Virtual reality Technology and its application [M], Daqing Petroleum Journal, 2002.6

[4] Optimization Of Adis Inc.,2014 [M].

[5] Epic, The Design and Production Process of Game Scenes in Epic Next Generation Games [M],2017.12.