51TECH is 51WORLD’s brand-new column, delivering the frontline analysis of cutting-edge and hard-core technologies.
We believe that technology transforms our life, we believe in the goodness of technology, and we hope to share our experience and thinking with the industry at 51TECH
In recent years, the concept of “big world” put forward in the field of smart transportation has posed new challenges for related simulation. Among them, the four major challenges for the static simulation of intelligent traffic are the construction of super-large maps of hundreds of square kilometers at the macro level, the integration of digital assets in the “big world” at the meso-level, the optimization of rendering efficiency on multiple platforms, and the construction of a composite digital asset library at the micro-level.
In this issue of 51TECH, we invite Wang Chao, the content director of 51WORLD Intelligent Driving and Transportation Division, to talk about the upgrade of static environment simulation technologies and the detailed explanation of creating a large traffic simulation environment.
The content team of 51WORLD Intelligent Driving and Transportation Division has been committed to providing full-stack content solutions for autonomous driving simulation and traffic simulation since its inception. It has accumulated rich experience in the rapid production of simulated maps, automated map generation, efficient rendering on multiple platforms, and aesthetic balance after delivering nearly 100 projects.
As a seasoned frontline static simulation engineer, Wang feels the challenges facing the current industry.
As the advent of the 5G era has pushed intelligent transportation to a new height, the concept of “big world” came into being in the field of traffic simulation. The big world denotes the upsurge of demand in the range of simulated visual maps and the length of road network. A transportation project may require hundreds or even thousands of kilometers of large or super large simulation maps for training, collecting vehicle data set, and assisting in decision-making. It also means an exponential increase in the volume of simulation functions.
Four major challenges come on the heels of this burgeoning industry: the construction of super-large maps of hundreds of square kilometers, the integration of digital assets in the “big world”, the optimization of rendering efficiency on multiple platforms, and the construction of a composite digital asset library.
01 Construction of super-large road networks
A bird’s eye view of the West Third Ring Project in Beijing
Since the localized simulation maps of the first perspective are limited and inadequate for smart transportation, the free-view maps with the car-tracking perspective and the bird’s-eye view are constantly yearned for. However, it needs an accurate restoration of hundreds of kilometers or even thousands of kilometers of the simulated terrain to realize a bird’s-eye view from the third perspective. For example: to simulate a 100- square-kilometer scene, it is necessary to recreate roads with a total length of 300 kilometers, of which about 40 kilometers are highways or first-level roads, about 160 kilometers are urban expressways or second-level roads and 100 kilometers of branch roads or roads of the third level and below, thus forming a huge road network.
A diagram of the simulated road network
Therefore, we need to efficiently integrate and utilize limited computing resources, so as to complete such large-scale simulation maps. Next, I will explain how we significantly reduce the unreasonable resource consumption of the GPU with stress tests.
How to do stress testing?
It must be clarified that since a traffic simulation map is the mapping and restoration of the real world, it depends on various data sources, including point cloud data, tilt photography, satellite images, panoramic images, aerial video, etc. However, these real data are more or less problematic, so a secondary design is needed to achieve accurate geographic location, matching regional style, and smooth operation efficiency.
We conduct research with our accumulated blank model testing data to evaluate the integration of the road network and the surface, and calculate a set of map segmentation schemes to facilitate the program to efficiently load each atomic map, thereby enabling transportation functions to make more flexible. In addition, we subdivide the smooth meshes to no less than 3 times, simulate no less than 100 dynamic vehicles and no less than 100 moving pedestrians, and adapt complex coloring and high-precision textures to the components of the blank model for specific tests. At the same time, we add greenery of no less than 5000 types to verify the frame rate of static scenes. 。
We group the testing according to the types of rendering, such as road networks, buildings, vegetation, etc., and strictly control the budget after stress testing, in order to improve the team’s efficiency, thereby shortening the development cycle. It is worth noting that even without ForceCulling, LOD, and HLOD, the current stress test method we use is generally very efficient, because this method can quickly assess the impact of the asset on the map rendering and quickly calculate the budget for various types of resources such as buildings, roads, surfaces, vehicles, and pedestrians, so as to reserve more space for dynamic functions.
When the stress testing is done abundantly, we can optimize the product for different situations. Next, I will talk about the difficulties in restoring the road network.
The road network should reveal the complexity of the real road, such as the undulating conditions of the road surface, the changes of the intersections, etc. Meanwhile, it should be able to meet different needs of smart transportation such as parallel computing, sensor simulation, vehicle dynamics, etc. Therefore, the production of a traffic simulation road network cannot be done simply with the curve editing tool of Spline.
51WORLD independently developed the WorldEditor to automate the geometric data and UV mapping of a road network, thus greatly improving the efficiency of producing a road network, on which a whole simulated world can be developed.
OpenDrive road network file exported by WorldEditor
At present, our team is capable of producing large maps of 100 square kilometers, and we aspire to rapidly recreate larger maps of several square kilometers and construct the road network of thousands of kilometers. Next, I will talk about how to effectively integrate digital assets in producing maps.
02 Integrating digital assets in the “Big World”
An integrated display of various digital assets in stress testing maps
The Big World is a combination of small cell maps, each containing a huge number of digital assets. Different projects have different acceptance criteria. Now, we integrate digital assets mainly in two directions: one is to add semantics, and the other is to lighten heavy assets.
Adding semantics to digital assets: assets used in sensor simulation are typical heavyweight digital assets and of the highest level. As time and weather vary, semantic information should be added to meet different requirements of sensor recognition training, such as the parameter of wet and dry environment changes, snow stains coverage parameters, etc. Up till now, we have processed more than 40 semantic heavyweight digital asset types, which maximizes the restoration of object properties and material changes in the real world.
The lightening of heavy assets is often used for the rapid retrieval and use of assets. For example, the digital assets loaded in the WorldEditor need to be lightweight. At the appearance, it seems feasible to make multiple versions of assets to meet different needs. However, the extremely high costs put off even the most fervent tech-lovers.
Therefore, we define varied requirements in different levels of LOD, encapsulate these assets with BP, upgrade and downgrade them more flexibly, adapt appendages, and replace materials to support sensor simulation, case testing, and vehicle dynamics.
For example, we divide the attributes of real-world vehicles into static and dynamic. In the static dimension, lightweight assets need a reasonable structure, a low number of faces, and a small number of materials, while heavyweight assets need to be clearly structured with rich materials and realistic textures. Meanwhile, both of them can flexibly add appendages, such as bus license plates, advertising stickers, warning lamps, etc. In the dynamic dimension, the first consideration should be given to adapt corresponding skeleton animations to different levels, restrict the luminous intensity, and control the number of particles, etc. Therefore, flexible upgrades, downgrades as well as adaption become the kernel of making composite assets.
03 Establishing a composite digital asset library
A digital asset library is the smallest unit to build a map. Whether a single asset meets the rendering standards in the field of traffic products directly affects the performance of a simulated map, which determines the feasibility of constructing a big world. Therefore, it is instrumental to build a composite digital asset library. We assess digital assets in terms of aesthetics, compliance, and feasibility.
Aesthetics: we should evaluate whether a digital asset faithfully restores the properties of objects in the real world, such as surface materials, size ratio, etc. and whether a digital asset meets the public aesthetic standards. For example, we construct pedestrian characters with 3D footage from live scanning and create behavioral actions from action sequences generated by Motion Capture, to make them more credible and beautiful.
Compliance: The compliance of a digital asset will have a direct impact on the cost of its subsequent iteration. The assessment of compliance not only includes models and materials but also hidden functions such as vertex shading, the role of multiple sets of UV, model LOD, physical materials collision occludes, luminous parts, multi-dimensional sub-materials, bones, audio code rate, animation frame rate, particle size range, source file format, naming rules, storage location, etc.
Feasibility: When a digital asset is referenced by a map, first it must be flexibly used by the level designer. And the second modification will not lead to repeated revision and construction of the level. Last, it can be modularly combined into a new resource. The gist lies in whether an asset has been dismantled carefully in the early stages of production.
A diagram of the composition of digital assets.
In static environment simulation for intelligent transportation and autonomous driving, the closest visual distance of a digital asset should be less than 1m, which is highly dependent on the digital asset library. The asset should be able to be displayed on multiple platforms, such as PC-based VR, MR and XR, and the access of cloud rendering also needs to consider the size of the resource package. It emphasizes the functions and effects, but the digital asset needs to be lightened, the seemingly contradictory requirement of which also exists in the film and game industry.
9 categories of digital assets.
Our digital asset library consists of nine categories that assume the largest elements of the real world. Each category is divided into no less than 8 sub-levels. In principle, any static digital asset should be dynamically scalable. Therefore, we’ve reserved the functional interfaces, such as LOD, UV, materials, and vertex information for a static 3D model, so that engineers can freely combine the tools into dynamic elements available, which are, of course, part of the asset library.
The proportion of working hours after cleaning the original non-standard digital assets of the vehicle to meet different functions
Since simulating each 3D asset in the virtual environment takes a large amount of time, it is unattainable to process every digital asset to be extremely aesthetical. If we grade the digital assets of film and games at 10, these required by the digital twin can only get 6. It also seems realistic to remove redundant artistic details and return the models, textures, and materials to the original state.
The composite digital assets commonly used for transportation projects
Although we don’t agree with each other in aesthetics, I believe that digital twins must meet the basic standards, such as the blue sky on a sunny day, reflections on the surface of the water, etc. How to strike a balance between intuition and emotion is the basis for defining digital assets, which is also bound up with costs.
04 Rendering efficiency on multiple platforms
Due to the secondary design of the maps, performance stress testing, and the maintenance of standardized composite digital assets, the rendering efficiency of transportation simulation projects we delivered in multiple platforms is generally up to standards, even outperforming peer products. At present, intelligent transportation and autonomous driving simulation projects are mostly based on the PC platform. Against this backdrop, how to realize efficient rendering on other platforms such as 5G cloud rendering streaming, VR and MR poses another challenge for the industry.
In addition to the scale of the map, the digital asset itself and the toolchain also affect rendering efficiency. Just like running, being able to run is one thing, but to run fast is another. It is not easy to run with data and functions. Of digital assets, different perspectives, models, textures, materials, lights, motion sequences, special effects, sound effects, UI, etc. also have an impact on rendering efficiency.
The path design of the dynamic camera in the large traffic simulation map.
How to improve the rendering efficiency of the simulated maps? It is ideal to work with lightened maps on high-performance PCs.
Currently, we have achieved the following goals in traffic projects based on PC hardware:
[Successful implementation cases]
Up to now, our team has guaranteed the frame rate in the L4 static simulation maps of hundreds of square kilometers (51WORLD all-element scenes are divided into five levels from L1 to L5, with the scenes of L4-L5 being able to restore detailed elements such as roads, traffic facilities, vehicles, and pedestrians), and it has been verified on recent large projects.
L1-L5 all-element scenes of the static simulation
A microsimulation of rapid road
A micro-simulation of a block
A meso-simulation of smart traffic
A macro-simulation of a transportation hub
In making the micro-simulation map of the traffic in Beijing’s West Third Ring, we covered a total mileage of 30 kilometers from Wanliu Bridge to Suzhou Bridge and 10 kilometers of branch roads, restoring a whole area of 400 square kilometers.
Thanks to the building capability of large scenes, we have adopted the WorldMap, a construction approach particularly helpful for building linear road maps, which enables us freely load and cull target levels while changing multiple camera positions, making it easier to balance the performances between multiple sub-maps.
The steps of producing the linear road network of Beijing’s West Third Ring
Transportation simulation aims for being realistic. If we personify a product, saying it has bones and flesh after solving the four major challenges above, the game doesn’t stop here. We want to make it more lifelike by perfecting the product in terms of macro geographic information, micro scenes management and dynamic systems.
A corner of the traffic case map
01 Restoring real geographic information
From the macroscopic dimension, the rules of constructing in the real-world should be applied in a virtual simulation, to ensure the authenticity of the simulated environment to the greatest extent.
The order of simulation is similar to that of the actual construction. First of all, in terms of geographical information, we should consider the earth’s latitude, longitude, time and weather, and reserve the real natural environment of mountains, rivers and lakes. Then, a piece of land is allocated to plan the road network, followed by the generation of different types of buildings. Finally, we scrutinize the details of the network, such as the road-related facilities and greening off the roads.
The 1:1 restoration of the Jiading demonstration area of the National Intelligent Network
02 Subdivided scene categories
We divide static simulation into more than 20 categories from the microscopic dimension, each with different priorities. Here are some major categories: geographical type, road type, road marking, traffic-related equipment, roadside facilities, main vehicles and traffic flow, pedestrians and animals, weather and time, and display perspectives. 51WORLD has subdivided toolchains for each category and we will consider the real situation of different projects. Taking the geographical type as an example: as Guangzhou is closer to the equator than Beijing, their green plants, surface textures, and light and shadow changes vary significantly.
The details of the static simulation of traffic-related facilities such as traffic lights, signs, radar speed measurement, etc. are easily overlooked, but 51WORLD has been constantly enriching and accumulating different styles. Apart from following the national standards, we underline the features in various regions. For example, the traffic lights in Beijing is of standard cylindrical shape, the lights in Shanghai have curved angles while these in Shenzhen have eaves like wearing a beret.
Categories divided according to the applied scenarios
03 A complete dynamic system
Static environment simulation needs a dynamic system to be faithful and realistic. We compare the dynamic systems of traffic flow, pedestrian flow, weather, and time changes as the neural networks in intelligent traffic simulation. Time and weather systems have the highest adhesion to static environment simulation.
People who has had offline rendering experience know that even if the world is made of white boxes, GI and lighting can make it real, good-looking and credible. Therefore, this technique can be capitalized on in real-time rendering. Light and shadow directly affect people’s perception. Continued improvement of light tracing technology will inevitably push environmental simulation to a higher level.
Perfect time and weather functions enable the R & D team to define traffic cases more accurately. Traditional real-time rendering engine fails to fully support the access and connection of real data, although in which dynamic weather and time are gradually improved. For example, if the customer asks for some rainfall in a sunny environment at 12, or vaguely seeing the outline of the sun in a cloudy evening, in the past, we have to produce the time and weather separately. However, as the demands surge, the lack of relevant tools will seriously reduce work efficiency and output quality. Besides, it will be very unfriendly to the semantic recognition of the sensors.
Therefore, our content team designed a time and weather system based on current project types, in which we set 8 common kinds of weather and 12 extreme conditions, reserved the interfaces of time, material, special effects, and animation. The secondary development of these functions will enable flexible utilization of 51Sim-One, and meet the customers’ highly customized needs for time and weather to the greatest extent. Now the senior programmers on the engineering team are working hard to upgrade this tool to make it handier in the future.
The 24-hour changing effect of the time system in the smart transportation project
Time and weather system effects
At present, our team can freely switch from process to quality control in static simulation, make realistic virtual scenes, and deliver the product in time. We believe that in the future, with the continuous iteration of lighting, materials, animation and other technologies in real-time rendering, the simulated maps, and dynamic effects will be presented as more real. In this regard, we organize regular training courses to keep the members in touch with the latest trends in the industry and aspire to sharpen our competitive edge.
Wechat Scan to follow 51VR