In this blog, we will learn about binary files and how to create and use them in our engine. We will create binary files for human-readable mesh files that we learned about in the previous blogs.
Let's begin with understanding what are binary files. Binary files are the raw data that can be loaded directly into memory for any required operations. Sometimes you might need some processing to be done before using the data for operations but essentially a binary file is just a stream of data. A good example is let's say in your program you have an array of 100 integers you can write the array memory to a file which is the binary file and whenever required the binary data can be loaded into memory and you can perform any operations you wish on the loaded array. So why do we need these binary files for our mesh don't we already have the human-readable mesh files to represent the mesh? Yes but these binary files are not to represent the mesh these files contain the actual mesh data that can be loaded into memory. It is a little confusing but let's understand it. Imagine the same array example above and let's save the array into a human-readable text format. If we have to use this text file to load the array in our program. we have to first parse the file get the required values in the array allocate required memory and then assign the values from the text file which is a lot of work to do. By storing the array into a binary file we avoid all the extra steps as we can directly load the binary data into memory and access it.
Advantages of Binary Files
Binary files have a couple of advantages the first one is the file size. Binary files are small in size compared to other formats because we only store the required data. For the above array example, you just store the size of the array in bytes but in text format as it is not the actual data the size will be large. Another advantage with a smaller size is that they are quick to load. Loading from disk is slower so it is always a good idea to keep the small size. Another advantage is that after loading the binary file you will need minimal to no processing to be done before you perform the actual operations on data which is not the case with a text file as there are extra steps as discussed above.
So if binary files are that good why do we have the human-readable format in the first place? Remember though both files represent the same data of an object, both files have different purposes. The human-readable file is for developers to be flexible and go through it understand. It can be used for debugging and other reasons which is why we need it to be understandable & maintainable. On the other hand, the binary file is for the machine so that it can operate on that data as quickly as possible. It is hard for someone to use the binary file and debug an issue as the format is not easy for humans to work with. But, machines are good at working with binary data so it is faster to load and process it.
So how are we going to generate these binary files and when do we use binary file & human-readable file. So we use the MeshBuilder that we created in previous blogs to create a binary file using the human-readable file during the build time and we use the binary file during the run-time to make it load fast. So we will not use the human-readable file during runtime so even if the binary file is corrupted for some reason we can use the human-readable file to generate it again using our mesh builder. Our build process is set up to generate the binary files for each of the meshes.
Let's look at an example of how I created my mesh binary file.
There are many ways you can do this and I did in this way so that there is less processing when we load the data.
The way I structured my binary data is I created a MeshInfo structure that represents the mesh data in my previous blog. I am using the same struct.
struct MeshInfo
{
uint16_t VertexCount = 0, IndexCount = 0;
Graphics::VertexFormats::sVertex_mesh* VertexData = nullptr;
uint16_t* IndexData = nullptr;
};
I am writing the data of this struct along with vertex data & Index data to a binary file in my mesh builder.
Following is how it looks in binary
The first piece of data I store is the actual mesh info data that it is VertexCount, IndexCount, Pointers for both vertex data and index data. So the data in the green square is vertex count data in the orange square is index count. The next 8 bytes(white area) is for pointers which we will update later. The next shaded blue area is vertex data and the data in the yellow box is index data. It is important for the mesh info data to be before vertex data & index data we will see why when loading the data.
As we are using the mesh info struct it is very easy to extract data during runtime. First, we create a mesh info pointer and assign the starting address of the binary data. That means we already have vertex count and index no need to create a new memory and copy data. we have avoided two copies. The next step is to update the vertex and index data pointers. Update the vertex data pointer to memory after the mesh info memory. This can be done easily using the size of the mesh info struct. we know how long the mesh info struct is so it should be simple. once the vertex data pointer is updated use it and the vertex count to update the index pointer which begins after the vertex data that's it.
A couple of things to keep in mind. We can use other layouts for the binary file which results in a smaller size for the binary file but you might have to do more processing during runtime and might also need to make some allocations and copying of data. I used this layout as it decreases a few steps both during the build time and during the runtime by skipping a few allocations and copies. You might have to change your layout based on your priorities.
One other optimization that we can do is we can update the winding order of the indices during the build time so that we do not need any processing to be done during the runtime.
So this makes the binary file to be different for both OpenGL & Direct 3D.
We have discussed that binary files are smaller and faster than human-readable files lets see how much smaller and faster are they.
Let's take the Santa mesh in the above gif as an example.
MeshFile Size Load & Extracting information
Human_Readable 712KB 0.0676729 sec
Binary 163KB 0.0002007 sec
The size of the human-readable file for Santa mesh is 712KB whereas the binary file is only 163KB which is a big difference. Similarly, loading and extracting information from a human-readable file took 0.06 sec whereas the binary file only took 0.0002 sec. Now imagine how much better performance we can achieve using binary files in a large project.
Controls:
ESC – exit.
Up/Down/Left/Right - Move Gear Object
W/A/S/D/Q/E - Move Camera Forward/Left/Baclward/Right/Down/Up
ENTER - Changes the mesh to square and effect to the animating color of the object.
Space - Toggle between main and secondary camera. Note: the secondary camera does not support the movement.
Comments