Modding:Serialization Formats

Vintage Story uses multiple formats to serialize data. This page compares then.

Formats

JSON

JSON is a text format. It holds tree data, where the branches are identified either by a name (JSON object) or by a contiguous integer (JSON array). The leaves of the tree are integers, floating points, booleans, strings, or null.

JSON is used for the configuration and asset files.

Reserializing a JSON file (deserializing the JSON into an object, then serializing the object back to JSON) strips all of the custom formatting allowed by json5. So typically the game treats JSON as read-only data. The exceptions are:

• If the config files are missing, the game writes a new file with default values. That default file is created by constructing the configuration object with default values, then serializing it into JSON.
• Some of the debug messages log objects in JSON format.
• When the server transfers its assets to the client, most of the asset types that have an Attributes field will serialize it back to json to transfer it to the client.

TreeAttribute

Similar to JSON, TreeAttribute holds tree data, where the branches are identified either by a name or contiguous integer (when the attribute is an array).

There are a few differences in the kind of tree data that TreeAttribute can hold as compared to JSON:

• Mixed arrays are not supported. For example, JSON allows an array to contain a mix of integers and strings, but that is not supported by TreeAttribute.
• The various decimal data types are distinguished. For example, 5 as int, 5 as long, and 5 as a float, and 5 as a double are all distinct in a TreeAttribute. However, JSON simply identifies 5 as a number.
• ItemStack is treated as a primitive type in TreeAttribute. ItemStack is treated as a generic object in JSON.

TreeAttribute is a binary format, which makes its serialization more CPU efficient than JSON. Because the binary format still has to store a string for every object entry, a serialized TreeAttribute takes roughly the same space as the equivalent JSON file.

TreeAttribute is used for transferring block entities from the server to client. It is also used to transfer entity attributes.

Protobuf

Protobufs serialize objects. They support most types of fields in the objects. A few annotations must be added to the classes so that the serializer knows which fields to serialize, and how to map the fields between versions for backwards compatibility.

Protobufs are used extensively for the network packets sent by VintagestoryLib. Vintage Story actually uses two different protobuf implementations. silentorbit/protobuf is used inside VintagestoryLib, and is only visible by decompiling the library. protobuf-net seems to be better supported, because it is publicly exposed in vsapi.

The Wireshark vs-protocol dissector can decode the outer Protobuf layer of the network packets. It is a helpful tool for debugging network problems.

BinaryWriter

The BinaryWriter and BinaryReader classes are built into .NET. They only directly support serializing simple types. Serializing complex types is a very tediuous process. It invovles manually writing the code to break the complex object into simple types, and serialize those simple types.

TreeAttribute internally uses a BinaryWriter. BinaryWriter is used to serialize entities, both over the network and to save files. It is also used for the network packets that set blocks.

Converting

The supported conversions are summarized in the diagram to the right. More detailed code is given in the sections below.

Serialized formats are represented with rectangles. Objects in memory are represented with ellipses

JsonTreeAttribute

This is dead code. ClassRegistryAPI has a function that accepts it, but nothing calls the function. Ultimately there is nothing that constructs a JsonTreeAttribute.

Object

The diagram uses "object" to describe an instance of any specific class that supports serialization. An example is BlockType.

object to json string

Use the JsonConvert class from Newtonsoft. JsonConvert.SerializeObject(obj)

json string to object

JsonConvert.DeserializeObject<T>(json)

JsonObject

This is a generic object for holding the memory representation of a JSON file. This type is used for the collectible attributes.

JSON String to JsonObject

JsonObject.FromJson(json_string) or new JsonObject(JToken.Parse(json_string)).

JsonObject to json string

jobject.ToString()

JsonObject to TreeAttribute

jobject.ToAttribute()

JsonObject to object

jobject.AsObject<T>()

TreeAttribute

Both the serialized binary format and the deserialized objects are called TreeAttribute.

TreeAttribute to json string

tree.ToJsonToken()

TreeAttribute to serialized TreeAttribute

tree.ToBytes()

Serialized TreeAttribute to TreeAttribute

tree.FromBytes(serialized)

Protobuf

Protobuf to object

SerializerUtil.Deserialize<T>(bytes)

Object to protobuf

SerializerUtil.Serialize(obj)

Wondering where some links have gone?
The modding navbox is going through some changes! Check out Navigation Box Updates for more info and help finding specific pages.