Modding:Deserializing protobufs from the command line

Protocol Buffers are one of the serialization formats used by Vintage Story. If one can obtain a file containing one of these serialized protobufs, then it is possible to deserialize it and view the contents from the command line.

Obtaining the serialized data

The main vcdbs file is a sqlite file, which contain tables, which contain protobuf serialized blobs. It is possible to use the sqlite3 command line tool to extract the serialized blob from a row into a file.

For example, the following reads the row with savegameid=1 from the gamedata table of the modtestworld world.

sqlite3 modtestworld.vcdbs "SELECT writefile('gamedata_1.binpb', data) FROM gamedata WHERE savegameid=1;"

Alternatively, all of the rows can be selected and written to a filename that contains the primary key of the table. This dumps all rows of the gamedata table (there's actually only one anyway).

sqlite3 modtestworld.vcdbs "SELECT writefile('gamedata_' || savegameid || '.binpb', data) FROM gamedata;"

Obtaining the schema

Without the schema, it is only possible to perform a decode_raw of the protobuf, which produces difficult to read output. It is better to obtain Vintage Story's protobuf schema so that the output contains properly labelled fields.

The Vintage Story source code does not contain a normal .proto schema file. Instead, the schema is implicitly defined by applying protobuf-net annotations on types. However, a .proto schema file can be extracted from those types using the vs-proto tool. Alternatively, one can use the already extracted schema available on the vs-proto GitHub page. Protobufs are resilient to errors. So even if the already extracted schema is slightly out of date, it will still mostly work.

Deserializing the file

protoc is a command line tool that can deserialize the binary file. The tool can be downloaded as an asset of the Protocol Buffers releases on GitHub.

The --decode option is used to tell protoc to deserialize from stdin. The option is undocumented on the official Protocol Buffers page where the tool is downloaded from. However, this stack overflow answer documents it. Specifically, the tool must be given the type name of the serialized data and the schema file that defines the type. It reads the serialized data from stdin, so the binary file must be redirected into the program. Annoyingly, if the schema file is outside the current directory (the path contains a slash), then that folder name must be given both in the schema file path, and with the --proto_path option.

The following example deserializes gamedata_1.binpb using the schema-1.19.8.proto schema file in the home directory.

$ protoc --decode SaveGame --proto_path=$HOME $HOME/schema-1.19.8.proto <gamedata_1.binpb
MapSizeX: 1024000
MapSizeY: 256
MapSizeZ: 1024000
Seed: 1727735926
LastEntityId: 82
ModData {
  key: "BlockIDs"
  value: "\n\007\010\000\022\003air\n\
...

Wondering where some links have gone?
The modding navbox is going through some changes! Check out Navigation Box Updates for more info and help finding specific pages.