1544b9ca07
This should be protocol backward compatible change (and breaking change in the source code) which should speed-up the process of gRPC data encoding / decoding. Also unify our CodecV2 usage. For https://github.com/siderolabs/talos/issues/9404 Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
zstd-dict
zstd-dict is a tool for generating a zstd dictionary used to compress resources containing partial or full Talos machine configurations, such as ClusterMachineConfig and ConfigPatch.
The dictionary is generated using a training set consisting of:
- Randomly generated Talos machine configuration YAMLs
- ArgoCD application manifests generated via the
helm templatecommand
Kubernetes manifests are included because they are commonly used in the inlineManifests section of Talos machine configurations.
This tool updates the client/pkg/compression/data/config.zdict dictionary file under the Omni project root. It also writes the input files used for training into the inputs/ directory.
The inputs/ directory can then be used as input to generate a dictionary using the zstd command-line tool.
To generate a dictionary via command line:
- Install
zstd - In the directory of this tool, run the following command:
export DICT_ID=1
zstd --train -r inputs -o ../../client/pkg/compression/data/config.$DICT_ID.zdict --dictID $DICT_ID --maxdict=64KB
Note: The dictionary trained via command line might produce different, potentially better results than the one generated by this tool. Compare the results of both dictionaries before deciding which one to use.
Note: When updating the dictionary, follow these steps to maintain backward compatibility:
- Increment the dictionary ID in this tool.
- Update the shell command above with the new dictionary ID.
- In the
client/pkg/compression/data/package, add the new dictionary file without removing the old ones. - Update the code in the compression package to handle multiple dictionary versions, ensuring that the decoder can still access and use the old dictionaries when needed.
These steps are crucial to maintain compatibility with data compressed using previous dictionary versions.