Using the Braiins crates registry
Recently, we set up our own crates registry. As we push more crates onto it, the necessity to familiarize yourself with it increases.
Currently, the registry running:
- production: https://registry.ii.zone
The registry is the kellnr implementation from BitFaltr. From the dependence fetching perspective, a registry works quite similarly to just getting your crates from crates.io.
You need to add a config for Cargo to your system (although it may
be already present in the repository, it will not be invoked with commands
that are unrelated to a particular crate such as cargo install).
# $HOME/.cargo/config.toml
[registries.braiins]
index = "sparse+http://registry.ii.zone/api/v1/crates/"
[source.braiins-cratesio]
registry = "sparse+http://registry.ii.zone/api/v1/cratesio/"
[source.crates-io]
replace-with = "braiins-cratesio"
Keep in mind that you can use the testing registry crates.io proxy if you want to help distribute the load.
This will make the registry available in your system. Now you can start adding dependencies from it, either by editing Cargo.toml, or using the appropriate command:
cargo add --registry braiins ii-clustering
Your system will now also use crates.io as proxied by kellnr. This can improve performance, and will reduce the bandwidth we create for crates.io (as it would be quite sad if we got cut off).
Preparing a crate for publishing
To be able to publish a crate, modify your Cargo.toml to include the
publish key, which will protect against publishing to crates.io, and
make sure all dependencies have either a crates.io version or registry
version (you can have both path and registry version at once).
[package]
name = "asset-server"
version = "0.1.0"
edition = "2021"
authors = ["Karel Vávra <karel.vavra@braiins.cz>"]
publish = ["braiins"]
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
anyhow = "1.0"
consul-async = { version = "0.3.0", registry = "braiins" }
consul-utils = { version = "0.2.0", registry = "braiins" }
ii-async-utils = { version = "0.3.0", registry = "braiins", default-features = false }
ii-metrics = { version = "0.2.2", registry = "braiins", features = ["scm_versioning"] }
ii-scm = { version = "0.2.0", registry = "braiins" }
protos = { version = "0.1.0", registry = "braiins" }
prost = "0.9"
maplit = "1.0.2"
parking_lot = { version = "0.12.1", features = ["deadlock_detection"] }
serde = { version = "1.0.114", features = [ "derive" ] }
structopt = "0.3"
tokio = { version = "1", features = ["full"] }
tracing = { version = "0.1" }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tonic = "0.6"
tonic-reflection = "0.3"
tonic-tracing-reload = { version = "0.1.0", registry = "braiins" }
warp = "0.3"
tokio-stream = "0.1.8"
futures = ">= 0.3.5"
h2 = "0.3"
Actually publishing the crate
We have chosen to make all crates managed by the CI, and by a CI account.
The actual version bumping and git operations are executed by cargo-bump, which does the following:
- bumps [
major,minor,patch,prerelease] version- the array above represents the legitimate values for the $VERSION_SPEC variable, which you have to configure to create your publishing jobs
- creates a commit and git tag
- a
--tag-prefixis pre-pended to names of both - if there is multiple crates in a repository, --tag-prefix is probably necessary
- a
There is a few peculiarities and specifics about the versioning behavior built on the job template below:
- The commit and tag are created by the user who triggered the pipeline
- If the jobs are not running on master, the last segment of the branch is added to the pre-release segment of the version.
- The pre-release version segment is pre-pended by the string
ci-. This is because this segment cannot start with a number, so for example jtr/feat/10407 would otherwise produce an invalid version
crate: ii-clustering: 1.2.14 ──────────┐ (supposing we don't want to bump X.Y.Z)
├───┬───── 1.2.14-ci-k-means-clustering.1
branch: lho/feat/k-means-clustering ───┘ │
│
│ (if missing, it is only added)
bump prerelease ───────┼───── 1.2.14-ci-k-means-clustering.1
│
│
│ (if present, last number is appended)
bump prerelease ───────┼───── 1.2.14-ci-k-means-clustering.2
│
│ (contrary to the more freeform semver)
│ (standard, prerelease version has to)
│ (satisfy the regex [a-zA-Z]\w+.\d+)
bump minor ───────┼───── 1.3.0-ci-k-means-clustering.1
│
│
│
bump major ───────┼───── 2.0.0-ci-k-means-clustering.1
│
│
│
bump patch ───────┼───── 2.0.1-ci-k-means-clustering.1
│
│
│
bump major ───────┼───── 3.0.0-ci-k-means-clustering.1
│
│
(bumping prerelease on master strips it) │
(without increasing any other segment) │
merge to branch master ┐ │
│ │
bump prerelease ┤ │
└──────┼───── 3.0.0
│
│
bump minor ───────┼───── 3.1.0
│
│
fork: lho/fix/precision-fix ┐ │
│ │
bump prerelease ┤ │
└──────┼───── 3.1.0-ci-precision-fix.1
│
│
│
bump prerelease ───────┼───── 3.1.0-ci-precision-fix.2
│
│
│
bump prerelease ───────┼───── 3.1.0-ci-precision-fix.3
│
▀
Here is a template you can use in your .gitlab-ci.yml:
publish-<which-version>:
when: manual
stage: publish
variables
VERSION_SPEC: <which-version>
before_script:
- !reference [.cargo-publish, before_script]
script:
- # if the crate being published is not in root
#- cd <crate-folder/>
- !reference [.cargo-publish, script]
The following variables have to be defined:
$VERSION_SPEC : one_of[major, minor. patch, pre-release]
$CRATES_REGISTRY_SSH_KEY : priv key of crates registry deploy key
(name might differ in monorepo)
$CARGO_REGISTRIES_BRAIINS_TOKEN: an auth token for the ci account
These must be readable. Your repo must also be able to accept commits
in the format of <name>?-<semver-version>.
You can use the following regex to combine it with rejecting improperly formatted commits:
^(^(fixup![ ])*(build|ci|docs|feat|fix|perf|refactor|style|test|chore)(\([a-z]+\)){0,1}:\s.+\s#\S+|(([A-Za-z_][A-Za-z0-9_-]*-)?(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?))$
Yes, the regex is long and ugly, but so is the SemVer specification.
On the side of Rust configuration, the following entries in the cargo config are necessary:
# .cargo/config.toml
[registries.braiins]
index = "sparse+http://registry.ii.zone/api/v1/crates/"
[source.braiins-cratesio]
registry = "sparse+http://registry.ii.zone/api/v1/cratesio/"
[source.crates-io]
replace-with = "braiins-cratesio"
You can add these entries to your $HOME/.cargo/config.toml config as well.
Also make sure to limit which registries a package can be published to:
[package]
# ... other fields
publish = ["braiins"]
You can install cargo-bump easily from the Braiins registry:
$ cargo install --registry braiins cargo-bump
Note: this version is different form the one in the crates.io registry, and that one will not work.
Publishing to monorepo
Publishing to the monorepo is quite similar, except to get access to the publishing
jobs, you need to push to the branch publish+<branchname>. This will spawn
a special pipeline which only contains the publishing jobs. This will check out
<branchname>, so it is not important what's on the publish branch.
The same jobs as specified above will be available for every package specified in
the file .ci-bump.toml:
Add entries like this:
[[packages]]
name = "name-of-crate"
path = "path/to/contain/folder"
This file is processed by ci-bumper, which dynamically generates a template
based on its contents. The template used by this crate can be found in the root
of the monorepo at /.ci-bump-template.yml. The generated yaml can be inspected
as the artifact .ci-bump.yml of the generate-publish-jobs CI job in the publish
pipeline.
The complete process is as follow:
- Make sure the crate you want to publish is mentioned in
.ci-bump.toml - Merge the changes you want published into master (unless you want to publish a special prerelease version)
- Open a MR from
publish+mastertomaster, updating the branch - This should create a publish pipeline, run the correct publish job, such as
p-example-major - Merge the MR
If the prerelease job is ran on the publish+master branch, it will strip any prerelease
suffix and try to publish that version. On any other branch, jobs will add a numbered prerelease suffix
based on the name of the branch.
Necessary care
Make sure that you follow semver to the best of your ability when versioning. The Semantic Versioning specification can be found here:
You should also make sure that you do not publish code you do not want to be published, as it is quite difficult to delete versions from Rust crate registries. As a matter of fact, it is an unsupported operation and has to be done by hand.
If for any reason you want to find out if your crate is publishable ahead of time,
use the cargo package command.
If for any reason you have to publish a crate by hand, always append the
--registry braiins flag, even if with a correct package.publish in the manifest,
Cargo would deduce the braiins registry. This is because if this key is accidentally
missing (perhaps because it simply isn't there or because you made a typo, in which case
Cargo would just throw a warning about an unknown key), you could publish the package
to crates.io and we would be screwed, as it would be very, very difficult getting
it off of there.
Troubleshooting
Here is a couple reasons you might be unable to publish a crate version to the registry:
- Version with the same number is already there
- Your dependencies are incorrect (missing registry versions)
- You have a caret dependency
- Your crate depends on files which are not specified in the
package.includekey - You are trying to publish a package that's too big because you forgot to remove unrelated
artifacts by .gitignore or by the
package.excludekey - Your crate depends on files which are outside its folder, that's trouble because the crate can't be packaged properly. You have to rewrite the crate.