Tour of Rust tooling

You unlock this door with the key of imagination. Beyond it is another dimension - a dimension of sound, a dimension of sight, a dimension of mind. You're moving into a land of both shadow and substance, of things and ideas. You've just crossed over into the Twilight Zone!

Imagine, if you will, a foreign Rust codebase with a lot of issues, and you gotta fix the issues as quickly and as effectively as possible. Doing everything by hand is a waste of time.

For that, we need to become a little familiar with Rust tooling.

Luckily, we don't have to imagine and speak theoreticals. For this workshop, we have prepared one such small codebase:

https://github.com/luciusmagn/workshop-tools-testauskameli

This is a fork of a hobby open-source project I developed with a friend a while ago. Thanks to the additions of the fork, it now has all the good stuff:

  • Incomprehensible user documentation (it's in Finnish)
  • It does not compile out of the box
  • Both missing and redundant dependencies
  • When it does, it causes a segfault despite being completely safe Rust

Our goal for today is to get it to compile, fix the segfault and the style issues, all by using tools available to us in the Rust toolchain, with the addition of a couple community tools.

We are not allowed to disable large sections of code as the "make it compile" part of the task, or to forgo any functionality.

The following command should work on both debug and release build profiles when we are done without panic!s or segfaults:

# in root of repository,
# for every file in ./echo-tests
cat file | cargo run --bin cli

This command spins up the cli testing frontend of the Testauskameli bot. The "echo" handler of the bot takes any message starting with echo , stripping it, capitalizing the first letter and returning the result.

If you believe you got the hang of Rust's tooling, consider this a time challenge, and see how fast you can fix the repository. When you complete the challenge, send an email to the following address with your time and I will make a leader-board here:

lukas.hozda@braiins.cz

For the rest of us, let's get started.

The repository contains no sub-modules and we are working on the master branch, so you can just clone it normally:

git clone https://github.com/luciusmagn/workshop-tools-testauskameli

rustup

Before we get to using tools from the toolchain, we must first get acquainted with rustup, the tool that manages toolchains.

It is generally advisable to install Rust via rustup, as it allows you to have more than one toolchain at once without issue and mix and match toolchain components, which is often a requirement for serious Rust development.

If you prefer having Rust as a system package, see if your system has a package for rustup. Having a Rust system package directly is only good enough for non-developers who want to install software running on stable.

You can look up how to install rustup here:

https://rustup.rs/

During installation, Rustup will ask you what flavor of Rust you want to install, feel free to keep the defaults. This should install the standard profile of stable for your current host triplet.

The Rust toolchain has many components, here is an overview:

  • rustc — The Rust compiler and Rustdoc.
  • cargo — Cargo is a package manager and build tool.
  • rustfmt — Rustfmt is a tool for automatically formatting code.
  • rust-std — This is the Rust standard library. There is a separate rust-std component for each target that rustc supports, such as rust-std-x86_64-pc-windows-msvc.
  • rust-docs — This is a local copy of the Rust documentation. Use the rustup doc command to open the documentation in a web browser. Run rustup doc --help for more options.
  • rls — RLS is a language server that provides support for editors and IDEs.
  • clippy — Clippy is a lint tool that provides extra checks for common mistakes and stylistic choices.
  • miri — Miri is an experimental Rust interpreter, which can be used for checking for undefined-behavior.
  • rust-src — This is a local copy of the source code of the Rust standard library. This can be used by some tools, such as RLS, to provide auto-completion for functions within the standard library; Miri which is a Rust interpreter; and Cargo's experimental build-std feature, which allows you to rebuild the standard library locally.
  • rust-analysis — Metadata about the standard library, used by tools like RLS.
  • rust-mingw — This contains a linker and platform libraries for building on the x86_64-pc-windows-gnu platform.
  • llvm-tools-preview — This is an experimental component which contains a collection of LLVM tools.
  • rustc-dev — This component contains the compiler as a library. Most users will not need this; it is only needed for development of tools that link to the compiler, such as making modifications to Clippy.

Rustup by default installs the default profile, which contains the rustc, cargo, rust-std for your target, rustfmt and clippy.

Many editors and Rust-support plugins for IDEs require a language server to give you auto-suggestions and realtime linting.

On the nightly toolchain, you can install rust-analyzer, which is the new, shiny and performant language server protocol implementation. You also need to opt for the rust-src component:

rustup component add rust-src
rustup +nightly component add rust-analyzer-preview

(keep in mind that at the time you read this, rust analyzer might already be available on stable, make sure to check its installation page)

Before doing this, we need to add the nightly toolchain:

rustup toolchain add nightly

Although it is seldom used outside of language testers, you can also add the beta toolchain channel, and sometimes you might need a specific toolchain version, wherein for stable, you can use the numeric version such as 1.55, and with nightly, you need the date it was released (as nightlies are released almost daily), such as nightly-2020-07-27.

For this project, we need the default profile and the nightly toolchain should be installed as well.

Cross-compilation

Although we won't be using this for this project, it helps to know that rustup helps facilitate cross-compilation. You simply need to add a target, which will download the standard library for the given host, the Rust compiler is cross-compiling by default:

rustup target add riscv64gc-unknown-none-elf

Cross-compilation is essentially zero-effort if you have Docker on your system and install cross.

It is a wrapper for Cargo and it will do all the necessary work for cross-compilation.

Compiling the project

Once you have cloned the repository, switch directory to it and try compiling it:

cargo build --workspace

The --workspace flag (new name for the obsolete --all flag) makes sure you are trying to compile all crates in the workspace, regardless of which folder you are in.

On stable 1.60, I get many errors:

#![allow(unused)]
fn main() {
warning: `testauskameli` (lib) generated 1 warning
error: could not compile `testauskameli` due to 34 previous errors; 1 warning emitted
warning: build failed, waiting for other jobs to finish...
error: build failed
}

Let's quickly inspect them:

  • We see some warnings about unused imports
  • Most of the errors are related to async traits and how they are currently not supported
  • There is one error about the feature gate

Let's start by looking at the last one in particular:

#![allow(unused)]
fn main() {
error[E0554]: `#![feature]` may not be used on the stable release channel
 --> testauskameli/src/lib.rs:4:1
  |
4 | #![feature(async_closure)]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^
}

This is a clear indicator, that we must use the nightly channel for this project. To use nightly, we need to do one of the following:

  • rustup default nightly
  • instead of just cargo type cargo +nightly in every command. This is better if you are mainly developing on stable though

Therefore, I suggest the first option.

Let's see how it changes when we compile with nightly:

#![allow(unused)]
fn main() {
Some errors have detailed explanations: E0276, E0277, E0432, E0706.
For more information about an error, try `rustc --explain E0276`.
warning: `testauskameli` (lib) generated 1 warning
error: could not compile `testauskameli` due to 33 previous errors; 1 warning emitted
warning: build failed, waiting for other jobs to finish...
}

Now that's an error resolved roight there.

Fixing the dependencies

Let's take a look at one of the async trait errors:

#![allow(unused)]
fn main() {
error[E0706]: functions in traits cannot be declared `async`
   --> testauskameli/src/lib.rs:176:5
    |
176 |     async fn send(&self, content: RunnerOutput, context: &Self::Context) -> Result<()>;
    |     -----^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |     |
    |     `async` because of this
    |
    = note: `async` trait functions are not currently supported
    = note: consider using the `async-trait` crate: https://crates.io/crates/async-trait
}

The notes of the compiler errors and warnings tend to be quite helpful.

If we look into the Cargo.toml of testauskameli/, we can see that the library is indeed missing this dependency:

[package]
name = "testauskameli"
version = "0.1.0"
edition = "2021"
authors = ["Luukas Pörtfors <lajp@iki.fi>", "Lukáš Hozda <me@mag.wiki>"]

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
tracing = "0.1"
tracing-subscriber = "0.2"
async-process = "1.3.0"
anyhow = "1.0.53"
either = "1.6.1"
flume = "0.10.10"
tempfile = "3.3.0"
rand = "0.8.4"
itertools = "0.10.3"
vial = "0.1.9"
diesel = "1.4.8"
toml = "0.5.8"
imageproc = { version = "0.22.0", optional = true }
rusttype = { version = "0.9.2", optional = true }
image = { version = "0.23.14", optional = true }
regex = { version = "1.5.4", optional = true }
which = "4.2.4"

[features]
default = ["snippets"]
snippets = ["nomeme", "echo", "haskell"]
nomeme = ["imageproc", "rusttype", "image", "regex"]
echo = []
haskell = []

Now, we could go on crates.io, look up async-trait, find the current version and then edit the TOML.

But we also could not do that.

There exists an extremely handy utility called cargo-edit, which adds Cargo sub-commands for manipulating dependencies in an efficient and non-intrusive way.

The simplest way to install it is via Cargo:

cargo install cargo-edit

For figuring out if software installed with Cargo needs an update, you can use cargo-update (not to be confused with cargo update command which updates dependencies in Cargo.lock to the latest semver-compatible version), which adds the cargo install-update sub-command to Cargo. Check out its help text to figure out its usage.

Now that we have installed cargo-edit, we have the following new sub-commands at our disposal:

cargo add          # add or modify a dependency
cargo rm           # remove a dependency
cargo upgrade      # upgrade dependencies in Cargo.toml (as opposed to cargo update)
cargo set-version  # set crate version

Of course, the correct command here is cargo add.

cd testauskameli # if you are not in the folder already
cargo add async-trait

Let's try compiling again:

#![allow(unused)]
fn main() {
╭[RAM 57%] bos ~/ws-tooling-testauskameli [master][!] rs v1.62.0-nightlytax:
╰ 07:02 lh-thinkpad magnusi » cargo build --workspace
    Updating crates.io index
   Compiling futures v0.3.21
   Compiling tokio-rustls v0.23.2
   Compiling tokio-rustls v0.22.0
   Compiling h2 v0.3.12
   Compiling testauskameli v0.1.0 (/home/magnusi/ws-tooling-testauskameli/testauskameli)
   Compiling async-tungstenite v0.11.0
   Compiling cli v0.1.0 (/home/magnusi/ws-tooling-testauskameli/cli)
   Compiling hyper v0.14.17
   Compiling hyper-rustls v0.23.0
   Compiling reqwest v0.11.9
   Compiling serenity v0.10.10
   Compiling bot v0.1.0 (/home/magnusi/ws-tooling-testauskameli/bot)
    Finished dev [unoptimized + debuginfo] target(s) in 54.01s
}

No errors this time, very good.

Redundant dependencies

However, there might dependencies that are unused and only serve to prolong the compilation times (as Cargo doesn't know if a library is actually used, it builds it anyway).

Finding these is usually a painful process of reading the code and/or commenting out deps at random or with some sort of an algorithm in the respective Cargo.toml.

Luckily, although it is not definitive, there exists a solution.

The cargo-udeps utility does exactly what we need.

You install pretty much the same way:

cargo install cargo-udeps --locked

(cargo install by default does not respect Cargo.lock and chooses latest semver-compatible dependency versions by default. The author of this crate suggests using --locked which makes Cargo respect the lockfile, however, it works fine for me without it too. Your mileage may vary).

As of April 2021, udeps still requires to be ran with the nightly toolchain, although you can use it for stable projects as well. Therefore, it is recommended to invoke it with the +nightly Cargo modifier:

#![allow(unused)]
fn main() {
cargo +nightly udeps --all-targets --workspace
}

We've added the two flags to make sure that the run covers all the crates in the workspace, and all targets within each crate. Otherwise, we might get a false alarm if a crate is only used in an example, a binary target or integration tests.

TIP: It is a common mistake to mix up --all and --all-targets. As mentioned previously, --all is an obsolete alias for --workspace and using it does not cover all targets of your crate.

Let's see what we get:

unused dependencies:
`bot v0.1.0 (/home/magnusi/ws-tooling-testauskameli/bot)`
└─── dependencies
     └─── "openssl-sys"
`testauskameli v0.1.0 (/home/magnusi/ws-tooling-testauskameli/testauskameli)`
└─── dependencies
     ├─── "diesel"
     ├─── "toml"
     ├─── "tracing-subscriber"
     └─── "vial"
Note: They might be false-positive.
      For example, `cargo-udeps` cannot detect usage of crates that are only used in doc-tests.
      To ignore some dependencies, write `package.metadata.cargo-udeps.ignore` in Cargo.toml.

Ohoho, good providence. Diesel is a massive dependency, and OpenSSL is a compatibility nightmare, the fact that we can get rid of them is very good, and so is the fact that we can get rid of the other three unused dependencies.

Unfortunately, there is no way to fix the Cargo.tomls automagically, so you need to edit the two of them by hand. However, this is still a much better option that manually going through all dependencies.

Always check if all parts of your project still compile after removing deps, per the note, there might be false positives.

Also keep in mind that cargo-udeps is not definitive, but a best effort tool. It is fairly easy to have a false negative.

If I were to cargo add rayon to the testauskameli crate, then even though it isn't used in the library, udeps will not report it. This is because it will already be included in your binary because rayon is in the dependency tree of some of your dependencies, and udeps can't distinguish this scenario as "unused".

However, the downside of this is merely one line of TOML, since it is in the dependency tree, it will be compiled anyway and you are not saving time removing it.

Lints and style

Now, if all went well, the program compiles without a hitch. Better yet, it doesn't (shouldn't) print any warning either.

This however doesn't mean that the code does not have many issues and does not have a number of instances of poor style.

In Rust, we use a tool called Clippy (after the MS Word mascot) to provide a large set of very in-depth lints. Here is a total list of the lints in case you are curious: https://rust-lang.github.io/rust-clippy/master/

If you have installed the default profile of Rust, you should already have clippy, if not, adding it is as easy as adding any other toolchain component:

rustup component add clippy rust-src

And, nowadays, running is very simple also, it is just a cargo subcommand:

cargo clippy --all-targets --workspace

Well, and suddenly, with clippy, it does not compile anymore.

On my machine, I see 1 error and 20 errors:

#![allow(unused)]
fn main() {
warning: `testauskameli` (lib test) generated 20 warnings (1 duplicate)
error: could not compile `testauskameli` due to previous error; 20 warnings emitted
warning: build failed, waiting for other jobs to finish...
warning: `testauskameli` (lib) generated 20 warnings (19 duplicates)
error: could not compile `testauskameli` due to previous error; 20 warnings emitted
}

That's a lot of warnings to fix by hand (unless you are looking for ways how to spend your time).

It is time to introduce the next utility: rustfix via the commands cargo fix and cargo clippy --fix.

This is essentially the same utility, except the former fixes general Rust warnings and the latter Clippy lints.

You see, for many more style-related warnings, the Rust toolchain provides what it considers an absolutely correct and machine applicable suggestion. Therefore, it can be applied with a machine if you so desire.

This is what rustfix is for. We do not have any regular warnings, so we shall reach after the clippy invocation:

#![allow(unused)]
fn main() {
cargo clippy --fix --workspace --all-targets --allow-dirty
}

(only include the last flag if you are too lazy to commit your changes thus far or have a reason not to do so in the first place)

Now we are left with 1 error and 6 warnings, which is much more manageable. This is what we are left with:

#![allow(unused)]
fn main() {
╭[RAM 56%] bos ~/ws-tooling-testauskameli [master][!⇡] rs v1.62.0-nightlytax:
╰ 08:03 lh-thinkpad magnusi took 5s » cargo clippy --all-targets --workspace
    Checking testauskameli v0.1.0 (/home/magnusi/ws-tooling-testauskameli/testauskameli)
warning: called `ok().expect()` on a `Result` value
   --> testauskameli/src/cmd.rs:108:25
    |
108 |               proc_limit: env::var("KAMELI_PROCESSLIMIT")
    |  _________________________^
109 | |                 .map_or(Ok(1), |s| s.parse())
110 | |                 .ok()
111 | |                 .expect("BUG: impossible"),
    | |__________________________________________^
    |
    = note: `#[warn(clippy::ok_expect)]` on by default
    = help: you can call `expect()` directly on the `Result`
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ok_expect

warning: statement with no effect
  --> testauskameli/src/snippets/echo.rs:31:9
   |
31 |         t[0];
   |         ^^^^^
   |
   = note: `#[warn(clippy::no_effect)]` on by default
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#no_effect

warning: the operation is ineffective. Consider reducing it to `17`
  --> testauskameli/src/snippets/nomeme.rs:78:45
   |
78 |                     let e = min(text.len(), 17 + 0);
   |                                             ^^^^^^
   |
   = note: `#[warn(clippy::identity_op)]` on by default
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

warning: the operation is ineffective. Consider reducing it to `(17 - k.len()) / 2`
  --> testauskameli/src/snippets/nomeme.rs:80:29
   |
80 |                     let x = (17 - k.len()) / 2 + 0;
   |                             ^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

error: written amount is not handled
  --> testauskameli/src/utils.rs:74:5
   |
74 |     f.write(&[1]).expect("darn");
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[deny(clippy::unused_io_amount)]` on by default
   = help: use `Write::write_all` instead, or handle partial writes
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unused_io_amount

warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> testauskameli/src/lib.rs:232:82
    |
232 | ...                   .send(RunnerOutput::WrongUsage(err.to_string()), &&context)
    |                                                                        ^^^^^^^^^ help: change this to: `&&context`
    |
    = note: `#[warn(clippy::needless_borrow)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
}

Clippy does not fix these issues automatically because they require inspection and might not be correct.

Let's go through them one by one.

#![allow(unused)]
fn main() {
warning: called `ok().expect()` on a `Result` value
   --> testauskameli/src/cmd.rs:108:25
    |
108 |               proc_limit: env::var("KAMELI_PROCESSLIMIT")
    |  _________________________^
109 | |                 .map_or(Ok(1), |s| s.parse())
110 | |                 .ok()
111 | |                 .expect("BUG: impossible"),
    | |__________________________________________^
    |
    = note: `#[warn(clippy::ok_expect)]` on by default
    = help: you can call `expect()` directly on the `Result`
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ok_expect
}

If you inspect the file, you see you can easily remove the .ok(), it's useless and probably a leftover from refactoring.

#![allow(unused)]
fn main() {
warning: statement with no effect
  --> testauskameli/src/snippets/echo.rs:31:9
   |
31 |         t[0];
   |         ^^^^^
   |
   = note: `#[warn(clippy::no_effect)]` on by default
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#no_effect
}

An errant statement like this usually means an issue of the "I typed the wrong thing" variety, however, in this case, it's actually not needed and we can just remove it.

#![allow(unused)]
fn main() {
warning: the operation is ineffective. Consider reducing it to `17`
  --> testauskameli/src/snippets/nomeme.rs:78:45
   |
78 |                     let e = min(text.len(), 17 + 0);
   |                                             ^^^^^^
   |
   = note: `#[warn(clippy::identity_op)]` on by default
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

}

Useless addition, it might have been 17 + offset_or_cfg_val, but now, the +0 is redundant. Delete it mercilessly (as suggested by clippy's lint documentation).

#![allow(unused)]
fn main() {
warning: the operation is ineffective. Consider reducing it to `(17 - k.len()) / 2`
  --> testauskameli/src/snippets/nomeme.rs:80:29
   |
80 |                     let x = (17 - k.len()) / 2 + 0;
   |                             ^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op
}

The same thing as before

#![allow(unused)]
fn main() {
error: written amount is not handled
  --> testauskameli/src/utils.rs:74:5
   |
74 |     f.write(&[1]).expect("darn");
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[deny(clippy::unused_io_amount)]` on by default
   = help: use `Write::write_all` instead, or handle partial writes
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unused_io_amount
}

Usually, it is a good idea to handle these, as clippy suggests. However, it is not necessary in this case, and it is also related to a part of code which is objectively extremely wicked, shockingly evil and vile. We will get back to this section of code later, for now, let's fix this issue by explicitly ignoring the value:

#![allow(unused)]
fn main() {
let _ = f.write(&[1]).expect("darn");
}

And here comes the final issue:

#![allow(unused)]
fn main() {
warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> testauskameli/src/lib.rs:232:82
    |
232 | ...                   .send(RunnerOutput::WrongUsage(err.to_string()), &&context)
    |                                                                        ^^^^^^^^^ help: change this to: `&&context`
    |
    = note: `#[warn(clippy::needless_borrow)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
}

I admit this issue is a bit more contrived than the others, but it is an issue that happens commonly, mostly when you have an my_string_literal: &str and you want to pass it to a function like fn do_something_with_str(s: &str) and instinctively do do_something_with_str(&my_string_literal). As clippy suggests, this reference is useless since the compiler derefs it immediately.

If we run clippy, we should now get no errors and warnings (unless new clippy lints have been introduced since the release of this guide).

Debugging

Now, it is time to dispense with the final beast.

If we run all the echo tests, we see some strange stuff:

╭[RAM 57%] bos ~/ws-tooling-testauskameli [master][!⇡] rs v1.62.0-nightlytax:
╰ 07:57 lh-thinkpad magnusi took 8s » for f in echo-tests/*
                                          cat $f | cargo run --bin cli
                                      end
   Compiling testauskameli v0.1.0 (/home/magnusi/ws-tooling-testauskameli/testauskameli)
   Compiling cli v0.1.0 (/home/magnusi/ws-tooling-testauskameli/cli)
    Finished dev [unoptimized + debuginfo] target(s) in 4.45s
     Running `target/debug/cli`
Output:

Hello, world
    Finished dev [unoptimized + debuginfo] target(s) in 0.14s
     Running `target/debug/cli`
Output:

#apital
    Finished dev [unoptimized + debuginfo] target(s) in 0.14s
     Running `target/debug/cli`
    Finished dev [unoptimized + debuginfo] target(s) in 0.14s
     Running `target/debug/cli`
Output:

In the first age, in the first battle, when the shadows first lengthened, one stood. Burned by the embers of Armageddon, his soul blistered by the fires of Hell and tainted beyond ascension, he chose the path of perpetual torment. In his ravenous hatred he found no peace; and with boiling blood he scoured the Umbral Plains seeking vengeance against the dark lords who had wronged him. He wore the crown of the Night Sentinels, and those that tasted the bite of his sword named him... the Doom Slayer.
    Finished dev [unoptimized + debuginfo] target(s) in 0.13s
     Running `target/debug/cli`
thread 'tokio-runtime-worker' panicked at 'attempt to subtract with overflow', testauskameli/src/snippets/echo.rs:30:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: JoinError::Panic(...)', cli/src/main.rs:49:16
fish: Process 11647, 'cargo' from job 1, 'cat $f | cargo run --bin cli' terminated by signal SIGSEGV (Address boundary error)

Oh boy, that doesn't sound good. We have one panic and one segfault, meaning that two tests fails.

(An apt observer might notice that the Capital test is a bit suspicious also)

For debugging, we have two options:

  • Use debugging tools provided by the standard library
  • Use a debugger

Debugging in the standard library

We won't be doing this in this case, since we don't even know where the issue is. Instead we will be using a debugger to get a bearing on where the issue occurs. However, let's briefly introduce some of the most common tools in the standard library also, for the sake of completeness

The Debug trait

This is a derivable trait that allows printing a type with the debug formatting string {:?}. This trait is implemented for many of standard library traits and you can derive it on your types if all the types they contain implement Debug also:

#![allow(unused)]
fn main() {
#[derive(Debug)]
struct MyNumber(usize);

println!("An example: {:?}", MyNumber(42));
}

This works well with the following.

The dbg!() macro

This macro prints and returns the value of a given expression for quick and dirty debugging.

#![allow(unused)]
fn main() {
let a = 2;
let b = dbg!(a * 2) + 1;
//      ^-- prints: [src/main.rs:2] a * 2 = 4
assert_eq!(b, 5);
}

Keep in mind, if you put in an owned type that isn't Copy, it might not be available, so if you want to use this macro as a free-standing statement, make sure to take the parameters by reference.

Inspecting iterators

Iterators are extremely common in Rust and it might be helpful to be able to check out how the values are looking at different stages of an iterator pipeline.

For this, the .inspect() method has been created. It gives a reference to current value, expecting a closure that does not return anything (only a unit), and it doesn't (or rather shouldn't, don't abuse interior mutability!) modify the values in any way.

#![allow(unused)]
fn main() {
let a = [1, 4, 2, 3];

// this iterator sequence is complex.
let sum = a.iter()
    .cloned()
    .filter(|x| x % 2 == 0)
    .fold(0, |sum, i| sum + i);

println!("{}", sum);

// let's add some inspect() calls to investigate what's happening
let sum = a.iter()
    .cloned()
    .inspect(|x| println!("about to filter: {}", x))
    .filter(|x| x % 2 == 0)
    .inspect(|x| println!("made it through filter: {}", x))
    .fold(0, |sum, i| sum + i);

println!("{}", sum);
}

Let's get to the debugger, though.

Debugging Rust with a debugger

You might (not) be surprised to hear that Rust supports the classic C/C++ debuggers, gdb and lldb. Personally, I am more familiar with gdb, but both should work just fine.

However, you might notice that Rust symbols in gdb are poorly readable, as they are de-sugared and contain many things that are implementation details.

For this reason, Rust provides two wrappers for these debuggers, respectively. That is rust-gdb and rust-lldb. These are thin wrappers that will preload scripts that resugar the symbols for Rust, making it readable and easier to debug.

Both wrappers should pre-installed and included in every Rustup profile, but make sure you have installed the underlying debuggers as well via your system's package manager or however you wish.

Well, first, let's use movie magic to figure out that the test causing the segfault is echo-tests/empty. Let's try running just it:

╭[RAM 59%] bos ~/ws-tooling-testauskameli [master][!⇡] rs v1.62.0-nightlytax:
╰ 08:53 lh-thinkpad magnusi » cat echo-tests/empty | cargo run --bin cli
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `target/debug/cli`
fish: Process 3095, 'cargo' from job 1, 'cat echo-tests/empty | cargo ru…' terminated by signal SIGSEGV (Address boundary error)

Yep, that's it.

The debug build is in the target/debug folder and the executable is called cli.

Let's try running it in GDB:

#![allow(unused)]
fn main() {
╭[RAM 59%] bos ~/ws-tooling-testauskameli [master][!⇡] rs v1.62.0-nightlytax:
╰ 08:54 lh-thinkpad magnusi took 4s » rust-gdb target/debug/cli
GNU gdb (GDB) 11.2
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from target/debug/cli...
(gdb) run < echo-tests/empty
Starting program: /home/magnusi/ws-tooling-testauskameli/target/debug/cli < echo-tests/empty
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7ffff7b8f640 (LWP 16243)]
[New Thread 0x7ffff798e640 (LWP 16244)]
[New Thread 0x7ffff778d640 (LWP 16245)]
[New Thread 0x7ffff758c640 (LWP 16246)]
[New Thread 0x7ffff738b640 (LWP 16247)]
[New Thread 0x7ffff718a640 (LWP 16248)]
[New Thread 0x7ffff6f89640 (LWP 16249)]
[New Thread 0x7ffff6d88640 (LWP 16250)]
[New Thread 0x7ffff6b87640 (LWP 16251)]

Thread 8 "tokio-runtime-w" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6f89640 (LWP 16249)]
testauskameli::snippets::echo::{impl#0}::try_or_continue::{async_block#0} () at testauskameli/src/snippets/echo.rs:30
30	        t[0] -= 32;
(gdb)
}

Now we precisely know, where the crash occurs, let's look at the surrounding area:

#![allow(unused)]
fn main() {
    async fn try_or_continue(&self, content: &str) -> Either<Runner, Mismatch> {
        let text = if let Some(start) = content.find("echo ") {
            content[start + 5..].to_string()
        } else {
            return Either::Right(Mismatch::Continue);
        };

        let t = utils::totally_safe_transmute::<&[u8], &mut [u8; 1]>(text.as_bytes());
        t[0] -= 32;

        Either::Left(Runner::new("echo", "test", || {
            info!("{} (echo)", text);
            Box::pin(async move { Ok(RunnerOutput::Output(text)) })
        }))
    }
}

Yeah, upon closer inspection, we see here that this is probably the most unsafe, incorrect and non-portable way to capitalize a letter in Rust.

We are essentially:

  • Taking a string and interpretting it as bytes
  • Taking the bytes and interpretting it as an array of one byte (regardless of the actual situation)
  • Taking the byte and deducting 32 from it, which is in the ASCII table the offset between capital and lowercase letters

When the string is empty, we are trying to write into memory that is not our own, producing a segfault.

Let's use reasonable rust to rewrite it:

#![allow(unused)]
fn main() {
    async fn try_or_continue(&self, content: &str) -> Either<Runner, Mismatch> {
        let mut text = if let Some(start) = content.find("echo ") {
            content[start + 5..].to_string()
        } else {
            return Either::Right(Mismatch::Continue);
        };

        if let Some(first) = text.get_mut(0..1) {
            first.make_ascii_uppercase();
        };

        Either::Left(Runner::new("echo", "test", || {
            info!("{} (echo)", text);
            Box::pin(async move { Ok(RunnerOutput::Output(text)) })
        }))
    }
}

Let's see how our test run looks now:

╭[RAM 59%] bos ~/ws-tooling-testauskameli [master][!⇡] rs v1.62.0-nightlytax:
╰ 09:10 lh-thinkpad magnusi took 15m34s » for f in echo-tests/*
                                              cat $f | cargo run --bin cli
                                          end
   Compiling testauskameli v0.1.0 (/home/magnusi/ws-tooling-testauskameli/testauskameli)
   Compiling cli v0.1.0 (/home/magnusi/ws-tooling-testauskameli/cli)
    Finished dev [unoptimized + debuginfo] target(s) in 2.02s
     Running `target/debug/cli`
Output:

Hello, world
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/cli`
Output:

Capital
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
     Running `target/debug/cli`
Output:


    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/cli`
Output:

In the first age, in the first battle, when the shadows first lengthened, one stood. Burned by the embers of Armageddon, his soul blistered by the fires of Hell and tainted beyond ascension, he chose the path of perpetual torment. In his ravenous hatred he found no peace; and with boiling blood he scoured the Umbral Plains seeking vengeance against the dark lords who had wronged him. He wore the crown of the Night Sentinels, and those that tasted the bite of his sword named him... the Doom Slayer.
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
     Running `target/debug/cli`
Output:

	  test

Looks good, even the panic disappeared! We have now successfully fixed the project with minimum programming and minimum effort. Tools are a great means of saving time during development and you are encouraged to look into them deeper.