Tools by buf.build

As your project grows larger, managing your interfaces correctly becomes an everincreasing priority. If you are using gRPC, your interfaces are stored in Protocol Buffer files, which become absolutely critical as that is the often the only different parts of the system care about. Improper care can create a mess and make it more difficult to manage your API.

In Braiins, we recently started looking into and cautiously implementing tools developed by Buf Technologies, which we shall dub buftools.

These tools are available here:

https://buf.build/

Installation method depends on your operating system. For Arch, buf can be found in AUR: https://aur.archlinux.org/packages/buf

The project cites the following as its goals:

  • API designs are often inconsistent: Writing maintainable, consistent Protobuf APIs isn't as widely understood as writing maintainable REST/JSON-based APIs. With no standards enforcement, inconsistency can arise across an organization's Protobuf APIs, and design decisions can inadvertently affect your API's future iterability.
  • Dependency management is usually an afterthought: Protobuf files are vendored manually, with an error-prone copy-and-paste process from GitHub repositories. Before the BSR, there was no centralized attempt to track and manage around cross-file dependencies. This is analogous to writing JavaScript without npm, Rust without cargo, Go without modules, and all of the other programming language dependency managers we've all grown so accustomed to.
  • Forwards and backwards compatibility is not enforced: While forwards and backwards compatibility is a promise of Protobuf, actually maintaining backwards-compatible Protobuf APIs isn't widely practiced, and is hard to enforce.
  • Stub distribution is a difficult, unsolved process: Organizations have to choose to either centralize their protoc workflow and distribute generated code, or require all service clients to run protoc independently. Because there is a steep learning curve to using protoc (and the associated protoc plugins) in a reliable manner, organizations often struggle with distributing their Protobuf files and stubs. This creates substantial overhead, and often requires a dedicated team to manage the process. Even when using a build system like Bazel, exposing APIs to external customers remains problematic.
  • The tooling ecosystem is limited: Many user-friendly tools exist for REST/JSON APIs today. On the other hand, mock server generation, fuzz testing, documentation, and other daily API concerns are not widely standardized or user friendly for Protobuf APIs. As a result, teams regularly reinvent the wheel and build custom tooling to replicate the JSON ecosystem.

While buftools contain a new protobuf compiler, which is faster, buf doesn't yet support Rust, so we do not use it. However, for us, the greatest benefit buf brings is the ability to lint protocol buffer files (including gRPC definitions) to enforce better API design and structure.

Buf also provides a schema registry, which works similarly to Cargo crates or PyPI, and essentially packages and versions protobufs. We haven't implemented it yet, but the jury is still out on that.

Buf as a linter

Invoking buf as a linter is done by supplying the lint subcommand to buf. In order to be able to use the tool, you need to organize the protobuf files you want to lint into a Buf module.

This can be done either with the buf mod init command, or you can also write the buf.yaml manifest by hand.

Here is an example from the monorepo, of how this manifest might end up looking:

# buf.yaml
version: v1

lint:
  use:
    - DEFAULT
    - RPC_NO_CLIENT_STREAMING
    - PACKAGE_NO_IMPORT_CYCLE
  except:
    - SERVICE_SUFFIX
    - RPC_PASCAL_CASE
    - PACKAGE_VERSION_SUFFIX
    - PACKAGE_DIRECTORY_MATCH
    - DIRECTORY_SAME_PACKAGE
    - RPC_REQUEST_STANDARD_NAME
    - RPC_RESPONSE_STANDARD_NAME
    - RPC_REQUEST_RESPONSE_UNIQUE
  ignore:
    # 3rd party files
    - google/
    # FIXME: Ideally there shouldn't be any ignores
    #   The idea behind ignoring those is that we can start fixing the mess
    #   that is already present without breaking everything everywhere
    - asset-server/
    - clearing/
    - dynamo.proto
    - flux/
    - market-data/
    - mining/
    - twitch/
    - user-cache/

breaking:
  use:
    - WIRE

The version field is required, there is no default, and at the time of this writing, the only two options are v1 and v1beta.

This module manifest is intended for linting mainly:

  • The lint.use key specifies lints we explicitly want. By default, it is implicitly defined like this:
  use:
    - DEFAULT

Which turns on all of the default lints. This should be your starting point if you want to add lints that are not enabled by default.

  • The lint.except key disables specified lint and can be used pick out lints from groups you have enabled such as the DEFAULT lint group.
  • The lint.ignore key should be self explanatory

As the config indicates, we are fixing the lints continually, and so we have put it into ignore so that we don't start breaking things in the CI and elsewhere.

You can find a complete list of the lint rules and what they do here: https://docs.buf.build/lint/rules

The last three lines of the config indicate another great feature of buftools.

Breaking change detection

As outlined in a previous chapter, backwards and forwards compatibility is a critical selling point of gRPC and protocol buffers (and in general, of designing a good API in mission critical software). However, the fact that it is geared towards being compatible both ways doesn't mean it automatically is, breaking changes can still be introduced due to human error.

For this reason, buftools include a breaking change detector, which is invoked similarly as the linter, with buf breaking.

The configuration is very similar:

breaking:
  use:
    - FILE
  except:
    - RPC_NO_DELETE

The full list of breaking rules can be found here:

https://docs.buf.build/breaking/rules

Formatting

The buf format command will rewrite your protobuf files in accordance with an established style.

By default, the output will be printed to stdout, so you most likely want to use -w option, which modifies the files in-place:

buf format -w

You can also display a diff:

$ buf format -d

» buf format -d dynamo.proto
diff -u dynamo.proto.orig dynamo.proto
--- dynamo.proto.orig	2022-09-01 09:21:19.449675408 +0200
+++ dynamo.proto	2022-09-01 09:21:19.449675408 +0200
@@ -50,8 +50,7 @@

 syntax = "proto3";

-package
-  dynamo;
+package dynamo;

 message UpstreamSpecification {
   // Upstream URL; example: stratum+tcp://pool.net:7770

In combination with --exit-code flag, which exists with a non-zero code if there is a diff, this is particularly useful in the CI and other verification places to ensure that code committed into a repository is already properly formatted.

Style guide

Buf technologies also provides a very useful style guide for keeping your files up to a reasonable standard:

https://docs.buf.build/best-practices/style-guide

Calling gRPC endpoints from the CLI with help from buf

While buf itself does not directly provide functionality to call a gRPC call, it does support generation of file descriptor sets which are usable by gRPC CLI tools on the fly, especially if gRPC reflection is not available on the server.

There is two options: grpcurl and ghz.

grpcurl: https://github.com/fullstorydev/grpcurl

ghz: https://ghz.sh/

ghz is more of a load testing and benchmarking tool.

To use buf with grpcurl:

$ grpcurl -protoset <(buf build -o -) ...

And to use ghz:

$ ghz --protoset <(buf build -o -) ...