CI/CD - Single projects

{
  inputs = {
    crane = {
      url = "github:ipetkov/crane";
      inputs = {
        flake-utils.follows = "flake-utils";
        nixpkgs.follows = "nixpkgs";
      };
    };
    fenix = {
      url = "github:nix-community/fenix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "nixpkgs/nixos-unstable";
  };

  outputs = { self, crane, fenix, flake-utils, nixpkgs }:
    flake-utils.lib.eachDefaultSystem (system: {
      packages.default =
        let
          craneLib = crane.lib.${system}.overrideToolchain
            fenix.packages.${system}.minimal.toolchain;
        in

        craneLib.buildPackage {
          src = ./.;
          pname = "docker-spider";
        };
    });
}

The flake.nix File

1. Inputs: The file specifies several inputs, including crane, fenix, flake-utils, and nixpkgs. These inputs are likely dependencies or tools required for your project. Each of these inputs has a URL, indicating where Nix can fetch them, and some have additional input dependencies themselves.

2. Outputs: The outputs section is where the actual build and deployment configurations are defined. This file uses flake-utils.lib.eachDefaultSystem, suggesting it's set up to handle multiple systems (or architectures). This is a common practice for building packages that are compatible with different operating systems or hardware architectures.

3. Build and Deployment Logic: Inside the outputs, you would typically define how your application or package is built and possibly how it should be deployed. This can include custom build scripts, packaging instructions, and more.

Creating a .gitlab-ci.yml for GitLab CI

To use this flake.nix in a GitLab CI pipeline, you need to create a .gitlab-ci.yml file that tells GitLab CI how to build and test your project using Nix. Here's a basic example:

stages:
  - build
  - test

build-job:
  stage: build
  image: nixos/nix
  script:
    - nix build
  artifacts:
    paths:
      - result

Explanation:

• Stages: Defined two stages, build and test. You can add more stages like deploy if needed.
• Build Job:
  • Uses the nixos/nix Docker image, which comes with Nix pre-installed.
  • Runs nix build to build your project using the flake.nix file.
  • Artifacts (result) are saved and can be used in later stages.
• Test Job:
  • Also uses the nixos/nix image.

This setup does no caching. If you can and have access to it, you can use caching in our CI by using the correct image and correct runner tags:

.image: &image
  image: docker.ii.zone/pool/main/nix-ci:latest

  tags:
    - nix

This setup will build and test your Nix project on GitLab CI every time you push changes to your repository. Make sure to adjust the test command and add any additional stages or jobs as needed for your specific project.

Multiple packages in a single project: Frontend Flake

{

  # this is the description of the flake. It is just a bit of metadata, not very important for Braiins usecase
  description = "Frontend Flake";
  # when using `nix develop`, this is what will be prepended
  nixConfig.bash-prompt-prefix = "(frontend) ";

  # dependencies of the frontend falek
  inputs = {
    # flakes
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable"; # We are using the unstable Nix channel here because Pepa needs a new NodeJS version
    flake-utils.url = "github:numtide/flake-utils";        # A library that provides a wrapper that projects your flake to different architectures
                                                           # among other things

    # a library used for filtering out source code
    # we use this to ensure that only what is needed is built
    # If we had projects completely separated into folders, we wouldn't need this
    # however, there is a common core between the JS packages
    nix-filter = {
      url = "github:numtide/nix-filter";
    };

    # this consumes a Python Poetry metadata to create a Nix environment to run and build our Python
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.flake-utils.follows = "flake-utils";
    };

    # our repo that provides docker base images for Nix
    # we use this for the debian basis
    base-images = {
      url = "git+ssh://git@gitlab.ii.zone/pool/base-images.nix";
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.flake-utils.follows = "flake-utils";
      inputs.nix-filter.follows = "nix-filter";
    };

    # adds treefmt support, so we can format everything via nix
    treefmt-nix = {
      url = "github:numtide/treefmt-nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    # git repos that are not flakes yet
    #
    # locale dependencies
    braiins-os-locale = {
      url = "git+https://github.com/braiins/bos-i18n.git";
      flake = false;
    };
    web-pool-locale = {
      url = "git+https://github.com/slushpool/web-pool-i18n.git";
      flake = false;
    };
    toolbox-locale = {
      url = "git+https://github.com/braiins/toolbox-i18n.git";
      flake = false;
    };
    insights-locale = {
      url = "git+https://github.com/braiins/insights-i18n.git";
      flake = false;
    };
  };
  outputs =
    # flakes
    { self                    # | outputs are a function that takes an attribute set
    , nixpkgs                 # | containing all of the inputs as parameter
    , flake-utils             # |
    , nix-filter              # | and returns an attrset with the actual outputs
    , poetry2nix              # |
    , base-images             # | in this case, the final attrset is created by
    , treefmt-nix             # | flake-utils
      # git repos from github # |
    , braiins-os-locale       # |
    , web-pool-locale         # | any Git repository can be an input to a flake,
    , toolbox-locale          # | doesn't have to be a flake. You can refer to files
    , insights-locale         # | from these repositories same as you would to files
    , ...                     # | of a derivation in a flake
    }:
    flake-utils.lib.eachDefaultSystem
      (localSystem:
      let
        # starting node version   # |
        NODE_VERSION = "19.6.0";  # |

        # Nixpkgs imports
        #
        # - Set up pkgs for current system
        # - Import lib and stdenv
        pkgs = (import nixpkgs) { inherit localSystem; }; #
        inherit (pkgs) lib;                               #
        inherit (pkgs) stdenv;                            #

        # Outside dependencies
        #
        # - node dependencies
        # - python dependencies
        # - git repositories
        git-deps = (import ./nix/git-deps.nix) {
          inherit braiins-os-locale web-pool-locale toolbox-locale insights-locale stdenv;
        };
        inherit (git-deps) makeGitDeps;

        yarn-deps = (import ./nix/yarn-files.nix) {
          inherit stdenv pkgs nix-filter;
        };
        inherit (yarn-deps) yarnFiles;

        python-env = (import ./nix/python-env.nix) {
          inherit stdenv pkgs nix-filter localSystem poetry2nix;
        };
        inherit (python-env) pythonEnv;

        # Internal definitions' imports
        config = (import ./nix/config.nix) {
          inherit stdenv pkgs nix-filter;
        };

        unpatchedYarnPackages = (import ./nix/packages.nix) {
          inherit stdenv pkgs nix-filter config makeGitDeps yarnFiles pythonEnv;
        };

        yarnPackages = with builtins; let
          patchGitInfo = package: stdenv.mkDerivation {
            name = "patched-${package.name}";
            buildInputs = [ pkgs.gnused ];
            phases = [ "buildPhase" ];
            buildPhase =
              let
                placeholder_hash = "REPLACE_THIS_WITH_GIT_HASH";
                placeholder_date = "REPLACE_THIS_WITH_GIT_DATE";
                head = substring 0 16 (self.rev or "0000000000000000");
                date =
                  let
                    _ = self.lastModifiedDate or "YYYYMMDDhhmmss";
                    d = {
                      YYYY = builtins.substring 0 4 _;
                      MM = builtins.substring 4 2 _;
                      DD = builtins.substring 6 2 _;
                      hh = builtins.substring 8 2 _;
                      mm = builtins.substring 10 2 _;
                      ss = builtins.substring 12 2 _;
                    };
                  in
                  "${d.YYYY}-${d.MM}-${d.DD}_${d.hh}:${d.mm}:${d.ss}";
              in
              ''
                mkdir -p $out
                cp -r ${package}/* $out/
                cd $out

                # By default, the files are read-only, so we need to change that
                chmod a+w -R .

                echo "build info"
                echo "  head = ${head}"
                echo "  date = ${date}"
                #               ↓ files ↓ non-binary  ↓ containing predefined placeholder strings              ↓
                paths=($(find . -type f -exec grep -I1q "${placeholder_hash}\|${placeholder_date}" {} \; -print))

                # the array expansion is not a valid NIX syntax, so we need to escape
                # the variable interpolation pattern to let it be evaluated by bash
                for path in ''${paths[@]}; do
                  echo "Patching the file ''${path}"
                  sed -i 's|${placeholder_hash}|${head}|g' ''${path}
                  sed -i 's|${placeholder_date}|${date}|g' ''${path}
                done

                # Make the files read-only again
                chmod a-w -R .
              '';
          };
        in
        (mapAttrs (name: value: (patchGitInfo value)) unpatchedYarnPackages.normal);

        dockerImages =
          (import ./nix/docker-images.nix) {
            inherit stdenv pkgs nix-filter base-images yarnPackages;
          };

        # create a list combining all paths, so we can symlinkJoin them
        paths = with builtins; (map (key: getAttr key yarnPackages) (attrNames yarnPackages));

        apps =
          let
            wrapped_command = cmd: {
              type = "app";
              program =
                let
                  # Merge Linux-only deps with general deps (because of Mac, naturally)
                  buildInputs = config.buildInputs ++ (lib.optionals pkgs.stdenv.hostPlatform.isLinux config.linuxBuildInputs);
                  env_script = pkgs.writeShellScript "script.sh" ''
                    # GitLab CI section start
                    CIS_NAME="wraper_setup"
                    CIS_LABEL="Nix command env setup"
                    CIS_OPTS="[collapsed=true]"
                    echo -e "\033[0Ksection_start:$(date +%s):$CIS_NAME$CIS_OPTS\r\033[0K$CIS_LABEL"

                    # ENV
                    export TERM=${config.env.TERM}

                    # PATH
                    export PATH="${lib.concatStringsSep ":" (builtins.map (x: "${x}/bin") (buildInputs))}:$PATH"
                    export PATH="${pythonEnv}/bin:$PATH"
                    export PATH="${pkgs.ruff}/bin:$PATH"

                    echo "PATH:"
                    sed 's/:/\n - /g' <<< ":$PATH"

                    # Install local poetry env so that "poetry run" can pick it up
                    poetry install --no-root --quiet --no-interaction

                    echo -e "\nPython prefix when running through bare 'python'":
                    echo -e "  $(python -c 'import sys; print(sys.prefix)')\n"

                    echo -e "\nPython prefix when running through 'poetry run':"
                    echo -e "  $(poetry run python -c 'import sys; print(sys.prefix)')\n"

                    # GitLab CI section end
                    echo -e "\033[0Ksection_end:$(date +'%s'):$CIS_NAME\r\033[0K "

                    ${cmd} $@
                  '';
                in
                "${env_script}";
            };
          in
          {
            # Linters
            lint-proto = (wrapped_command "make proto-lint proto-format-diff");
            lint-js = (wrapped_command "make lint-js");
            lint-yarn = (wrapped_command "make lint-yarn");
            lint-styles = (wrapped_command "make lint-styles");
            lint-configs = (wrapped_command "make lint-configs");

            # Tests
            ci-test-static = (wrapped_command "make ci-test-static");
            ci-test-unit = (wrapped_command "make ci-test-unit");

            # util
            wrapped-make = (wrapped_command "make");
          };
      in
      {
        inherit apps;

        # Set formatter and enable packages by including them here.
        #
        # To add a docker image, go to
        # ./nix/docker-images.nix
        #
        # To add a package, go to
        # ./nix/packages.nix
        #
        # Keep in mind that you will have to define a new dependency to the docker images,
        # which you will have to pass in as a parameter (see insights-docker for an example).
        formatter = treefmt-nix.lib.mkWrapper pkgs {
          projectRootFile = "flake.nix";
          programs.nixpkgs-fmt.enable = true;
          programs.black.enable = true;
          settings.formatter.ruff = {
            command = pkgs.ruff;
            options = [ "check" ];
            includes = [ "*.py" ];
          };
        };
        packages = yarnPackages // (lib.optionalAttrs pkgs.stdenv.hostPlatform.isLinux dockerImages) // unpatchedYarnPackages.cypress // {
          default = pkgs.symlinkJoin {
            name = "all";
            inherit paths;
          };
          nodeDeps = yarn-deps.yarnFiles;
          pythonEnv = python-env.pythonEnv;
        };

        # Default X86 shell
        # You can create shells for other platforms and configurations
        devShells.default = pkgs.mkShell {
          # Default x86_64-linux shell
          # This shell will be invoked if you just type:
          #
          # ```
          # nix develop
          # ```
          #
          # You can create more shells following this example,
          # but you will need to call them something else
          buildInputs = config.buildInputs ++ [
            pkgs.nodejs_20
            pkgs.netcat-gnu
          ];

          shellHook = ''
            export PATH="${pythonEnv}/bin:$PATH"
            export PATH="${pkgs.ruff}/bin:$PATH"
            poetry env use "$(python -c 'import sys; print(sys.prefix)')/bin/python"
          '';

          inherit (config) env;
        };
      });
}

Multiple packages in a single project: Tooling Flake