pricehiller.com

Private Git Submodule Authentication and Nixos Rebuilds

Tue, 03 Mar 2026 00:00:00 GMT

A Quick Tip

If you are using a private git submodule as an input to a Nix flake and then use that input as part of a NixOS configuration you may end up seeing authentication errors when sudo nixos-rebuild switch is invoked. This is because in a clean working tree, Nix will attempt to resolve those submodules to their Git upstreams and check them out from there. sudo nixos-rebuild switch may drop your environment variable SSH_AUTH_SOCK and thus the private submodule can’t be checked out as your user’s ssh-agent can’t provide the credentials for authentication.

To resolve this you need sudo to include your user’s SSH_AUTH_SOCK environment variable into its execution environment.

Instead of using a plain sudo nixos-rebuild switch, use sudo --preserve-env=SSH_AUTH_SOCK nixos-rebuild switch. That should resolve any authentication issues (assuming your ssh-agent’s auth is valid for the upstream where the Git submodule exists of course).

You could also set this globally in your NixOS configuration without needing to specify --preserve-env for sudo every time via:

{...}
{
  security.sudo.extraConfig = ''
    Defaults env_keep+=SSH_AUTH_SOCK
  '';
}

See sudoers(8) for env_keep.

I ran into this issue and it sent me on a witch hunt to figure out what was wrong. I’m very interested as to why you don’t need to send along SSH_AUTH_SOCK to sudo when the working tree is dirty. My best guess is that the private submodule gets resolved as a path flake reference when the working tree is dirty and otherwise it gets treated as a full on git reference when it’s clean. So Nix doesn’t need authentication for the private repository in the first case (treated as a real path), but does when the input is coerced into a git reference. Annoying.

Anyhow — my hope is I’ve saved someone out there a small headache.

Nix Submodules Haven't Been Painful for a While

Sat, 07 Feb 2026 00:00:00 GMT

Context

Around a year and a half ago I wrote a post complaining about Nix and Git Submodules. That is no longer accurate and hasn’t been for a while. I’ve been kinda forgetting to write this despite a TODO item sitting in my backlog for quite a while.

Yeah, it’s been 301 days since I made a note to write this. I’m lazy I guess.

Wait, so what changed?

Since February 4th, 2025, you can set input.self.submodules = true; in a Nix Flake and it will include your submodules as part of the Nix builds, fetches, etc.

It had a few bugs here and there for a month or two afterwards where it occasionally broke, but for a good while now it’s been rock solid. All of my complaints about needing to pass .?submodules=1 to the end of various Nix commands were rendered invalid since then.

An Example

So let’s say you have a Nix Flake in a Git repository that depends on submodules. For example, let’s say you use Agenix, Sops-Nix, or some other secrets management that stores those secrets in the repository. Currently, the recommendation is to just encrypt those secrets and leave the encrypted files publicly visible. I think that came straight from the bad idea fairy.

Let’s say we wanted to be a little more cautious and stop Harvest now, decrypt later strategies from working against us. A good strategy then, is to store the secrets outside of the public repository in a separate, private, Git repository and then pull those secrets in by adding them as a Git submodule to the public repository. I’ve done exactly that in my Nix config, see the secrets submodule — it’s a private submodule so random folks can’t just download my secrets and wait until a possible vulnerability crops up in Age and then harvest everything I have in there.

So, what I do now is simply add self.submodules to my Nix Flake inputs and use the secrets as a path input, over here. This works great! Public users can still browse my slop repository with all its sharp edges and terrible abstractions without being able to copy down even encrypted secrets.

Here’s a snippet of that Flake if you don’t want to browse:

{
  inputs = {
(69 lines hidden)
    nix.url = "git+https://github.com/nixos/nix?shallow=1";
    deploy-rs.url = "github:serokell/deploy-rs";
    nixos-facter-modules.url = "github:nix-community/nixos-facter-modules";
    nixos-hardware.url = "github:NixOS/nixos-hardware/master";
    nixpkgs.url = "git+https://github.com/NixOS/nixpkgs?shallow=1&ref=nixos-unstable";

    nixpkgs-unstable.url = "git+https://github.com/NixOS/nixpkgs?shallow=1&ref=nixpkgs-unstable";
    nixpkgs-stable.url = "git+https://github.com/NixOS/nixpkgs?shallow=1&ref=nixos-25.11";

    harmonia.url = "github:nix-community/harmonia";
    nix-post-build-hook-queue = {
      url = "github:newam/nix-post-build-hook-queue";
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.treefmt.follows = "";
    };
    hyprland.url = "git+https://github.com/hyprwm/Hyprland?shallow=1";
    fenix = {
      url = "github:nix-community/fenix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    lanzaboote = {
      url = "github:nix-community/lanzaboote";
    };
    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    agenix = {
      url = "github:yaxitech/ragenix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    impermanence.url = "github:nix-community/impermanence";
    disko = {
      url = "github:nix-community/disko";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    blog = {
      url = "git+https://git.pricehiller.com/price/Blog";
    };
    treefmt-nix.url = "github:numtide/treefmt-nix";
    apple-emoji-linux.url = "github:samuelngs/apple-emoji-linux";
    neovim-nightly-overlay = {
      url = "github:nix-community/neovim-nightly-overlay";
      inputs = {
        nixpkgs.follows = "nixpkgs";
      };
    };
    rofi-tools = {
      url = "github:szaffarano/rofi-tools";
      inputs = {
        nixpkgs.follows = "nixpkgs";
      };
    };
    zsh-completions = {
      url = "github:zsh-users/zsh-completions";
      flake = false;
    };
    oisd-blocklist = {
      url = "github:sjhgvr/oisd";
      flake = false;
    };
    copyparty = {
      url = "github:9001/copyparty";
      inputs = {
        nixpkgs.follows = "nixpkgs";
      };
    };
    nixcord.url = "github:kaylorben/nixcord";

    self.submodules = true;
    secrets = {
      url = ./secrets;
      inputs = {
        nixpkgs.follows = "nixpkgs";
        agenix.follows = "agenix";
        treefmt-nix.follows = "treefmt-nix";
      };
    };
  };

(248 lines hidden)
  outputs =
    inputs@{ self, nixpkgs, ... }:
    let
      inherit (self) outputs;
      forAllSystems =
        function:
        nixpkgs.lib.genAttrs
          [
            "aarch64-linux"
            "i686-linux"
            "x86_64-linux"
            "aarch64-darwin"
            "x86_64-darwin"
          ]
          (
            system:
            function (
              import nixpkgs {
                inherit system;
                overlays = [
                  inputs.agenix.overlays.default
                  self.overlays.modifications
                  self.overlays.additions
                ];
              }
            )
          );
      treefmtEval = forAllSystems (pkgs: inputs.treefmt-nix.lib.evalModule pkgs ./treefmt.nix);
    in
    {
      formatter = forAllSystems (
        pkgs: treefmtEval.${pkgs.stdenv.hostPlatform.system}.config.build.wrapper
      );
      packages = forAllSystems (
        pkgs:
        let
          bootstrapISO = self.nixosConfigurations.bootstrapper.config.system.build.isoImage;
        in
        {
          inherit bootstrapISO;
          default = bootstrapISO;
        }
      );
      overlays = import ./overlays { inherit inputs; };
      apps = forAllSystems (pkgs: {
        default = {
          type = "app";
          meta.description = "Run nixos-rebuild switch";
          program =
            pkgs.writeShellApplication {
              name = "nixos-rebuild-wrapper";
              runtimeInputs = with pkgs; [
                nix-output-monitor
                nixos-rebuild
                hostname
              ];
              text = ''
                MSG="Switching to NixOS configuration: '$(hostname)'"
                HEADER=$(printf "%''${#MSG}s\n" | tr ' ' "=")
                echo
                echo "$HEADER"
                echo "$MSG"
                echo "$HEADER"
                echo
                sudo nixos-rebuild switch --flake ".#$(hostname)" --accept-flake-config |& nom
              '';
            }
            |> pkgs.lib.getExe;
        };
        repl = {
          type = "app";
          meta.description = "Launch interactive repl to inspect config";
          program =
            pkgs.writeShellApplication {
              name = "inspect-flake";
              runtimeInputs = with pkgs; [
                git
              ];
              text = ''
                nix repl --file "$(git rev-parse --show-toplevel)"/repl.nix
              '';
            }
            |> pkgs.lib.getExe;
        };
      });
      devShells = forAllSystems (pkgs: {
        default = pkgs.mkShell {
          packages = with pkgs; [
            age
            agenix
            age-plugin-yubikey
            nixos-rebuild
            nixos-install-tools
            inputs.deploy-rs.packages.${pkgs.stdenv.hostPlatform.system}.deploy-rs
          ];
          shellHook = ''
            export RULES="$PWD/secrets/secrets.nix"
          '';
        };
      });
      checks = forAllSystems (pkgs: {
        formatting = treefmtEval.${pkgs.stdenv.hostPlatform.system}.config.build.check self;
      });
      nixosConfigurations =
        let
          clib = (import ./lib { lib = nixpkgs.lib; });
        in
        {
          orion =
            let
              hostname = "orion";
            in
            nixpkgs.lib.nixosSystem {
              specialArgs = {
                inherit self;
                inherit inputs;
                inherit outputs;
                inherit hostname;
                inherit clib;
              };
              modules =
                let
                  age-secrets = {
                    config = inputs.secrets.secrets.${hostname};
                  };
                in
                [
                  ./modules/nixos/base-programs.nix
                  ./modules/nixos/btrfs-rollback.nix
                  ./modules/nixos/grafana-alloy.nix
                  ./modules/nixos/vector.nix
                  ./modules/nixos/logviewer.nix
                  ./modules/nixos/persistence.nix
                  ./modules/nixos/dns
                  inputs.home-manager.nixosModules.home-manager
                  {
                    home-manager = {
                      sharedModules = [
                        inputs.agenix.homeManagerModules.default
                        inputs.nixcord.homeModules.nixcord
                        age-secrets
                        ./modules/hm/link-file.nix
                      ];
                      backupFileExtension = "hm.backup";
                      extraSpecialArgs = {
                        clib = (import ./lib { lib = nixpkgs.lib; });
                        inherit inputs;
                      };
                      useGlobalPkgs = true;
                      useUserPackages = true;
                      users.price = import ./users/price/home.nix;
                    };
                  }
                  inputs.nix-post-build-hook-queue.nixosModules.default
                  inputs.nixos-facter-modules.nixosModules.facter
                  inputs.nixos-hardware.nixosModules.dell-xps-15-9530
                  inputs.lanzaboote.nixosModules.lanzaboote
                  inputs.impermanence.nixosModules.impermanence
                  inputs.agenix.nixosModules.default
                  inputs.disko.nixosModules.disko
                  {
                    config = {
                      nixpkgs.overlays = [
                        inputs.neovim-nightly-overlay.overlays.default
                        inputs.nix-post-build-hook-queue.overlays.default
                        self.overlays.modifications
                        self.overlays.additions
                      ];
                    };
                  }
                  age-secrets
                  ./hosts/${hostname}
                ];
            };
          luna =
            let
              hostname = "luna";
            in
            nixpkgs.lib.nixosSystem {
              specialArgs = {
                inherit self;
                inherit inputs;
                inherit hostname;
                inherit nixpkgs;
                inherit clib;
              };
              modules = [
                ./modules/nixos/btrfs-rollback.nix
                ./modules/nixos/mail.nix
                ./modules/nixos/grafana-alloy.nix
                ./modules/nixos/openssh.nix
                ./modules/nixos/base-programs.nix
                ./modules/nixos/vector.nix
                ./modules/nixos/persistence.nix
                ./modules/nixos/dns
                inputs.nixos-facter-modules.nixosModules.facter
                inputs.impermanence.nixosModules.impermanence
                inputs.agenix.nixosModules.default
                inputs.disko.nixosModules.disko
                inputs.harmonia.nixosModules.harmonia
                inputs.copyparty.nixosModules.default
                {
                  config = inputs.secrets.secrets.${hostname};
                }
                ./hosts/${hostname}
              ];
            };
          bootstrapper =
            let
              hostname = "bootstrapper";
            in
            nixpkgs.lib.nixosSystem {
              specialArgs = {
                inherit self;
                inherit inputs;
                inherit hostname;
                inherit nixpkgs;
                inherit clib;
              };
              modules = [
                ./modules/nixos/openssh.nix
                ./modules/nixos/base-programs.nix
                {
                  config = {
                    nixpkgs.overlays = [
                      inputs.neovim-nightly-overlay.overlays.default
                    ];
                  };
                }
                ./hosts/${hostname}
              ];
            };
        };
      deploy.nodes =
        let
          deploy-rs = inputs.deploy-rs;
        in
        {
          luna = {
            hostname = "luna.hosts.pricehiller.com";
            fastConnection = true;
            profiles.system = {
              sshUser = "root";
              path = deploy-rs.lib.x86_64-linux.activate.nixos outputs.nixosConfigurations.luna;
            };
          };
        };
    };
}

If you’re keeping your encrypted secrets in a publicly visible repository, consider vendoring those out to an external Git repository and pulling ‘em back in as a Git submodule.

Anyhow, I can now cross that way overdue item off my TODO list.

Using a Nix Attrset to specify Librewolf settings

Fri, 07 Feb 2025 00:00:00 GMT

A minor nuisance is discovered

I was tinkering with my Nix config and ended up going through my Librewolf configuration. I realized that the settings in the Librewolf Home Manager module do not support Nix attrsets. That bothered me.

For your eyes, here was how I had Librewolf configured during that session:

  # ... snip ...
  programs.librewolf = {
    enable = true;
    settings = {
      "webgl.disabled" = false;
      "privacy.clearOnShutdown.history" = false;
      "privacy.clearOnShutdown.downloads" = false;
      "privacy.clearOnShutdown.cookies" = false;
      "network.cookie.lifetimePolicy" = 0;
      "network.trr.mode" = 3;
      "network.trr.uri" = "https://dns.mullvad.net/dns-query";
      "network.trr.default_provider_uri" = "https://dns10.quad9.net/dns-query";
      "network.trr.strict_native_fallback" = false;
      "network.trr.retry_on_recoverable_errors" = true;
      "network.trr.disable-heuristics" = true;
      "network.trr.allow-rfc1918" = true;
    };
  };
  # ... snip ...

Notice all the string paths like "network.trr.mode", "network.trr.uri", and so on. Nix has these beautiful things called attrsets (in fact, the programs.librewolf stuff itself is an attrset), here’s an example:

{
  hello = {
    world = true;
    nested.path.item = false;
  };
}

Notice that the nested.path.item looks an awful lot like "network.trr.mode" and the other string paths in the Librewolf settings.

So! It got me thinking, I don’t really like repeating the same "network.trr.mode and whatnot over and over; furthermore, those strings make it a pain to organize those settings under their parents. Meaning I can’t just have all of Librewolf’s network settings hanging out under a network top level key.

I wanted to set those paths in an attrset — I needed to set those paths in an attrset.

What I wanted to do was set my Librewolf configuration like so:

  # ... snip ...
  programs.librewolf = {
    enable = true;
    settings = {
      webgl.disabled = false;
      privacy.clearOnShutdown = {
        history = false;
        downloads = false;
        cookies = false;
      };
      network = {
        cookie.lifetimePolicy = 0;
        trr = {
          mode = 3;
          uri = "https://dns.mullvad.net/dns-query";
          default_provider_uri = "https://dns10.quad9.net/dns-query";
          strict_native_fallback = false;
          retry_on_recoverable_errors = true;
          disable-heuristics = true;
          allow-rfc1918 = true;
        };
      };
    };
  };
  # ... snip ...

Here’s the key though, that attrset has to be converted back to the string paths we saw earlier as that’s what the Home Manager module requires for the settings.

Writing a function to convert an attrset to a flattened bunch of string paths

Here’s what I cooked up (probably still pink and raw in the center, eat at your own risk) and I’ll show it mostly in its full, hideous, glory:

{ lib, ... }:
let
  # Converts an attrset to the string representation of its paths
  #   {
  #     hello = { world = true; };
  #     goodbye = { moon = "bye"; }
  #   }
  #     Becomes
  #   {
  #     "hello.world" = true;
  #     "goodbye.moon" = "bye";
  #   }
  #     Works for arbitrarily nested attrsets
  attrsToStringPath =
    attrs:
    let
      _attrsToStringPath =
        _parent:
        let
          parent = if isNull _parent then "" else "${_parent}.";
        in
        attrs:
        lib.attrsets.foldlAttrs (
          acc: _name:
          let
            name = "${parent}${_name}";
          in
          value:
          acc // (if builtins.isAttrs value then _attrsToStringPath name value else { "${name}" = value; })
        ) { } attrs;
    in
    _attrsToStringPath null attrs;
in
{
  programs.librewolf = {
    enable = true;
    settings = attrsToStringPath { # <========= THIS LINE IS IMPORTANT, this is where the function is used!
      webgl.disabled = false;
      privacy.clearOnShutdown = {
        history = false;
        downloads = false;
        cookies = false;
      };
      network = {
        cookie.lifetimePolicy = 0;
        trr = {
          mode = 3;
          uri = "https://dns.mullvad.net/dns-query";
          default_provider_uri = "https://dns10.quad9.net/dns-query";
          strict_native_fallback = false;
          retry_on_recoverable_errors = true;
          disable-heuristics = true;
          allow-rfc1918 = true;
        };
      };
    };
  };
}

By abusing recursion we slowly accumulate and build our full string paths together for each value. Now I have some small piece of mind that I can go nuts with my attrset fun in my config.

Maybe someone out there will find this useful or look at this and go “they know that’s built into nixpkgs.lib right?” To that person I say this: I keep a cookie jar of lead chips around to snack on.

The Withering Fucking Pain of Nix and Git Submodules

Fri, 26 Jul 2024 00:00:00 GMT

Context

I have now spent an ungodly amount of time trying to get my Nix build for my blog to work with a Treesitter syntax parser. I am now declaring defeat. Fuck that.

The first bone to pick is that Nix’s Git submodule support is utterly, utterly, busted. Like, see here. That’s recent relative to this post, but these sorts of issues around Git submodules are a constant affair when it comes to Nix flakes. There’s one of two primary crapalicious ways of getting around this. There are more, but I’m choosing to stick to the two actually good solutions if you’re dead set on using Git submodules.

The first (and worse of the two) workaround

When you invoke nix build, invoke it with nix build '.?submodules=1' which should bring the git submodules along for the ride assuming they’re actually checked out.

This is awful. Actually, genuinely, terrible. Now any time I want to build the given project I have to remember to add in some extra arguments in my invocations to ensure dependencies are pulled into the build. Just what the fuck? Why is this considered an acceptable design? Half the reason behind me using Nix in the first place is so dependencies take care of themselves and reproducibility is easily achieved. By having to remember to pass in '.?submodules=1', I’m now worrying about dependencies that ideally should be handled in the code itself.

And guess what? Any downstream usage of the flake elsewhere will also now have to call all their commands with '.?submodules=1' as far as I can tell. Lovely. It pollutes out beyond just the single project, it screws up all of ‘em.

That ain’t gonna cut it, trash.

The second (and somewhat better, but still crappy) workaround

So another way of doing this is to define an additional input in my flake.nix like so:

{
  inputs = {
    src-with-submodules = {
      flake = false;
      url = "git+file:./.?submodules=1";
    }
  };

  outputs = { src-with-submodules }: {
    #... building derivations and whatever else
  };
}

And then in all my output derivations if they need the submodules instead of referencing say a local path like ./., reference in the src-with-submodules input instead. This still kinda sucks as now I have to keep that in mind when writing any Nix modules/code and it’s very much not typical to do that.

My issue

I used the second approach to pull in my Git submodules. Now to be clear, my Blog, at the time of writing, is made of twine, nearly snapping twigs, duct tape, and prayers. It ain’t well written at all and that includes the Nix flake. I admit that.

Still, even with that caveat, my cargo run invocation, even when fully offline after vendoring dependencies, works correctly. My precious syntax highlighting from Treesitter gets applied as expected. Under a nix build? Nope. Not in any way I’ve tried. If someone out there wants to hit me up and call me a dumbass, please feel free — but while you’re doing that shoot me a solution. I’ll do some downright strange things for a proper solution right now.

Maintaining the Status Quo

Are my failures probably related to incompetence on my part? Yup, prob-a-lee. Still, I’m tired. So very tired of fucking with it. It’s just syntax highlighting. I give up. Maybe I’ll come back to it another day and finally work out how to correctly do it. For now though, fuck it. Sublime syntax highlighting will have to do.

Avoiding Collisions in Hash Tables

Mon, 29 Apr 2024 00:00:00 GMT

A Quick Intro to Hash Tables and Collisions

A hash table maps a unique input to a value most typically stored in an array.

The goal of a hash table is to take advantage of a hash function that produces a position within an array where a given value might be stored for a unique key. This allows fast lookups into the array for the given key.

An example hash table of names to weight might be:

Key (name)	Value (weight)	Position in Array
Xavier	202	0
Sam	150	1
Preston	168	2

Where the values (with their keys) are stored in an array like so:

,
┌───────────────────────────────────────┐
│Xavier, 202 |  Sam, 150  | Preston, 168│
└───────────────────────────────────────┘
       0           1             2

When we do a lookup into that hash table for “Sam” the associated value would be 150 store at index 1 in our array in this case.

Here’s the issue, to store a value into the backing array we’re using a hash function which for performance reasons generally doesn’t output a unique position to store a value within the array. This means there’s a possibility for two different keys to map to the same place in memory, known as a collision.

We don’t want to overwrite existing data, and thus we have strategies for resolving collisions. In no particular order here’s the one’s we’ll go through:

Separate Chaining
Linear Probing (Open Addressing)
Coalesced Hashing
Double Hashing
Brent’s Method

Separate Chaining

Considered to be the simplest technique for resolving collisions, each slot in the array references a linked list (or other similar data structure) of records that collide on that slot. Each record stores its key and associated value.

To insert into a hash table using chaining, call a hash function to get an index in which to store a given value in an array. Instead of directly inserting the value at that index, instead create a link between that index and a new node for a linked list.

If we have multiple records under a single index in the array, each record points to the next one and we insert onto the last record in that position.

Linear Probing

Instead of storing colliding records under a separate data structure, open addressing instead stores directly on the array using a method known as probing.

When inserting a new record that collides with a preexisting record in the array, probing searches for the next available empty spot in the array and inserts the value there.

For example, given a slight modification of an earlier array like so:

, or (Null)
┌────────────────────────────────────────────────────────────────┐
│Xavier, 202 | (Null) | (Null) | Sam, 150 | (Null) | Preston, 168│
└────────────────────────────────────────────────────────────────┘
       0         1         2        3          4          5

Let’s say we want to insert a new person Jane with weight 120 and that our hash function informs us to insert at index 3. There’s an issue though, Sam already exists at index 3 and thus we cannot insert Jane into index 3. Thus we employ linear probing to find the next empty location in the array. The next empty location in this case is index 4, and thus we insert “Jane, 120” at position 4 in the array.

Our newly formed array will look something like this:

, or (Null)
┌───────────────────────────────────────────────────────────────────┐
│Xavier, 202 | (Null) | (Null) | Sam, 150 | Jane, 120 | Preston, 168│
└───────────────────────────────────────────────────────────────────┘
       0         1         2        3           4            5

Note that we are not just storing the associated value with a key into the array, we are storing both the key and value (the full record) in the array. This is important for searching for a given value. Since our hash function may not return the true position of a record, we have to check the keys against the search key until we find the correct record. The improvement over a simple linear search across the entire array though, is that our hash function informs us the index a given record must be at or be after.

Coalesced Hashing

Coalesced Hashing combines Open Addressing with Separate Chaining.

Where separate chaining will store colliding records in a separate data structure (linked lists), coalesced hashing will store the records directly in the array while linking the records together in something known as a collision chain.

When inserting using coalesced hashing and we have a collision occur, we check if the record stored at the index points to another record and so and so forth until we reach the end of the links. Once we reach the end we linearly probe for the nearest open slot in the array and insert the new record into that position and add a link from the last record in the chain pointing to the newly inserted record.

The reason one might want to use coalesced hashing over just a simple linear probe approach is that a linear probe could potentially go over unrelated or empty records until finding the record being searched. With a coalesced approach, one can instead follow the chain of links until either the record is found or the end of the chain is reached. This avoids going through unrelated records during a search.

Double Hashing

To handle collision resolution, double hashing does what’s on the tin. It employs two hash functions to determine the position in the array for a record mixed with probing.

During insertion, double hashing uses one hash value to index into an array and then repeatedly steps forward a set interval determined by a second hash function until an empty slot is found upon which its index is returned.

By using a second hash function data ends up more uniformly spread across the underlying array of the hash table and results in a more probabilistic approach to determining indices.

Brent’s Method

Brent’s method is a way of minimizing the average time of searching the underlying array of a hash table. Brent’s method defines the hash function as: hash(key) = key mod M and a incrementing function as i(key) = Quotient(Key / M) mod M where M refers to the size of the table.

If a collision occurs during insertion, Brent’s method will then do a check if we should move the colliding record to reduce the number of probes made during searches. If the number of probes required to retrieve a item is 3 more then a preexisting record will be shifted no more than 3 indices over to reduce the number of probes required during a search.

Nitpicking a Data Structures and Algorithms Midterm Question

Fri, 01 Mar 2024 00:00:00 GMT

Context

I’m currently enrolled at a university working towards a Computer Science degree. As part of my degree plan I have to take a Data Structures and Algorithms (DSA) course.

At the time of writing we had our midterm yesterday morning, and I missed marks on a question that I know for a fact had no correct answer. This post is an analysis as to why the question’s multiple choice answers were all incorrect.

I’m writing this knowing I’m being a bit of a pedant and knowing that it’s only a single question. But for whatever reason this keeps circling around in my mind throwing around all the chairs and spray painting gang markers on the walls. As such, I’m writing to get the poltergeist out.

The Question

Here’s roughly the question that was asked:

Given the following code:
#include <stdio.h>

int main(int argc, char *argv[]) {
    int arr[2][2] = {{1, 2}, {3, 4}};
    for (int i = 0; i < 2; i++) {
        for (int j = 0; j < 2; j--) {
            printf("arr[%d][%d] = %d, ", i, j, arr[i][j]);
        }
    }
}
What will the program do?

a.) There will be a warning
b.) Infinitely loop
c.) Output: arr[0][0] = 1, arr[0][1] = 2, arr[1][0] = 3, arr[1][1] = 4,
d.) Output: [0][-1] = 1, arr[0][-2] = 2, arr[1][-1] = 3, arr[1][-2] = 4,

I chose a, after realizing that’s the closest answer to being correct. Unfortunately this question is asking about the most horrid thing on Earth and a large reason languages like Rust exist — undefined behavior (UB)¹.

Answer a, of course, is not correct. The exam had answer b as correct, which is also wrong. In fact, none of the above answers are correct.

Why is the above code undefined behavior (UB)?

The code given to us has an out-of-bounds array access. Take a look at the for loops. The first for loop is all good, no problem there. The inner one, however, is our problem child.

int arr[2][2] = {{1, 2}, {3, 4}};
for (int i = 0; i < 2; i++) {
    for (int j = 0; j < 2; j--) { // This line is bad!
        // Eventually the sub-indexing of j will be out of bounds!
        printf("arr[%d][%d] = %d, ", i, j, arr[i][j]);
    }
}

The inner for loop decrements the variable j and will continue looping until j is not less than 2 (so loop until j is more than or equal to 2). This means the array will eventually be indexed by whatever integer is stored into j and since j is decremented from 0 it will be negative.

As a result the array will be sub-indexed by a negative number, which is outside the bounds of the array. This is our undefined behavior.

Now that we’ve identified the issue with this problem let’s get into why none of the provided answers are correct.

Why is answer `a` wrong?

To save you some scrolling, answer a was There will be a warning. In this course, questions have been asked with identical (or extremely close at least) answers like a which were interested in compiler behavior.

When I was marked wrong on this answer, I was actually quite surprised. Even more so when I recreated the identical code and attempted to compile it with gcc and got the following compilation error:

t.c: In function ‘main’:
t.c:7:13: error: iteration 1 invokes undefined behavior [-Werror=aggressive-loop-optimizations]
    7 |             printf("arr[%d][%d] = %d, ", i, j, arr[i][j]);
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
t.c:6:27: note: within this loop
    6 |         for (int j = 0; j < 2; j--) {
      |                         ~~^~~
cc1: all warnings being treated as errors

It seemed apparent to me that the question had no right answers. Imagine my surprise then when the professor compiled the same code in an online compiler and received neither an error or warning. That made me lose a bit of sanity and a few hours later I realized the culprit, I have a gat dang alias hiding out in my zsh config ruining property values:

alias gcc="gcc -Werror -Wall -Wpedantic -Warray-bounds -O3"

I bring this up because I spent perhaps a few too many hours trying to figure out what the heck was going on. A simple command -v gcc would’ve found it immediately, but I was too dumb to check that for quite some time 😞.

I also bring it up because my problem with a difference in compiler settings being a problem is a problem. By having potential answers be about compiler behavior the question immediately invokes a ton of implicit compiler configuration and flags that were (to my knowledge) never clearly stated in my course. The problem then is no longer about DSA — instead it’s about knowing the quirks of a compiler that may have differences on every single system and configurations out there.

Ok, but let’s just assume we’re rocking standard gcc with no flags, and no configuration elsewhere. Thus, there shouldn’t be any compiler warnings or errors. With that caveat in mind, answer a can’t be right.

Why are answers `c` & `d` wrong?

I’m lumping these two together because they’re wrong for similar reasons and I figure I should cover all of my bases.

Answers c & d were:

c.) Output: arr[0][0] = 1, arr[0][1] = 2, arr[1][0] = 3, arr[1][1] = 4,
d.) Output: [0][-1] = 1, arr[0][-2] = 2, arr[1][-1] = 3, arr[1][-2] = 4,

c is wrong because the inner loop is being continuously decremented. In theory the sub-indexing integer j should never get a positive value. Purely in theory.

d is wrong because by indexing negatively into the array, the output values are incredibly unlikely to match the actual values stored in the array; on top of that, indexing negatively into an array in C invokes UB, making this necessarily wrong anyhow.

Why is the supposedly correct answer, `b`, also wrong?

Answer b stated that “the program will infinitely loop”.

I’ll paste the code again here for your reference so you don’t have to scroll back up to see it.

#include <stdio.h>

int main(int argc, char *argv[]) {
    int arr[2][2] = {{1, 2}, {3, 4}};
    for (int i = 0; i < 2; i++) {
        for (int j = 0; j < 2; j--) {
            printf("arr[%d][%d] = %d, ", i, j, arr[i][j]);
        }
    }
}

On the surface level it certainly seems this code would infinitely loop. Our inner loop integer j is always being set more and more negatively and it appears it will never be more than two. After all, subtracting one from a negative will still leave you with a negative.

There are two reasons that come to my mind at a glance why the above code actually won’t run until the Sun swallows the Earth and time becomes meaningless.

Why `b` is Wrong, Part Uno

The first reason that code won’t run infinitely is the realities of undefined behavior.

On my laptop running Linux, kernel version 6.7.6 at the time of writing, the code above will have a segmentation fault within the first few seconds of running. The Linux kernel actually has quite a few built-in security features and it won’t let my code willy nilly access random addresses which sometimes happens with the negative indexing.

Outside of the kernel security mechanisms, since we’re in UB land who’s to say what will actually happen as its undefined. So at best saying it runs forever is a guess — not actually something that can be proven easily.

Why `b` is Wrong, Part Dos

The second reason is the much stronger of the two. Let’s, the for the sake of argument, say the above code never has a segmentation fault. It will always be able to decrement j and sub-index into the array.

Even with these concessions, the program will still not run infinitely.

To understand why, we have to dive into how integers work in C.

ISO/IEC 9899:2023 Purgatory (Oh the misery)

So let’s go see what the actual published standard has to say on what an integer is. If we go on over to open-std.org and go check out the latest published draft of “ISO/IEC 9899 - Revision of the C standard” we end up with this big ol’ document. Crack ‘er open and head down to page 41, section 6.2.6.2 (fun to pronounce as “sixty-two sixty-two”).

An integer, according to the standard, can be signed or unsigned. Since the integer we’re interested in, j, is a signed integer, let’s go through how a signed integer is standardized as in C.

For signed integer types, the bits of the object representation shall be divided into three groups: value bits, padding bits, and the sign bit. … the signed type uses the same number of N bits as values bits and sign bit. N - 1 are value bits and the remaining bit is the sign bit. … If the sign bit is zero, it shall not affect the resulting value. If the sign bit is one, it has value -(2^(n-1)).

Note one more thing, the standard does not define the actual size of an integer. That’s left to each implementation. Since we’re using the gcc compiler, let’s see what the gcc docs have to say on the size of a integer.

The integer data types range in size from at least 8 bits to at least 32 bits. The C99 standard extends this range to include integer sizes of at least 64 bits. You should use integer types for storing whole number values (and the char data type for storing characters). The sizes and ranges listed for these types are minimums; depending on your computer platform, these sizes and ranges may be larger.

Ah… that’s, uhhh, a pain — it’s gonna vary by architecture.

On my system, integers get 32 bits when using gcc and thus can represent 4,294,967,295 distinct numbers. Remember, the integer we’re working with is signed, thus half of those numbers will be negative. That means the largest and smallest numbers my integers can be are 2,147,483,647 and -2,147,483,648 respectively.

Hmmm… what happens when we exceed either of those numbers? Let’s find out.

#include <stdio.h>
#include <limits.h>
#include <stdlib.h>

int main()
{
    printf("Starting number is: %d\n", INT_MIN);
    printf("New number is: %d\n", INT_MIN - 1);
    return EXIT_SUCCESS;
}

INT_MIN in this case is the smallest integer representable.

gcc warns about an “integer overflow”… interesting.

❯ gcc program.c

t.c: In function ‘main’:
t.c:8:43: warning: integer overflow in expression of type ‘int’ results in ‘2147483647’ [-Woverflow]
    8 |     printf("New number is: %d\n", INT_MIN - 1);
      |                                           ^

Well let’s run it and see what the output is.

❯ ./a.out

Starting number is: -2147483648
New number is: 2147483647

Oh, huh. Neat.

So if we exceed the bounds the sign flips, literally an overflow or, in this case, an underflow.

Actually for real this time why `b` is wrong

With the overflow and underflow behavior now covered, let’s take a look back at the code given to me in class:

#include <stdio.h>

int main(int argc, char *argv[]) {
    int arr[2][2] = {{1, 2}, {3, 4}};
    for (int i = 0; i < 2; i++) {
        for (int j = 0; j < 2; j--) {
            printf("arr[%d][%d] = %d, ", i, j, arr[i][j]);
        }
    }
}

Now with everything we’ve covered let’s go point by point:

So long as j is less than 2 this will continue to run.
Integer j will be decremented for every execution of the inner loop.
Integer types have a maximum capacity and if we exceed that capacity the sign changes.
Eventually, j will reach the smallest value an integer can represent and on the next loop it will underflow and become a massively positive number.
Integer j will then eventually be more than 2 in value and thus the condition j < 2 will be invalidated and the loop will end.

To prove the code won’t infinitely loop, let’s actually run the code with a few small modifications to speed things up:

#include <limits.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    int arr[2][2] = {{1, 2}, {3, 4}};
    // Move j's initializer up here so we can access it after the loop.
    // Also, set j to `INT_MIN` so our loop doesn't have to decrement
    // 2147483648 times before the theorized underflow occurs which
    // just cuts down on how long it takes to underflow.
    int j = INT_MIN;
    for (int i = 0; i < 2; i++) {
        for (; j < 2; j--) {
            // Comment out the arr access to avoid seg faults
            // printf("arr[%d][%d] = %d, ", i, j, arr[i][j]);
            printf("J -> %d\n", j);
        }
    }
    // If the loop isn't infinite, we should see this message
    printf("THE LOOP WAS NOT INFINITE, J VALUE: %d\n", j);
}

Running the above outputs:

❯ gcc prog.c && ./a.out

J -> -2147483648
THE LOOP WAS NOT INFINITE, J VALUE: 2147483647

Y u p. That’s underflow for sure.

We can clearly see where j was the smallest possible integer and then wrapped around to the largest possible integer and in doing so caused the inner loop to end.

Thus, answer b is also wrong.

A quick summary

As a result of all the above points, we can readily see that the potential answers we could pick from are all wrong.

a.) There will be a warning
b.) Infinitely loop <- An integer underflow will eventually occur causing the looping to end
c.) Output: arr[0][0] = 1, arr[0][1] = 2, arr[1][0] = 3, arr[1][1] = 4,
d.) Output: [0][-1] = 1, arr[0][-2] = 2, arr[1][-1] = 3, arr[1][-2] = 4,

A quick summary of why each is wrong:

a.) We won’t get a warning unless we specify compiler flags/config for gcc.
b.) An integer underflow will occur causing the looping to end and thus the program has a finite execution period, and also the program will almost certainly have a segmentation fault causing the looping to end.
c.) j will never sub-index into 1 or 2.
d.) Indexing j negatively invokes undefined behavior, so we can’t know the output. As well as the fact that the pattern for accessing the first index is wrong anyhow.

Is My Pedantry (🤓) Satisfied Now?

Somewhat, to be wholly honest I am still annoyed I got marked off on that question. At least now I can point somewhere and say “See, look! I was right!” and then stick my fingers in my ears and cry in the corner.

I actually skipped how integers really work under the hood in terms of their binary representation. If you’re interested, here’s the binary of the earlier underflow code example:

Starting number is: -2147483648 | Binary: 10000000000000000000000000000000
      New number is: 2147483647 | Binary: 01111111111111111111111111111111

That opens up a whole ‘nother can of worms of two’s complement and actually how the bits are stored and manipulated. A deeper look at why the underflow behavior occurs through how integers are added at the binary level can also be used to disprove answer b. I cut a full break down of this out, because I realized it was somewhat irrelevant. The quick integer whole number example makes the same point without nearly as much technicality.

The main takeaway: anytime you end up in undefined behavior land with C things get wacky real quick and all numeric types are actually black magic pretending to be approachable.

If you don’t know what undefined behavior is, Wikipedia has an article on it here. ↩

Setting up NixOS on my Home Server

Sun, 29 Oct 2023 00:00:00 GMT

What does this article cover?

Why I chose NixOS
Installing NixOS from the minimal installer.
Using Flakes to configure the system
An Erase Your Darling’s Setup on tmpfs which wipes the system on every reboot. This seems crazy, but with NixOS this is incredibly powerful.
Self hosted Gitlab with a Gitlab Runner
Secrets management with Agenix

This is not a general NixOS tutorial, I’m assuming you have some level of familiarity with what a Nix Flake is and what NixOS is. I strongly recommend against using this article as a how to for NixOS. This is not that. This is more my musings and interesting tidbits I came upon whilst messing about with NixOS and a small bit of a guide to those who may need Gitlab on NixOS.

Why I Chose NixOS

If you don’t care, you can skip to the next section here. I do a minuscule amount of ranting as to my reasoning, as you can probably guess.

My background in managing systems is via Ansible and that’s how I’ll be approaching the why.

One setup that you can do with NixOS is an “Erase Your Darling’s” configuration. Using BTRFS, ZFS, or another file system with snapshot support, one can rollback the system on every reboot to the last snapshot or mount the system on tmpfs and wipe it on reboot. By doing this we can have NixOS declaratively set the system state such that anything outside of the config that hasn’t been explicitly set to persist on reboot gets wiped. This ensures the NixOS configuration is the source of truth.

A criticism I have of Ansible is that it actively requires users to practice good check in control with it. Meaning they have to remember or create a procedure to ensure all configuration ends up in Ansible. Not too hard if it’s a one person show, but becomes increasingly bothersome as more and more individuals need to contribute. In effect, this causes little “hacks” accumulate on the system outside of version control by accident. Procedures to check-in everything into VCS only solves this so much. My preference is to ensure the systems force us to take the correct action by default instead of those actions being opt-in.

Even in the scenario in which you are perfect about getting everything into VCS, Ansible still falls short. Ansible doesn’t necessarily define what is on a system and how it’s configured, it defines how we’d like a system to be configured via a declarative set of (ideally) idempotent steps. These steps may fail, and, furthermore, there is nothing at the system level enforcing the entering of these steps into Ansible. NixOS enforces this completely.

This all comes from a, perhaps extreme, ideology of mine that pretty much everyone will take the path of least resistance when working on something for a longer period of time. They may commit to “Yeah, of course keep everything in VCS” and some may be able to keep to that — most won’t. Understanding this then, we must enforce below the human level the desired outcomes of systems engineering we want. We want everything checked into VCS? Great, ensure everything not checked in is wiped, all that work lost. Everyone will become comfy real fast with checking changes in under such a system.

Another hill I am 99% willing to die on is ensuring that a server, or for that matter, any system should be able to reproduced with minimal or no human interaction. My server went down in Chicago? No problem, I have that configuration in VCS, I’ll just put it on one in New York.

I believe these two elements lead to high velocity and more resilient infrastructure over time and are worth the upfront investment to achieve as the dividends are huge.

Ideology and other ranting done. Let’s get into it and thanks for reading that lore dump if you actually did 😉.

Installing NixOS

A quick preface as to how I chose to setup my system. I’m rolling with an “Erase Your Darling’s Setup on tmpfs” such that my system is wiped on every boot with the goal of keeping the system consistent with what I define in my Nix configs.

I wrote a install script to set all of this up for me, using BTRFS mostly for reasons of compression more than snapshots. I’ll have to reconsider BTRFS in the future. For now though, I’m using it.

As part of this install I do one thing that I’ve noticed most NixOS configurations are not doing, that being taking full advantage of setting disk labels. Many configurations I’ve referenced run nixos-generate-config and call it a day after checking that into version control. That works fine, but I far prefer being able to define my filesystem.nix ahead of time. As such, disk labels.

For instance, Luna, my home server, has its file system set ahead of time like so:

{ config, lib, pkgs, modulesPath, ... }:
{
  fileSystems = {
    "/" = {
      device = "none";
      fsType = "tmpfs";
      options = [ "defaults" "noatime" "mode=755" ];
    };

    "/boot" = {
      device = "/dev/disk/by-label/NixOS-Boot";
      fsType = "vfat";
      options = [ "defaults" "noatime" ];
      depends = [ "/" ];
    };

    "/nix" = {
      device = "/dev/disk/by-label/NixOS-Primary";
      fsType = "btrfs";
      options = [ "subvol=@nix" "compress=zstd" "noatime" ];
    };
  };

  zramSwap.enable = true;
}

Those disk labels, NixOS-Boot and NixOS-Primary, are set by my install script. Since I use the same script to install NixOS everywhere I know ahead of time what labels to target. Magic.

Then I install NixOS after having defined a configuration with my install script.

So a typical setup would look something like this:

Define the system ahead of time in my hosts/ directory
Install NixOS onto a flash drive (or into Ventory)
Start up the system using that minimal boot drive
Git clone my NixOS configuration, git clone https://gitlab.orion-technologies.io/philler/nixos.git && cd nixos
Identify the disk I’m going to install to via lsblk
Run my install script: bash install.bash -d <DISK_HERE> -H <NEW-HOST> (optionally passing -e to enable encryption)
Wait for it to be done. Potentially cry when something goes wrong with the install.
Reboot

So where does Erase Your Darling’s come in?

Right. So, the install script does a fair bit of heavy lifting, you may have noticed up in the filesystem.nix I pasted up there that I had the root path, /, as an fsType of tmpfs. This is my erasure method. I then use Impermanence to declare which directories shouldn’t be nuked on reboot. Those get saved into /nix/persist/ and then symlinked back out on every reboot. I declare my default Impermanence setup like so:

environment.persistence = {
  "/nix/persist" = {
    hideMounts = true;
    directories = [
      "/var/lib"
      "/var/log"
      "/etc/nixos"
      "/opt"
      "/persist"
    ];
    files = [
      "/etc/machine-id"
      "/etc/nix/id_rsa"
    ];
  };
};

So now, with that defined in my flakes, I get all of those defined directories and files persisted on reboot by default. For each host I can then further define more directories to persist in the host-specific config.

Not too much to it, but quite powerful.

Gitlab, Dollar Tree Github

I’m sure this heading won’t be incendiary or anything.

At the time of writing, I self-host Gitlab to store all of my code and run CI/CD operations. Originally I handled this all in Ansible and a bit of spit shine on Debian.

Now NixOS is really cool, it provides a built-in Gitlab service! Which I don’t use… yeah. The Gitlab service NixOS provides is a touch out of date and because Gitlab is the worst platform on the face of the planet of Earth when it comes to administration you can’t restore a backup on even slightly different versions of Gitlab. So the built-in module is right out 😦.

I’m not screwed, there’s another solution. Gitlab provides a docker image. That’s what I ultimately went with.

I have a docker directory nestled within Luna’s config with a default.nix file that looks like so:

{ pkgs, ... }:
{
  environment.systemPackages =  with pkgs; [
    docker_24
    docker-compose
  ];

  virtualisation = {
    oci-containers.backend = "docker";
    containers.enable = true;
    docker = {
      enable = true;
      autoPrune.enable = true;
      package = pkgs.docker_24;
    };
  };
  imports = [
    ./gitlab.nix
  ];
}

I don’t really need docker-compose, but I’m saving myself trouble ahead of time and just shoving it on the system. Increased attack surface? Yeah, a bit. Actually a likely issue? Nope, and it’s damn useful.

The gitlab.nix file its importing looks like:

{ lib, config, specialArgs, ... }:
let
  gitlab_home = "/opt/gitlab";
  hostname = "gitlab.orion-technologies.io";
in
{
  virtualisation.oci-containers.containers.gitlab = {
    image = "gitlab/gitlab-ee:latest";
    autoStart = true;
    ports = [
      "127.0.0.1:8080:80"
      "2222:22"
    ];
    volumes = [
      "${gitlab_home}/config:/etc/gitlab"
      "${gitlab_home}/logs:/var/log/gitlab"
      "${gitlab_home}/data:/var/opt/gitlab"
    ];
    extraOptions = [
      "--shm-size=256m"
      "--hostname=${hostname}"
    ];
  };

  networking.firewall.allowedTCPPorts = [
    2222
  ];

  age.secrets.gitlab-runner-reg-config.file = specialArgs.secrets + "/gitlab-runner-reg-config.age";
  services.gitlab-runner = {
    enable = true;
    services = {
      default = with lib; {
        registrationConfigFile = config.age.secrets.gitlab-runner-reg-config.path;
        dockerImage = "alpine";
        tagList = [
          "alpine"
          "default"
        ];
      };
    };
  };

  services.nginx.virtualHosts."${hostname}" = {
    locations."/".proxyPass = "http://127.0.0.1:8080";
    forceSSL = true;
    enableACME = true;
  };
}

I’ll break it down real quick for those poor basta— folks who end up deciding to put Gitlab on NixOS and have the same issues as me.

Part uno, the actual Gitlab instance on Docker.

virtualisation.oci-containers.containers.gitlab = {
  image = "gitlab/gitlab-ee:latest";
  autoStart = true;
  ports = [
    "127.0.0.1:8080:80"
    "2222:22"
  ];
  volumes = [
    "${gitlab_home}/config:/etc/gitlab"
    "${gitlab_home}/logs:/var/log/gitlab"
    "${gitlab_home}/data:/var/opt/gitlab"
  ];
  extraOptions = [
    "--shm-size=256m"
    "--hostname=${hostname}"
  ];
};

This is almost verbatim copied over from Gitlab’s docs, the only thing of note here is the volume mount path. That variable gitlab_home was defined earlier as /opt. I persist that directory between reboots, so we’re good to store stuff there.

Make sure you’ve allowed port 2222 on the firewall:

networking.firewall.allowedTCPPorts = [ 2222 ];

If you forget to do so, you won’t be able to use SSH keys on some git operations.

Part dos, the reverse proxy.

I like SSL, I hope you do too. Instead of allowing Gitlab to manage its own proxy, I prefer having a single proxy on the host that handles all of that for the various containers that may be running.

I have a primary nginx.nix service file that contains the following:

{ config, ... }:
{
  services.nginx = {
    enable = true;
    recommendedProxySettings = true;
    recommendedOptimisation = true;
    recommendedGzipSettings = true;
    recommendedTlsSettings = true;
  };
  security.acme = {
    acceptTerms = true;
    defaults.email = "price@orion-technologies.io";
  };
}

A bit of basic setup for all the services and ensuring ACME (by the way the ACME RFC is actually insanely well written, give it a read if you have time to kill) plays nice.

And then all I have to define is the specifics for a given virtual host.

services.nginx.virtualHosts."${hostname}" = {
  locations."/".proxyPass = "http://127.0.0.1:8080";
  forceSSL = true;
  enableACME = true;
};

In the config above the hostname variable is my Gitlab A record: gitlab.orion-technologies.io.

That’s the nix side of things, we have a bit more to do on the Gitlab side of things. Which I am sad to say took me more than just a few minutes to figure out because I didn’t reference the docs, RTFM. Gitlab-wise we need to set the following in our gitlab.rb:

external_url 'https://gitlab.orion-technologies.io'
nginx['listen_port'] = 80
nginx['listen_https'] = false
gitlab_rails['gitlab_shell_ssh_port'] = 2222

Of course replace the external URL with yours. We disable HTTPS in the Nginx config because we are managing that on the host with our own reverse proxy. Bind it on to port 80 which we pass out to the host accessible at 8080 to the docker container. The one setting that wasted the most time of mine was gitlab_shell_ssh_port by a mile. Why did it not work? Maybe I had the port closed on the network firewall, the world may never know. Real quick though, make sure you set gitlab_shell_ssh_port to the external port that you’re opening on the host, not the internal.

Part tres, the Gitlab runner

services.gitlab-runner = {
  enable = true;
  services = {
    default = with lib; {
      registrationConfigFile = toString ./path/to/runner/config;
      dockerImage = "alpine";
      tagList = [
        "alpine"
        "default"
      ];
    };
  };
};

You may have noticed I slightly changed the assignment for the registrationConfigFile. We’ll talk about secrets management in a sec. For the general use, if you don’t want to handle secrets in your NixOS configuration, just set that path to something like /opt/gitlab/gitlab-runner-config and store the config there. You’ll need to get a runner token from the Admin area which is located in the middle of nowhere. If you forgot how to get to the stuck in Utah Admin panel button I’ve gotchu. Go to your Gitlab instance’ home page, on the top left there is “Search or go to…”, click that and then click “Admin Area” in the modal that pops up. They couldn’t have hidden that better if they tried, I swear.

From here head to “CI/CD” > “Runners” and in the top right you should see a nice shiny “New Instance Runner” button: Now fill those details out for your runner and hit “Create Runner” at the bottom. Do not close the page, we’re going to need the url and token from it.

Now remember the registrationConfigFile option mentioned earlier? This is when that becomes relevant. Create a file at the registrationConfigFile path on the NixOS system and place the following into it:

CI_SERVER_URL=
REGISTRATION_TOKEN=

So for me it would look something like:

CI_SERVER_URL=https://gitlab.orion-technologies.io
REGISTRATION_TOKEN=wdjD-dwa23AsdahSAli-ALWO

At this point when we run nixos-rebuild switch everything should be fine and dandy.

This is really cool, but if we want to go a step further we can actually manage this file along with our NixOS configuration by encrypting it. If that seems a bit too iffy feel free to stop now. If you think that sounds neat, managing secrets in NixOS, then onwards into the light!

Agenix, where’s my damn Yubikey plugin support?!

There’s many ways to manage secrets in Nix the main ones are agenix and sops-nix. I ultimately settled on Agenix over sops-nix because Yubikey is close to native support with Agenix and I have a Yubikey. The header of this section isn’t that serious, and perhaps I’ll get annoyed enough to shepherd something to the finish line.

Age is self described as “…a simple, modern and secure file encryption tool, format…”. I treat it basically as PGP if it was sane.

If you have a Yubikey I recommend storing a master encryption key using age-plugin-yubikey which massively improves storing age keys in the PIV store

The way I use this is shove a single master key in my Yubikey, then generate an age key for each host. So Luna, for instance, has its own age key stored in an encrypted external drive. When I go to install Luna I decrypt the key and copy it over to the server to a path where Nix will look for it when I do a rebuild. I keep a master key on my Yubikey so in the scenario in which my external drive explodes into a million shards I still have a way of accessing the encrypted information and vice versa. The yubikey dies? I look at the external drive. Is it perfectly redundant? No. But for my needs it’s good enough. You can take this as far as you’d like.

First things first to actually using this then. Get agenix installed and ready to roll and I recommend also installing age or rage the encryption tool agenix wraps.

Agenix prefers you to use SSH keys, I’m not doing that. I’m purely using age keys with SSH left untouched. Go ahead and create your age key for your host age-keygen -o host.key and make sure you save it somewhere secure.

In the nix flake, at the top level, create a secrets directory and in it create a secrets.nix file.

The secrets.nix file is used purely by agenix to encrypt and modify secrets. Within it we’ll define the recipients, the public keys that can encrypt the secrets where the age private key we generated can decrypt them. Get the public key from your generated age key and let’s modify the secrets.nix file.

Here is my secrets.nix file:

let
  keys = rec {
    master = "age1yubikey1qfnj0k4mkzrn8ef5llwh2sv6hd7ckr0qml3n9hzdpz9c59ypvryhyst87k0";
    orion-tech = {
      luna = [
        "age1jgwqs04tphuuklx4g3gjdg42mchagn2gu7sftknerh8y8l9n7v7s27wqgu"
        master
      ];
    };
  };
in

{
  "gitlab-runner-reg-config.age".publicKeys = keys.orion-tech.luna;
}

Take note that I have a master key and then a machine specific key for Luna. In the body we define the file that will be encrypted with the given public keys, in my config that would be gitlab-runner-reg.config.age which allows either the master key to decrypt it or Luna’s key to decrypt it.

Let’s go ahead and create the Gitlab runner config:

agenix -e gitlab-runner-reg-config.age -i /path/to/your/age/key

This will open the editor defined in your $EDITOR with the decrypted contents of the gitlab-runner-reg-config.age file.

Let’s go ahead and fill that out and save it in this format from earlier:

CI_SERVER_URL=
REGISTRATION_TOKEN=

Great! We have our secret. How do we access it in our flake?

Earlier we had a services.gitlab-runner which had a registrationConfigFile setting. Instead of using toString <PATH-HERE> we’ll now use our secret like so:

age.secrets.gitlab-runner-reg-config.file =  ../secrets/gitlab-runner-reg-config.age;
services.gitlab-runner = {
  enable = true;
  services = {
    default = with lib; {
      registrationConfigFile = config.age.secrets.gitlab-runner-reg-config.path;
      dockerImage = "alpine";
      tagList = [
        "alpine"
        "default"
      ];
    };
  };
};

Replace ../secrets/gitlab-runner-reg-config.age with the path to your secret.

You’re done, that’s it. Pretty neat, huh?

If you don’t like storing your secrets in a public repository you can of course add the secrets directory in as a git submodule from a private repository if you so desire. You can read up on git submodules if that caught your interest here.

The End?

The end indeed. You should have Gitlab with a runner rocking now on your server and, if you followed the secrets section, secrets managed in your config.

If you want to see my config in all of its hideousness, you can find it here;

If you found anything wrong or egregious feel free to file an issue on the Github Mirror.