A thing I’ve been missing is the ability to easily share files with insecure devices. I’ve tried a bunch of solutions over the years, including using Syncthing and Nextcloud, but they were neither nice to use nor run. Recently, a friend found a much better trade-off in terms of setup complexity and ease-of-use. So, let’s build such a file-sharing service with rclone, Nginx, and Kubernetes.
The full code for this post is available here.
Imagine that we’re travelling and we need to print a document from email. The printshop’s computer has Internet access, but we really don’t want to login to our email from there. Instead, we want to download the document to our phones, put it somewhere, then download it onto the public computer.
What we need is some sort of private server to which we can upload files very easily from our phones, tablets, or laptops. We also need this to work over restricted Internet connections in case we have to use hotel WiFi. In other words, we have to use HTTP.
Natively, HTTP doesn’t support file upload and editing—we need something more than a plain webserver to implement the file handling logic. We don’t want this server to be too complicated to setup and run—something like Nextcloud is very featureful, but far too heavy. Luckily, there’s an extension to plain HTTP called WebDAV that does what we want. Unluckily, there isn’t great support for it on the server side. For example, the Nginx DAV module works at a superficial level, but behaves erratically if you pay close attention.
The server I’ve found to work best is rclone. We basically
run rclone serve webdav
on a directory and it just works. The
problem is that it doesn’t have support for multiple endpoints with
different permissions, so we still need Nginx for this.
Now that we know roughly what we want, let’s formalize it.
Practically, we have a /data
directory on the server that we want to
provide authenticated read/write access to over WebDAV. We also need
to provide unauthenticated public access to /data/pub
. We’ll run
rclone
for the private files and serve only on localhost. We’ll
then make Nginx our entry point, have it serve the public files
itself, and have it authenticate and proxy access to the rclone
server.
We’ll put both rclone
and Nginx in a single container and run this
as a Kubernetes pod. The /data
volume will be a
Persistent Volume not tied to any particular machine. This will
let our pod migrate to other nodes during maintenance and minimize
service interruption. We’ll also let Kubernetes terminate HTTPS
connections since it’s good at that and saves us from having to bundle
certificate management in our container.
Let’s build this inside-out, starting with the container.
We’re going to use Nix to build the container since it keeps all of our configuration in one place and ensures repeatable builds.
The full flake for this is available here. The core of it is
the container
package. It just creates a few directories that Nginx
needs at runtime, includes the fakeNss
package so that Nginx doesn’t
complain about missing users, and then calls the start script. Nix
sees our dependencies, so we don’t need to include nginx
or rclone
or anything else manually.
packages.container = pkgs.dockerTools.buildLayeredImage {
name = "rclone-webdav";
tag = "flake";
created = "now";
contents = [
pkgs.fakeNss
];
extraCommands = ''
# Nginx needs these dirs
mkdir -p srv/client-temp
mkdir -p var/log/nginx/
mkdir -p tmp/nginx_client_body
'';
config = {
ExposedPorts = { "80/tcp" = { }; };
Entrypoint = [ "${startScript}/bin/startScript" ];
Cmd = [ ];
};
};
flake.nix
The start script starts rclone
and nginx
as two Bash jobs, then
waits for rclone
to terminate. We could’ve done something more
complicated with a dedicated “pid 0” process,
but we wouldn’t gain much and it would complicate our container.
startScript = pkgs.writeShellScriptBin "startScript" ''
set -m
${pkgs.rclone}/bin/rclone serve webdav --log-level INFO --baseurl priv --addr 127.0.0.1:8081 /data/ &
${pkgs.nginx}/bin/nginx -c ${nginxConfig} &
fg %1
'';
flake.nix
Next, we need the config file for Nginx. We set up the /pub
endpoint to serve files from /data/pub
. We set up the /priv
endpoint as a proxy to the rclone
process after authenticating. A
slightly weird thing about this setup is that we access the
/data/pub/file1
through the /pub/file1
URL, but we write to it
through the /priv/pub/file1
URL.
A very important setting is client_max_body_size 200M
. Without it,
we wouldn’t be able to upload files larger than 1 MB. There is a
corresponding necessary setting in the Ingress config later.
nginxConfig = pkgs.writeText "nginx.conf" ''
daemon off;
user nobody nobody;
error_log /dev/stdout info;
pid /dev/null;
events {}
http {
server {
listen 80;
client_max_body_size 200M;
location /pub {
root /data;
autoindex on;
}
location /priv {
proxy_pass "http://127.0.0.1:8081";
auth_basic "Files";
auth_basic_user_file ${authConfig};
}
location = / {
return 301 /pub/;
}
location / {
deny all;
}
}
}
'';
flake.nix
Finally, we need the authentication config for Nginx. This is where
things get tricky. We need to include the username and password for
/priv/
access in the container somewhere, but we don’t want them to
appear in the repo—that’d just be bad security practice (and I
certainly can’t include my password in the repo because I’m making it
public). The standard solution is to pull these in as environment
variables from the CI or build host, but we can’t do that here because
Nix flakes are hermetic. That is, they cannot use anything at all
from outside the repo. So, we cheat. We commit a users.env
file
with dummy environment variables to the repo and we overwrite it in
the CI with the right values.
authConfig = pkgs.runCommand "auth.htpasswd" { } ''
source ${./users.env}
${pkgs.apacheHttpd}/bin/htpasswd -nb "$WEB_USER" "$WEB_PASSWORD" > $out
'';
flake.nix
The .gitlab-ci.yml
is essentially unchanged from my previous
post. The only new addition is the echo "$PROD_CREDENTIALS" > users.env
line which dumps the username and password from a
Gitlab variable into the users.env
file for the
build to use.
build-container:
image: "nixos/nix:2.12.0"
stage: build
needs: []
only:
- main
- tags
variables:
CACHIX_CACHE_NAME: scvalex-rclone-webdav
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
before_script:
- nix-env --install --attr nixpkgs.cachix
- nix-env --install --attr nixpkgs.skopeo
- cachix use "$CACHIX_CACHE_NAME"
script:
- mkdir -p "$HOME/.config/nix"
- echo 'experimental-features = nix-command flakes' > "$HOME/.config/nix/nix.conf"
- mkdir -p "/etc/containers/"
- echo '{"default":[{"type":"insecureAcceptAnything"}]}' > /etc/containers/policy.json
# 👇 👇 👇 NEW LINE 👇 👇 👇
- echo "$PROD_CREDENTIALS" > users.env
- cachix watch-exec "$CACHIX_CACHE_NAME" nix build .#container
- skopeo login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
- ls -lh ./result
- 'skopeo inspect docker-archive://$(readlink -f ./result)'
- 'skopeo copy docker-archive://$(readlink -f ./result) docker://$IMAGE_TAG'
With all this in place, we can build the container locally with nix build .#container
and have Gitlab build it for us in CI. See the
Justfile for other useful local recipies.
Now, let’s operationalize our container and run it in Kubernetes.
The whole config is available here. It uses kubectl
’s
built-in Kustomize to force everything into a dedicated
namespace.
The setup is fairly simple: the container is run in a single pod by a StatefulSet, it is fronted by a Service, and it is backed by a Longhorn PersistentVolume. It’s important to use a StatefulSet instead of a Deployment here to make sure the volume is properly detached before the pod is restarted or replaced.
apiVersion: v1
kind: Service
metadata:
name: rclone-webdav
spec:
selector:
app: rclone-webdav
type: NodePort
ports:
- port: 80
protocol: TCP
name: http
targetPort: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rclone-webdav
labels:
app: rclone-webdav
spec:
replicas: 1
serviceName: rclone-webdav
selector:
matchLabels:
app: rclone-webdav
template:
metadata:
labels:
app: rclone-webdav
spec:
containers:
- name: rclone-webdav
image: registry.gitlab.com/abstract-binary/rclone-webdav-container:main
imagePullPolicy: Always
ports:
- containerPort: 80
name: http
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
spec:
storageClassName: longhorn
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 2G
Finally, we expose our service to the outside world with an
Ingress. In my setup, I use
ingress-nginx
as the actual ingress implementation
and cert-manager
to manage TLS certificates.
Importantly, we have to configure the maximum upload size again with
the nginx.ingress.kubernetes.io/proxy-body-size: 200m
annotation.
Without this, file uploads would be limited to 1 MB.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rclone-webdav
namespace: rclone-webdav
annotations:
cert-manager.io/cluster-issuer: 'letsencrypt'
nginx.ingress.kubernetes.io/proxy-body-size: 200m
spec:
ingressClassName: nginx
tls:
- hosts:
- files.abstractbinary.org
secretName: files-certs
rules:
- host: files.abstractbinary.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: rclone-webdav
port:
number: 80
…And that’s all there is to it. Like with most container and Kubernetes things, there’s seemingly a lot of code, but most of it is boilerplate or just common patterns.
]]>axum
webserver setup.
]]>
I revamped this blog’s RSS feed. The downside is that
RSS readers will probably show the last few posts again. Sorry about
that. On the upside, the new feed contains full post contents and is
standards compliant. This post lists the changes I made, mostly to
the axum
webserver setup.
I normally write in a “let’s build something together” voice, but this post is more of a “let’s all laugh at Alex for failing to set the Content-Type” kind of situation, so I’ll use the first person singular.
For the most part, I followed the recommendations from Kevin Cox’s excellent “RSS Feed Best Practises”.
Content-Type
headerLet’s start small: what was content type of the atom feed?
$ curl -v https://scvalex.net/atom.xml > /dev/null 2>&1 | grep -i content-type
< content-type: text/xml
That’s not good. It’s supposed to be application/atom+xml
(or application/rss+xml
before the switch to Atom). The value was being inferred by my
server’s ServeDir
middleware from the xml
extension of the file. Since there was no way to configure
ServeDir
, I wrote a new axum handler
just for
/atom.xml
route:
...
let app = Router::new()
.route("/atom.xml", get(atom_handler))
...
async fn atom_handler(
State(state): State<SharedState>,
) -> Result<Response<BoxBody>, (StatusCode, String)> {
use http::header::CONTENT_TYPE;
get_static_file(state.dist_dir.join("atom.xml"))
.await
.map(|mut res| {
res.headers_mut().insert(
CONTENT_TYPE,
HeaderValue::from_static("application/atom+xml"),
);
res
})
}
async fn get_static_file<P: AsRef<std::path::Path>>(
file: P,
) -> Result<Response<BoxBody>, (StatusCode, String)> {
use axum::body::boxed;
use tower::util::ServiceExt;
ServeFile::new(file.as_ref())
.oneshot(Request::new(()))
.await
.map(|res| res.map(boxed))
.map_err(|err| internal_server_error(err, "static file not found"))
}
This is annoyingly verbose and it might have been easier read the file
directly instead of offloading to ServeFile
, but
I didn’t know if the latter did anything special and I had bigger
issues to tackle.
Let’s see that it worked:
$ curl -v http://localhost:3000/atom.xml > /dev/null 2>&1 | grep -i content-type:
< content-type: application/atom+xml
Good. The even better news is that the code only gets less verbose from here.
To quote MDN:
Cross-Origin Resource Sharing (CORS) is an HTTP-header based mechanism that allows a server to indicate any origins (domain, scheme, or port) other than its own from which a browser should permit loading resources.
I think CORS is essentially a security scheme to prevent websites from making requests from Javascript to other websites using users’ credentials. For instance, when you’re visiting example.com, it would be bad if the site could get your browser to run Javascript that makes requests to the AWS API with your own logged-in cookie. This is prevented by the AWS API serving its pages without the CORS header authorizing example.com to make requests against it, so users’ browsers will throw an exception if this is attempted. Honestly, it seems like it would have been better to not normalize allowing random sites to run random code on users’ machines, but I guess that ship has sailed.
The CORS policy defaults to “not allowed”, so if no Access-Control-*
headers are sent, XMLHttpRequests and fetch()
calls will fail. What headers did my site deliver for the
atom feed?
$ curl -v https://scvalex.net/atom.xml > /dev/null 2>&1 | grep -i 'access-control'
$
It delivered no headers. This meant that any browser-based RSS reader
was going to fail to fetch the feed. I fixed this by annotating the
/atom.xml
route with the Cors
middleware:
let app = Router::new()
.route(
"/atom.xml",
get(atom_handler).layer({
use tower_http::cors::{Any, CorsLayer};
CorsLayer::new()
.allow_methods([http::Method::GET])
.allow_origin(Any)
}),
)
And now the headers are present:
$ curl -v http://localhost:3000/atom.xml > /dev/null 2>&1 | grep -i 'access-control'
< access-control-allow-origin: *
< vary: access-control-request-method
< vary: access-control-request-headers
The choice to switch from RSS2 to Atom was mostly an aesthetic one on my part. The RSS spec has always felt a bit off to me. To give one example, dates in RSS must be formatted like “Sat, 07 Sep 2002 00:00:01 GMT” which just seems weird for a machine-readable file. For comparison, dates in Atom are formatted like “2022-12-18T00:00:00Z” which is exactly what I would expect.
More importantly, RSS has only one element for post contents, whereas Atom has separate “summary” and “content” elements. This came in handy since I was planning to include full post contents in the feed.
Other than that, the two formats are the same for the purposes of my blog and the conversion was little more than renaming some elements.
Before now, I’d include only the first paragraph of every post in the RSS feed. Conceptually, it had the same information as the All Posts page.
My original reasoning from ten years ago was that I wanted people to click through to the website so that I could see them in analytics—I valued seeing a number go up more than people actually reading my posts. These days, I’d much rather engage with readers regardless of how they see my content. Also, I suspect a fair number of them wouldn’t show up in analytics anyway because they block third party Javascript—I certainly do.
The more recent reason for not including the full post text in the RSS feed was a limitation on the part of my site generator. This is a Rust program which reads a bunch of input markdown files and Liquid templates, combines them somehow, and generates a directory of static files. The generator can run both in batch mode where it builds everything from scratch, but also in a polling mode where it watches input files for updates and only builds what has changed.
The generator has to handle three kinds of input text files:
Liquid templates that are loaded into
liquid-rust
. Some of these are small “macros” of
HTML to be included in pages, and some are templates of full HTML
meant to wrap content. For example, I have an img_float
macro to
generate an <img>
with a title and an alt text, and I have a
blog_post
template which has the <html>
and <head>
elements,
and whose <body>
is just the contents of a variable.
Non-markdown files that are run through the templating engine once. These may also contain content from other pages (e.g. the All Posts page includes the first paragraph of every post).
Markdown files. These are first passed through the templating engine to resolve small macros, then passed through the Markdown renderer to become HTML, then passed through the templating engine again to be included in a layout template.
The generator needs to know the dependencies between files so that it can decide in what order to process them in batch mode. And in polling mode, it needs to determine which processing steps need to be re-run when a file is updated. Graphically, the dependencies look like this:
The graph looks clean, but the dependencies are hard to pin down. For
instance, the only way to tell that “post 69” depends on the
img_float
template is to parse the post’s text. That’s annoying in
itself, but it might not be enough because img_float
may also
include other templates. The only way to fully resolve this is to
pass “post 69” through the Liquid templating engine, but if we do
that, then there’s no point in figuring out the dependencies between
posts and templates because the work has already been done. In
practice, we just have to assume a dependencies between “any page” and
“all the templates”. So, the dependencies really look more like this:
Now that we know that everything depends on the opaque store of templates, the next step is rendering markdown. This is easy dependency-wise because each markdown file generates one item of HTML page content.
The next problem is determining which page contents are included by
which final pages. For instance, posts/69/index.html
includes only “HTML of post 69”, but posts/index.html
includes the first HTML paragraph of all
posts, and /atom.xml
includes the full HTML contents of the most
recent five posts (or only the first paragraphs of same before the
revamp). We have to parse a page’s contents to determine what it
includes, but the inclusion is done through Liquid templates, and
Liquid templates may include other Liquid templates, and it’s again
the situation where the only way to resolve the dependencies is to do
all the templating work. So really, the graph looks like this:
Empirically, I’ve used a lot of static site generators in the last ten years, and every single one failed to handle dependencies correctly. Some didn’t even try, and some tried but failed to update pages in some cases. Given this, I wasn’t enthusiastic about trying to write my own dependency inference and tracking.
That said, if you squint hard enough at the above diagram, it begins to look like a staged process: first you load all the template files into the template store, then you read all Markdown files and pass them through Liquid, then you convert all the Markdown to HTML, and finally do the second Liquid pass to generate all the static files. The stages are sequential, but the work inside each stage can be parallelized simply in batch mode. And in polling mode, the common case is for one post Markdown to change, so the generator can skip re-processing the Liquid templates and doing the expensive Markdown→HTML conversion for any other Markdown files. Then, the final stage of creating the static files is run fully. This is wasteful, but it’s pretty fast when written in Rust—updates complete in less than one second for my blog. This is what the generator does now and it works.
Originally though, I did try to do dependency inference, failed, then realized that I could cheat if I only supported including the first paragraph of posts in other pages. The trick was that I never used templates in the first paragraph, so I could pass the source through just the Markdown renderer to get the final output. These first paragraphs were then stored separately from the posts and could be included in other pages. In practice, this hack only needed to work for All Posts and the RSS feed, so it was good enough.
With full posts in the Atom feed, the only remaining problem was relative URLs. Pages on this site use internal links like the following:
<a href="/posts/68/">...</a>
<a href="#conclusion">...</a>
<img src="/r/68-dive-container-screenshot-small.png">
I like these relative links because they look clean, but they’re
unlikely to work in RSS readers. Since I didn’t want to change the
standalone pages, I instead wrote a small Liquid
Filter
to make the URLs absolute:
fn absolutize_urls(
input: &dyn ValueView,
site_base_url: &str,
page_rel_url: &str,
) -> OrError<Value> {
use lol_html::{element, html_content::Element, rewrite_str, Settings};
if input.is_nil() {
return Ok(Value::Nil);
}
let html = input.to_kstr();
let absolutize_f = |attr| {
move |el: &mut Element| {
if let Some(href) = el.get_attribute(attr) {
if href.starts_with('/') {
el.set_attribute(attr, &format!("{site_base_url}{href}"))
.unwrap();
} else if href.starts_with('#') {
el.set_attribute(
attr,
&format!("{site_base_url}{page_rel_url}{href}"),
)
.unwrap();
}
}
Ok(())
}
};
let html = rewrite_str(
&html,
Settings {
element_content_handlers: vec![
element!("a[href]", absolutize_f("href")),
element!("img[src]", absolutize_f("src")),
],
..Settings::default()
},
)?;
Ok(Value::scalar(html))
}
This uses the lol_html
crate to find <a>
and
<img>
elements whose URLs begin with a /
and prepends the site’s
base URL to them. It also finds URLs that start with a #
and
prepends both the site’s base URL and the page’s relative URL to them.
I then called the filter in the atom.xml
template like so:
<summary type="html">
<![CDATA[{{post.description | absolutize_urls: "https://scvalex.net", rel_url | unclassify}}]]>
</summary>
<content type="html">
<![CDATA[{{post.content_html | absolutize_urls: "https://scvalex.net", rel_url | unclassify}}]]>
</content>
There’s also a second filter in there to remove the class
attributes
from elements since CSS-in-separate-files doesn’t get loaded by RSS
readers.
Writing an app to generate my site has been a mixed bag, but mostly filled with goodness. The bad part is that I occasionally have to write something like an RSS feed generator from scratch. In theory, if I used an off-the-shelf static site generator, this would be done for me. In practice, I’ve run into issues I couldn’t stomach in just about every static site generator that I’ve tried; sometimes it was output instability across upgrades, sometimes it was questionable output, and sometimes it was lack of features and of configurability—there was always something. Ever since I wrote my own generator, whenever I need a new feature, I can just write it. This brings a lot of peace of mind.
]]>Let’s build a container around a Rust webserver and some static files using Nix and Gitlab CI. The process is what you’d expect, but there are a few details that are annoying to puzzle out.
Let’s briefly look at the final config, then go through the interesting bits in later sections.
First up is the Nix flake which has the build
definitions. The packages.site
output is the Rust
webserver built with naersk
. Then there’s the packages.container
output created with
buildLayeredImage
from nixpkgs
. The
container includes both the site
binary and the ./dist
directory
of static files. The latter doesn’t have a dedicated package and is
just included as-is.
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
flake-utils.url = "github:numtide/flake-utils";
naersk = {
url = "github:nix-community/naersk";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs = { self, nixpkgs, flake-utils, naersk }:
flake-utils.lib.eachDefaultSystem (
system:
let
pkgs = nixpkgs.legacyPackages."${system}";
naersk-lib = naersk.lib."${system}";
in
rec {
packages.site = naersk-lib.buildPackage {
src = ./site;
pname = "site";
};
defaultPackage = packages.site;
packages.container = pkgs.dockerTools.buildLayeredImage {
name = "scvalex.net";
tag = "flake";
created = "now";
config = {
ExposedPorts = { "80/tcp" = { }; };
Entrypoint = [ "${packages.site}/bin/site" ];
Cmd = [ "serve" "--dist-dir" ./dist "--listen-on" "0.0.0.0:80" ];
};
};
}
);
}
flake.nix
Next up is the Gitlab CI config file which tells the
build host how to actually build our project. Essentially, we want to
create the container with nix build .#container
and then upload it
to our registry with skopeo
. Practically, there’s lots of
pomp and ceremony around this:
build-container:
image:
name: "nixos/nix:2.12.0"
variables:
CACHIX_CACHE_NAME: scvalex-scvalex-net
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
before_script:
- nix-env --install --attr nixpkgs.cachix
- nix-env --install --attr nixpkgs.skopeo
- cachix use "$CACHIX_CACHE_NAME"
script:
- mkdir -p "$HOME/.config/nix"
- echo 'experimental-features = nix-command flakes' > "$HOME/.config/nix/nix.conf"
- mkdir -p "/etc/containers/"
- echo '{"default":[{"type":"insecureAcceptAnything"}]}' > /etc/containers/policy.json
- cachix watch-exec "$CACHIX_CACHE_NAME" nix build .#container
- skopeo login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
- ls -lh ./result
- 'skopeo inspect docker-archive://$(readlink -f ./result)'
- 'skopeo copy docker-archive://$(readlink -f ./result) docker://$IMAGE_TAG'
.gitlab-ci.yml
With the overview done, let’s zoom in on the Nix flake. It starts with the usual boilerplate for Rust flakes (see A NixOS flake for Rust, egui, and OpenGL if you need a refresher).
The first interesting bit is the definition of the packages.site
output:
packages.site = naersk-lib.buildPackage {
src = ./site;
pname = "site";
};
These unassuming four lines not only tell Nix how to build our Rust project, they also implicitly specify the runtime dependencies of the binary.
$ ldd site/target/release/site
linux-vdso.so.1 (0x00007ffd7b787000)
libstdc++.so.6 => /nix/store/2vqp383jfrsjb3yq0szzkirya257h1dp-gcc-11.3.0-lib/lib/libstdc++.so.6 (0x00007fd89673b000)
libgcc_s.so.1 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/libgcc_s.so.1 (0x00007fd896721000)
libm.so.6 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/libm.so.6 (0x00007fd896641000)
libc.so.6 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/libc.so.6 (0x00007fd896438000)
/nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/ld-linux-x86-64.so.2 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007fd897749000)
Just copying the binary to an empty container wouldn’t work because it
would be missing all of its dynamic libraries. We could manually
include these in the container definition, but then they might not be
the right versions. Luckily, we don’t have to worry about any of this
because naersk
handles all the details for us as
long as we use it to build the project.
Next is the definition of the packages.container
output:
packages.container = pkgs.dockerTools.buildLayeredImage {
name = "scvalex.net";
tag = "flake";
created = "now";
config = {
ExposedPorts = { "80/tcp" = { }; };
Entrypoint = [ "${packages.site}/bin/site" ];
Cmd = [ "serve" "--dist-dir" ./dist "--listen-on" "0.0.0.0:80" ];
};
};
This uses buildLayeredImage
. There
are a few other ways to build containers with Nix, but this is the one
that seems most mature.
The name
and tag
fields are self-explanatory. It doesn’t matter
what we pick here because we’ll rename the image when pushing it to
the container registry.
The ExposedPorts
, Entrypoint
, and Cmd
fields are the same ones
from regular Dockerfiles
. The interesting bit is
that we can just refer to packages.site
and ./dist
(note the lack
of double-quotes) directly. This causes Nix to include them and all
their dependencies in the image. We don’t need a separate step where
we list the contents of the container (although we could do that if we
wanted to; see the docs).
Once built with nix build .#container
, we can inspect the result
with dive
:
$ nix build .#container
$ gunzip --stdout ./result > ~/tmp/container.tar
$ dive ~/tmp/container.tar --source docker-archive
In the left pane, we see the six layers of the container. Each
contains a Nix store path and is a few megabytes in size. In the
right pane, we see the final filesystem of the container: it’s just
one big /nix/store
. We see the site
binary, the dist
directory
of static files, and the gcc
, glibc
, libidn2
, and libunistring
dependencies.
The only thing we haven’t talked about is the created = "now"
option
to buildLayeredImage
. If we don’t set
it, the image gets a creation time of 1970-01-01 00:00:01Z. When we
later push this to Gitlab, it will look like this in the UI:
As far as I can tell, this doesn’t interfere with anything. In
particular, the container registry garbage collection can still figure
out which is the actual oldest image to delete. That said, it does
make the UI harder to read, and since I don’t need reproducible
container builds, I just set created
to now
to get real
timestamps.
So far, we’ve told Nix how to build our project and container. Next, we need to tell Gitlab CI how to run the Nix build and how to upload the container image to the registry.
First off, we use a recent nixos/nix
image so that we don’t have to
worry about updating channels.
build-container:
image:
name: "nixos/nix:2.12.0"
Next, we set a couple of variables, install cachix
and
skopeo
in the build container, and configure cachix
. We
don’t strictly need the latter, but it makes the Nix builds
significantly faster by caching intermediate artifacts. The $IMAGE_TAG
is going to be the name of our final image
and it follows this format: registry.gitlab.com/NAMESPACE/PROJECT:BRANCH
. The $CI_REGISTRY_IMAGE
and $CI_COMMIT_REF_SLUG
variables are set automatically by Gitlab.
variables:
CACHIX_CACHE_NAME: scvalex-scvalex-net
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
before_script:
- nix-env --install --attr nixpkgs.cachix
- nix-env --install --attr nixpkgs.skopeo
- cachix use "$CACHIX_CACHE_NAME"
Next, we enable Nix flakes:
script:
- mkdir -p "$HOME/.config/nix"
- echo 'experimental-features = nix-command flakes' > "$HOME/.config/nix/nix.conf"
Next, we disable checking for container image signatures because we know exactly where our image came from and where it’s going:
- mkdir -p "/etc/containers/"
- echo '{"default":[{"type":"insecureAcceptAnything"}]}' > /etc/containers/policy.json
If we don’t set the above policy, we get errors like the following:
time="2022-12-13T14:57:35Z" level=fatal msg="Error loading trust policy: open /etc/containers/policy.json: no such file or directory"
skopeo
error if the policy is not setError: payload does not match any of the supported image formats:
* oci: open /etc/containers/policy.json: no such file or directory
* oci-archive: open /etc/containers/policy.json: no such file or directory
* docker-archive: open /etc/containers/policy.json: no such file or directory
* dir: open /etc/containers/policy.json: no such file or directory
podman
error if the policy is not setNext, we build our container and wrap the invocation with
cachix
:
- cachix watch-exec "$CACHIX_CACHE_NAME" nix build .#container
Finally, we log in to the container registry with skopeo
,
output some debugging information, and upload the image to the
registry. The $CI_REGISTRY
, $CI_REGISTRY_USER
, $CI_REGISTRY_PASSWORD
variables are set automatically by Gitlab.
- skopeo login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
- ls -lh ./result
- 'skopeo inspect docker-archive://$(readlink -f ./result)'
- 'skopeo copy docker-archive://$(readlink -f ./result) docker://$IMAGE_TAG'
An alternative to skopeo
is podman
which has the
advantage of being more familiar to people. To use it, we just
replace all the skopeo
related lines with:
- nix-env --install --attr nixpkgs.podman
- podman login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
- podman load < ./result
- podman image ls --all
- podman push scvalex.net:flake "$IMAGE_TAG"
The disadvantage of podman
is that we introduce an intermediate step
of loading the image to the build container’s local registry.
And there you have it: the recipe for bundling a Rust program and some files into a container using Nix and Gitlab CI. Like everything else involving multiple complex systems, it’s conceptually easy to do, but practically complicated to configure.
]]>Currency
type in Rust. We’ll iterate through several
versions, starting with the ubiquitous str
, and finishing with a
stack-allocated type built with Rust’s new const
generics.
]]>
Let’s design a Currency
type in Rust. We’ll iterate through several
versions, starting with the ubiquitous str
, and finishing with a
stack-allocated type built with Rust’s new const
generics.
The full code for this post is available here.
To begin with, let’s not even bother with type-safety. After all, everything’s a string or byte array if you go down far enough in the stack.
#[derive(Debug)]
pub struct Event {
pub account: String,
pub net_cash: f64,
pub gbp_cash: f64,
pub currency: String,
pub narrative: String,
}
pub fn handle_event(acc: &str, net_cash: f64, cur: &str, narrative: &str) -> Event {
info!(
"Account {} {}ed with {} {}: {}",
acc,
if net_cash < 0.0 { "debit" } else { "credit" },
net_cash,
cur,
narrative
);
Event {
account: acc.into(),
net_cash,
gbp_cash: fx_convert(net_cash, cur, "GBP"),
currency: cur.into(),
narrative: narrative.into(),
}
}
pub fn fx_convert(amount: f64, _cur1: &str, _cur2: &str) -> f64 {
amount * 1.2
}
This is the code we’ll be iterating over. It’s a stub of some
financial logic. The bit we’re interested in is the handling of the
currency values: the currency
field of Event
, the cur
parameter
of handle_event
, and the two cur[12]
parameters of fx_convert
.
The two aspects we want to improve on are type-safety and ergonomics.
The code above isn’t very type-safe because it’s trivial to use
incorrect &str
values as currencies. For example, the intended use
of the function is obvious if you look at its implementation, but
imagine running into this call in the wild: handle_event("EUR", 100.0, "ACC1", "Credit 100€")
. It compiles, and it’s wrong, but it
doesn’t look wrong.
Ergonomics-wise, using &str
everywhere is pretty ok, but it’s going
to get worse when we introduce strong types, and then it’s going to
get better again as we iterate.
enum
The first thing we might do is consider the problem domain. There’s a finite and small number of countries, so isn’t there some fixed list of currencies? Indeed there is: ISO4217 has the list of all valid currency codes.
So, we could create an enum
of all the valid currencies and have
some functions convert back and forth between &str
and Currency
:
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub enum Currency {
GBP,
EUR,
}
impl fmt::Display for Currency {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Currency::GBP => write!(f, "GBP"),
Currency::EUR => write!(f, "EUR"),
}
}
}
impl str::FromStr for Currency {
type Err = eyre::Error;
fn from_str(str: &str) -> Result<Self, Self::Err> {
match str {
"EUR" => Ok(Currency::EUR),
"GBP" => Ok(Currency::GBP),
_ => bail!("Unknown currency: '{str}'"),
}
}
}
#[derive(Debug)]
pub struct Event {
pub account: String,
pub net_cash: f64,
pub gbp_cash: f64,
pub currency: Currency,
pub narrative: String,
}
pub fn handle_event(acc: &str, net_cash: f64, cur: Currency, narrative: &str) -> Event {
info!(
"Account {} {}ed with {} {}: {}",
acc,
if net_cash < 0.0 { "debit" } else { "credit" },
net_cash,
cur,
narrative
);
Event {
account: acc.into(),
net_cash,
gbp_cash: fx_convert(net_cash, cur, Currency::GBP),
currency: cur,
narrative: narrative.into(),
}
}
pub fn fx_convert(amount: f64, _cur1: Currency, _cur2: Currency) -> f64 {
amount * 1.2
}
This is pretty great in all regards, except that it blows up on unknown currencies. We could list all the ISO4217 currency codes, or we could use one of the several crates that do this, but we’d still have a problem whenever the list changed.
To illustrate, let me share a story from an old job. We had a variant
type of currencies very much like the above, and everything was great
until Venezuela changed its currency on short notice. In 2018, they
replaced VEF
with VES
. Of course, we had to support the new
currency in a hurry because of business reasons, but when we looked,
there was a huge number of production systems that had been rolled
linking to the library with the hard-coded currency codes. My team
was familiar with some of the systems, and we could roll them on short
notice, but most of the list was random apps, some of which didn’t
even have listed maintainers and hadn’t been rolled in years. Worse,
we couldn’t even tell which of the apps were actually affected by the
change in the currency list because some only depended on the library
indirectly, and not all would be expected to see the new currency in
practice. Ultimately, we identified a few key systems that absolutely
had to support the new currency, rolled them, and let everyone else
handle errors in their apps whenever they popped up. This was clearly
suboptimal, but it was good enough… until a couple of years later
when we had to support currencies like Tether (USDT
)—again for
business reasons—and then the floodgates were open for random
strings as currencies.
The core issue here is that “currency” is a real-world concept, and our code has to conform. We can’t just hardcode a list of currencies, and hope the world agrees and never changes. Our currency type must be future proof and has to support everything that will get thrown at it.
There is an obvious workaround to the problem with the enum
: we
could add an Other(String)
variant. This way, when we encounter a
currency that we hadn’t hardcoded a variant for, we could just shove
it into Other
. Practically, this just makes our type a newtype
wrapper around String
with extra complexity around comparisons.
It’s not good.
So, we’re stuck with using some dynamic type like a String
. The
first thing we can do is define a type alias:
type Currency = String;
pub struct Event {
...
pub currency: Currency,
...
pub fn handle_event(acc: &str, net_cash: f64, cur: &Currency, narrative: &str) -> Event {
...
This helps with documenting the struct and function signature, but
doesn’t do anything for type-safety since we can still mix-up any
String
in place of a Currency
.
The construct that actually improves type-safety is a newtype.
I think the word “newtype” comes from Haskell where it’s an actual syntactic construct. At least, that’s why I’ve always called the pattern this.
Practically, we define a struct
with a single String
field. Since
this is a completely new type, we have to re-implement all the traits
for it. Some we can derive, and some we have to write ourselves.
There are crates like aliri_braid
that automate
the boilerplate, but we’ll write it out manually to see what’s going
on.
#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct Currency(String);
impl fmt::Display for Currency {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
}
}
impl From<&str> for Currency {
fn from(str: &str) -> Self {
Self(str.to_string())
}
}
// More impls here: Serialize/Deserialize, ToSql/FromSql,
// From<String>, From<&String>, etc.
#[derive(Debug)]
pub struct Event {
pub account: String,
pub net_cash: f64,
pub gbp_cash: f64,
pub currency: Currency,
pub narrative: String,
}
pub fn handle_event(acc: &str, net_cash: f64, cur: &Currency, narrative: &str) -> Event {
info!(
"Account {} {}ed with {} {}: {}",
acc,
if net_cash < 0.0 { "debit" } else { "credit" },
net_cash,
cur,
narrative
);
Event {
account: acc.into(),
net_cash,
gbp_cash: fx_convert(net_cash, cur, &"GBP".into()),
currency: cur.clone(),
narrative: narrative.into(),
}
}
pub fn fx_convert(amount: f64, _cur1: &Currency, _cur2: &Currency) -> f64 {
amount * 1.2
}
This is pretty good for type-safety: we can no longer pass an account
string in place of a currency, and we have to explicitly turn strings
into currencies with .into()
. It’s a bit of a pain to use, though.
Having to call clone()
whenever we move a currency value adds noise
to the code, and having to write &"GBP".into()
every time we have a
special case gets old very quickly.
If all we cared about was type-safety, this blogpost could end here, but I don’t think that’s enough. In my experience, if code is ugly to read and annoying to write, then mistakes creep in. They might not be typing mistakes, but that’s little solace after shipping a buggy program.
For instance, imagine code like the following. It uses our Currency
for currencies and BigDecimal
for numbers, both
types requiring explicit references and clones.
let ev = FxEvent {
cur1_amount: cur1_amount.clone(),
cur2_amount: &cur1_amount / fx_rate(&cur1, &cur2),
cur1: cur1.clone(),
cur2: cur1.clone(),
}
The bug is easy to spot in this 5 line snippet, but imagine if this was in the middle of 1000 lines of convoluted business logic.
If we didn’t have all the clones and references, the code would be the following, and it would be basically impossible to get wrong:
let ev = FxEvent {
cur1_amount,
cur2_amount: cur1_amount / fx_rate(cur1, cur2),
cur1,
cur2,
}
Let’s look at the two problems one at a time. What’s going on with
the clone
calls? Our type contains a String
value, which is
heap-allocated and doesn’t implement Copy
, so Rust
requires us to manually create copies of it by calling clone()
.
There’s no way around this: we could change the specific way our type
contains the string using an Rc<_>
or a
Cow<_>
or something else, but as long as the type
potentially contains a heap-allocated value, it cannot have Copy
.
As for the string conversions, we’d rather define a GBP
constant
once and use it everywhere instead of creating fresh "GBP"
values in
lots of places. The problem is that we can’t use heap-allocated types
as const
values, so no constant String
s, and no constant types
that contain String
s. We could work around this with
lazy_static
, but then we’d have to refer to the
constant with *GBP
or &*GBP
which looks ugly.
To fix both our problems, we need to stack-allocate our type, and for that, we need to give it a fixed size known at compile time.
A solution would be to wrap a &'a str
instead of a String
. The
reference has a size known at compile time, so it can have Copy
.
The problem is that we now have a lifetime to deal with, and this
lifetime will “infect” any type that contains Currency<'a>
, and any
function that uses it:
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct Currency<'a>(&'a str);
impl<'a> fmt::Display for Currency<'a> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
}
}
impl<'a> From<&'a str> for Currency<'a> {
fn from(str: &'a str) -> Self {
Self(str)
}
}
#[derive(Debug)]
pub struct Event<'a> {
pub account: String,
pub net_cash: f64,
pub gbp_cash: f64,
pub currency: Currency<'a>,
pub narrative: String,
}
pub fn handle_event<'a>(
acc: &str,
net_cash: f64,
cur: Currency<'a>,
narrative: &str,
) -> Event<'a> {
...
All the <'a>
annotations don’t look great, and if we added more
types containing references, it would only get worse. Additionally,
this limits what we can do with Currency<'a>
because we can only use
it within the lifetime of 'a
. For instance, if we read the currency
from a file or a database, we wouldn’t be able to return the value to
a higher level in the program, and we wouldn’t be able to use it in a
closure or a future either. While storing references like this might
make sense if we were talking about lots of data, our Currency
is
effectively 3 bytes, so all this awkwardness is unjustified.
We could work around this by using the 'static
lifetime instead of
'a
. The former doesn’t need to be propagated through enclosing
types, so it wouldn’t make the code uglier, and it also wouldn’t
complicate our borrowing story since all lifetimes are smaller than
'static
. The problem is that this is effectively the enum
solution all over again: we’d have to hardcode all the possible
currencies and blow up whenever we encounter a new one.
This won’t do. A good solution needs to be fixed size, clean to write, and also support arbitrary strings.
A newish addition to the Rust language is const
generics. This essentially lets us parametrize
types over primitive values. That is, before you could have Vec<T>
where T
was some type, but now you can also have Array<N>
where
N
is a number. Instead of basing our newtype on String
, we’ll
instead have it wrap a new type called FixedStr<N>
:
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct FixedStr<const N: usize> {
buf: [u8; N],
}
impl<const N: usize> FixedStr<N> {
pub fn as_str(&self) -> &str {
// By construction, this should never fail
str::from_utf8(&self.buf[..]).expect("invalid utf8 in FixedStr")
}
}
impl<const N: usize> fmt::Display for FixedStr<N> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.as_str())
}
}
impl<const N: usize> str::FromStr for FixedStr<N> {
type Err = eyre::Error;
fn from_str(str: &str) -> Result<Self, Self::Err> {
// This fails if `str` isn't exactly `N` bytes long.
Ok(Self {
buf: str.as_bytes().try_into()?,
})
}
}
A FixedStr<N>
is a struct that contains a single field which is an
array of length N
of u8
values. Our currency is going to be a
[wrapper around] FixedStr<3>
.
We have to use an array instead of an str
because, unlike the
latter, arrays have a size known at compile time and can be
stack-allocated. But users of our type would rather deal with str
s,
so we handle conversions to and from in the as_str()
function and
the FromStr
implementation. Note that the conversion is fallible in
both directions; we could avoid some of this because we know that the
contents of the array are always valid UTF-8 by construction, but that
would require using unsafe
, and that would muddy the presentation.
The most important thing we get from this custom type is the Copy
trait. Rust knows that this type is safe to bitwise copy, so we don’t
have to ever manually clone
it.
With our FixedStr<N>
type in hand, we define our newtype:
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct Currency(FixedStr<3>);
impl fmt::Display for Currency {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0.as_str())
}
}
impl str::FromStr for Currency {
type Err = eyre::Error;
fn from_str(str: &str) -> Result<Self, Self::Err> {
// This fails if `str` isn't exactly 3 bytes long.
Ok(Self(
str.parse().wrap_err("converting '{str}' into Currency")?,
))
}
}
// More impls here: Serialize/Deserialize, ToSql/FromSql,
// TryFrom<String>, TryFrom<&String>, etc.
#[derive(Debug)]
pub struct Event {
pub account: String,
pub net_cash: f64,
pub gbp_cash: f64,
pub currency: Currency,
pub narrative: String,
}
pub fn handle_event(
acc: &str,
net_cash: f64,
cur: Currency,
narrative: &str,
) -> eyre::Result<Event> {
info!(
"Account {} {}ed with {} {}: {}",
acc,
if net_cash < 0.0 { "debit" } else { "credit" },
net_cash,
cur,
narrative
);
Ok(Event {
account: acc.into(),
net_cash,
gbp_cash: fx_convert(net_cash, cur, "GBP".parse()?),
currency: cur,
narrative: narrative.into(),
})
}
pub fn fx_convert(amount: f64, _cur1: Currency, _cur2: Currency) -> f64 {
amount * 1.2
}
Since the underlying type has Copy
, so does our newtype. As a
result, we no longer pass references to it around, and we no longer
have clone()
calls. There’s still the fairly ugly "GBP".parse()?
,
but we’ll fix that a bit later.
Also, since the underlying type has fallible conversions to and from
&str
, so does our type: we now have FromStr
and TryFrom
instead
of just From
.
This is all pretty good, but there’s an obvious question we’ve been
avoiding. What about the other stringy types? Specifically, what
about accounts? Clearly those should have their own type, but
FixedStr<N>
won’t work. Account strings are usually assigned by
banks, brokers, and clearing firms and they don’t all have the same
length. That said, we can reasonably assume they have some max
length, so let’s add support for that to our type.
We replace FixedStr<N>
with StackStr<N>
. The big new thing is the
len: u8
field. When we store a string shorter than N
, we’ll put
its length in len
. I’m choosing to make the field a u8
because
Rust is going to be copying values of this type around willy-nilly, so
it really shouldn’t be too big. Despite this, the const generics
parameter still has to be of type usize
because the size of the
array must always be of type usize
. We’ll just have
to do some runtime checks to enforce the maximum size.
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct StackStr<const N: usize> {
len: u8,
buf: [u8; N],
}
impl<const N: usize> StackStr<N> {
pub fn as_str(&self) -> &str {
// By construction, this should never fail
str::from_utf8(&self.buf[..self.len as usize]).expect("invalid utf8 in FixedStr")
}
}
impl<const N: usize> fmt::Display for StackStr<N> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.as_str())
}
}
impl<const N: usize> str::FromStr for StackStr<N> {
type Err = eyre::Error;
fn from_str(str: &str) -> Result<Self, Self::Err> {
let bytes = str.as_bytes();
let len = bytes.len();
if len > u8::MAX as usize {
bail!("StackStr can be at most {} big", u8::MAX);
}
if len >= N {
bail!("String '{str}' does not fit in StackStr<{N}>");
}
let mut buf = [0; N];
buf.as_mut_slice()[0..len].copy_from_slice(bytes);
Ok(Self {
len: len as u8,
buf,
})
}
}
The small change is in as_str()
. Instead of returning
&self.buf[..]
as UTF-8, we instead return &self.buf[..self.len as usize]
.
The big change is in FromStr
. It’s fundamentally the same as
before, but with added checks that the maximum length isn’t greater
than 256, and that the given string doesn’t exceed the maximum length
from the type.
The StackStr<N>
struct is packed by default and takes N+1 bytes of
memory.
With the underlying type done, we just need to define our Currency
newtype. It’s just the same code as before. And in order to make an
Account
newtype, we just have to copy-paste that code again… let’s
automate the boilerplate generation.
macro_rules!
A very simple way to generate boilerplate in Rust is with
macro_rules!
. For the most part, we just write
the code we want the macro to generate with holes that are filled in
by the macro arguments. It’s very much like cpp
macros,
except there’s no chance of syntax tokens getting combined in weird
ways, and it’s easy to make multi-line macros.
macro_rules! newtype {
($name:ident, $size:literal) => {
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct $name(StackStr<$size>);
impl $name {
pub const fn new_unchecked(buf: [u8; $size], len: u8) -> Self {
Self(StackStr::new_unchecked(buf, len))
}
}
impl fmt::Display for $name {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0.as_str())
}
}
impl str::FromStr for $name {
type Err = eyre::Error;
fn from_str(str: &str) -> Result<Self, Self::Err> {
// This fails if `str` is too big
Ok(Self(str.parse().wrap_err("converting '{str}' into $name")?))
}
}
};
}
newtype!(Currency, 4);
newtype!(Account, 8);
Once we have the macro, we define each newtype with a single line of code. While not the ideal way of creating newtypes—my kingdom for an Ocaml functor—it’s good enough.
const
valuesThe only remaining niggle is the ad-hoc Currency
creations like
&"GBP".into()
. We’d rather these be constants that we could just
reference with names like GBP
. Unfortunately, this is where our
luck runs out. With stable Rust, the best I could come up with is
this const
constructor:
impl<const N: usize> StackStr<N> {
pub const fn new_unchecked(buf: [u8; N], len: u8) -> Self {
Self { len, buf }
}
}
const GBP: Currency = Currency::new_unchecked([b'G', b'B', b'P', 0], 3);
const EUR: Currency = Currency::new_unchecked([b'E', b'U', b'R', 0], 3);
This works, but is pretty ugly, and there’s nothing to stop us from
constructing an invalid value by getting the len
wrong, or by using
bad UTF-8 code points in the array.
At some point in the future, the following will be possible, but we
need the const_mut_refs
unstable feature to get it to
compile today:
impl<const N: usize> StackStr<N> {
// Doesn't compile
pub const fn new_from_str(str: &'static str) -> Self {
let mut buf = [0; N];
unsafe {
std::ptr::copy_nonoverlapping(str.as_ptr(), &mut buf as *mut _, str.len());
}
Self {
len: str.len() as u8,
buf,
}
}
}
Something could also probably be implemented with procedural macros, but that seems like a disproportionate amount of work for what is ultimately going to be just a few lines requiring careful coding.
To sum up, we’ve seen how to create a type-safe and ergonomic
Currency
type through the judicious use of newtypes. The
“application” code looks pretty good too, and there’s no opportunity
to misuse the stringy types:
#[derive(Debug)]
pub struct Event {
pub account: Account,
pub net_cash: f64,
pub gbp_cash: f64,
pub currency: Currency,
pub narrative: String,
}
pub fn handle_event(
account: Account,
net_cash: f64,
currency: Currency,
narrative: &str,
) -> eyre::Result<Event> {
info!(
"Account {} {}ed with {} {}: {}",
account,
if net_cash < 0.0 { "debit" } else { "credit" },
net_cash,
currency,
narrative
);
Ok(Event {
account,
net_cash,
gbp_cash: fx_convert(net_cash, currency, GBP),
currency,
narrative: narrative.into(),
})
}
pub fn fx_convert(amount: f64, _cur1: Currency, _cur2: Currency) -> f64 {
amount * 1.2
}
It may seem like it took a lot of work, but most of it was in the old
versions. The final version is only 42 lines for StackedStr<N>
, and
26 lines for the newtype!
macro. With these ~70 lines in place, we
can define more newtypes in just single lines of code, so there’s
really no excuse not to promote all stringy types to actual types.
Thanks to Francesco Mazzoli for reading drafts of this post.
Let’s write an app that watches some files for changes, runs them through Liquid templates (rs), and then compiles the output with LaTeX. It’ll take about 200 lines of code. This scheme is very useful when you need to produce some kind of document from data available to a Rust program, and when you expect to be iterating on the templates a lot.
The full code for this post is available here.
We’re going to write a little app that generates party invitations. The example is a bit contrived, but demonstrates the basic scheme:
The user writes the app, and runs it with cargo run -- --watch
.
This app is where the data comes from. Instead of trying to write
some general app that munges arbitrary data, the user is supposed to
adapt the code to their own needs. In our case, it reads a CSV
file, but it could just as easily query a database or scrape the
web. Since gathering the data is done only once per run, it’s fine
if it’s slow.
The app watches the templates/
directory for changes. On every
change, the files in templates/
are run through the Liquid
templating engine. This might sound slow, but it’s actually fast
enough on small amounts of data. Mind you, “small” on a 2018
mid-range Thinkpad is still larger than most usecases. For instance,
it takes my laptop under a second to prepare the source for a 400
page PDF, and it takes two seconds to process the 400 files which
make up this website, although that also involves copying files,
translating markdown, and shelling out to external commands.
The output is saved to the out/
directory, and xelatex
is run on
one of the files to generate the final PDF.
I mention speed here because generating this website used to take 30 seconds with a Node.js app. Rust cut the time down to 2 seconds. To be fair, the Node app was more complicated, used React to compose the site, and Astro to partially hydrate and generate static pages, but that’s par for the course in that ecosystem, and the output of the Node.js and Rust apps was almost byte-for-byte identical.
The example in this post is for a PDF document generator, but the
scheme can easily be adapted to make other things. As mentioned
above, this website is generated like this, the differences being that
it also translates markdown to HTML before applying templates, and
that it calls html-tidy
on the output instead of xelatex
.
Let’s build our app from the bottom up. We need to check if file contents have changed, so let’s write some code to compute hashes for a given list of files.
fn hash_files(files: Vec<PathBuf>) -> OrError<HashMap<PathBuf, u64>> {
let mut hashes = HashMap::new();
for f in files {
let hash = hash_file(&f)?;
hashes.insert(f, hash);
}
Ok(hashes)
}
fn hash_file<P: AsRef<Path>>(path: P) -> OrError<u64> {
use seahash::SeaHasher;
use std::{hash::Hasher, io::Read};
let mut file = File::open(path.as_ref())?;
let mut buf = [0; 4096];
let mut hasher = SeaHasher::new();
loop {
match file.read(&mut buf)? {
0 => return Ok(hasher.finish()),
_ => hasher.write(&buf),
}
}
}
The code is self-explanatory. The only interesting bit is the choice
of hash function: we use seahash
which describes
itself as “blazingly fast”. The main concern here is speed because we
rehash all the files on every change. Even if we get hash collisions,
and some changes aren’t detected (it’s never happened to me in years
of using seahash
, but it could happen), that would be fine for our
use case because the user would just notice the lack of update, and
make some other change to the input files. So, we don’t need a strong
cryptographic hash; we just need a very fast hash.
Next, we need to get the list of files to hash. We want to scan the
templates/
directory and its subdirectories, and also ignore
temporary files. Instead of trying to write this logic from scratch,
we use ignore
:
fn scan_dir(dir: &str) -> OrError<Vec<PathBuf>> {
info!("Scanning {dir}");
use ignore::Walk;
let mut res = vec![];
for file in Walk::new(dir) {
let file = file?;
if !file.metadata()?.is_file() {
continue;
}
res.push(file.into_path());
}
Ok(res)
}
As its name implies, ignore
respects .gitignore
and other similar
files, which saves us from having to come up with our own scheme for
ignoring files.
Next, we need to watch templates/
for changes. We run a
notify-debouncer-mini
on a separate
thread. Whenever the debouncer notifies us of a change via its
mpsc::channel
, we rescan the directory,
recompute all file hashes, and if they have changed, we write the
entire list of files to the output channel.
fn run_watcher(dir: &str) -> OrError<mpsc::Receiver<Vec<PathBuf>>> {
use notify::RecursiveMode;
use std::time::Duration;
let (notify_tx, notify_rx) = mpsc::channel();
let (watcher_tx, watcher_rx) = mpsc::channel();
let watcher_loop = {
let dir = dir.to_string();
move || -> OrError<()> {
let mut debouncer =
notify_debouncer_mini::new_debouncer(Duration::from_millis(250), None, notify_tx)?;
let mut hashes = hash_files(scan_dir(&dir)?)?;
debouncer
.watcher()
.watch(Path::new(&dir), RecursiveMode::Recursive)?;
loop {
match notify_rx.recv()? {
Err(errs) => error!("Notify errors: {errs:?}"),
Ok(events) if events.is_empty() => {}
Ok(_) => {
let paths = scan_dir(&dir)?;
let new_hashes = hash_files(paths.clone())?;
if hashes != new_hashes {
hashes = new_hashes;
watcher_tx.send(paths)?;
}
}
}
}
}
};
std::thread::spawn(move || match watcher_loop() {
Ok(()) => error!("Watcher loop ended without error"),
Err(err) => error!("Watcher loop ended with error: {err}"),
});
Ok(watcher_rx)
}
Recomputing all the file hashes on every change seems like it would be
slow, and it certainly is wasteful. However, this code is part of a
human interaction loop, so it doesn’t need to be the fastest—it just
needs to be fast enough for a human not to mind. On my laptop, it
takes one second to scan and hash a directory of 9000 files of 32KB
each. That’s plenty fast. If we needed to, we could use the lower
level notify
crate, only react to file
modifications, and then only rehash the modified files. That said,
our efforts would be better spent on optimizing the template rendering
from the next section which is 11x slower.
Next, we need to run the files through the Liquid templating engine. There’s a bit of pomp and ceremony because we have to handle “partial” templates separately. Then, we render the full templates in parallel, while also passing data to them.
// Render the given templates. Files that end with ".liquid" are partial
// templates that can be `include`d in other templates. All other
// files are run through the template engine, and rendered to
// `out_dir`.
fn render_templates(
in_dir: &str,
files: Vec<PathBuf>,
out_dir: &str,
data: &HashMap<String, liquid::model::Value>,
) -> OrError<()> {
use liquid::{
partials::{EagerCompiler, InMemorySource},
ParserBuilder, ValueView,
};
use rayon::prelude::*;
info!("Rendering templates: {files:?}");
let mut templates = InMemorySource::new();
for f in &files {
let path = f.strip_prefix(in_dir)?.to_str().unwrap();
if path.ends_with(".liquid") {
templates.add(path.to_string(), std::fs::read_to_string(f)?);
}
}
let parser = ParserBuilder::new()
.stdlib()
.partials(EagerCompiler::new(templates))
.build()?;
files
.par_iter()
.map(|f| {
let mut globals: HashMap<String, &dyn ValueView> = HashMap::new();
globals.insert("data".to_string(), data as &dyn ValueView);
let path = f.strip_prefix(in_dir)?.to_str().unwrap();
if !path.ends_with(".liquid") {
let out_file = PathBuf::new().join(out_dir).join(path);
let template = parser.parse(&std::fs::read_to_string(f)?)?;
info!("Rendering to {}", out_file.display());
template.render_to(&mut File::create(out_file)?, &globals)?;
}
Ok(())
})
.collect::<Vec<OrError<()>>>()
.into_iter()
.collect::<OrError<Vec<()>>>()?;
info!("Done rendering templates");
Ok(())
}
We chose Liquid Templates because it’s good enough: the syntax is fairly clean, it’s used by a large company like Shopify, the Rust library is mature, and it’s extensible. I confess I have not put much thought into this choice, so there might be better overall choices of templating engines.
For instance, the main template for our party invitation example is
this. Liquid lets us apply the invite.liquid
partial once for each guest.
\documentclass{article}
\pagestyle{empty}
\usepackage{geometry,fontspec,tikz}
\geometry{a6paper,landscape,hmargin={1cm,1cm},vmargin={1cm,1cm}}
\setlength\parindent{0pt}
\begin{document}
\obeylines
{% for guest in data.guests %}
{% include "invite.liquid" guest: guest %}
{% endfor %}
\end{document}
“Partial” templates in this context mean templates that can be
included into other templates. These are our reusable components.
The one ugly thing about them is that variables are dynamically
scoped, so partial templates implicitly have access to all the
variables in the templates that include them. Under templates/
,
partials have the .liquid
extension. They are read and added to an
InMemorySource
in the Rust code, and that in
turn is passed to the full template Parser
.
The parallelism in this function is neatly encapsulated in the call to
Rayon’s par_iter()
. That takes care
of spawning as many threads as we have cores, and running the
rendering code on them.
The data we want to pass to the templates must first be converted to
the dynamic types of Liquid: Value
and &dyn ValueView
. There’s not much to say here other than
that we’ll see some mapping and wrapping action later.
Rendering the templates is as easy as calling parse
and render_to
. For simplicity, we just re-parse
and re-render all the templates on every file change. It takes my
laptop 11 seconds to do the 9000 templates from the previous section,
so the speed is about 800 templates/second. Like with hashing, this
could be optimized by only considering the templates which have
changed in each loop, but it’s not really necessary unless we have
multiple thousands of files.
Finally, we tie all these functions together in the main
program.
We use clap
for command line argument parsing, and
csv
to load the guest list. The code is
straightforward, but it gets messy around the aforementioned
conversions to Liquid’s dynamic types.
type OrError<T> = Result<T, anyhow::Error>;
const TEMPLATES_DIR: &str = "templates";
const OUT_DIR: &str = "out";
#[derive(Parser)]
struct Opts {
#[clap(long)]
watch: bool,
#[clap(long)]
guest_list: String,
}
#[derive(Deserialize, Serialize)]
struct Guest {
name: String,
address: String,
}
fn main() -> OrError<()> {
setup_log()?;
let Opts { watch, guest_list } = Opts::parse();
let mut data: HashMap<String, Value> = HashMap::new();
let guests: Vec<Value> = csv::Reader::from_reader(File::open(guest_list)?)
.deserialize()
.map(|r: Result<Guest, _>| Value::Object(liquid::to_object(&r.unwrap()).unwrap()))
.collect();
data.insert("guests".to_string(), Value::array(guests));
render_templates_and_compile_latex(scan_dir(TEMPLATES_DIR)?, &data)?;
if watch {
let updates = run_watcher(TEMPLATES_DIR)?;
loop {
render_templates_and_compile_latex(updates.recv()?, &data)?;
}
}
Ok(())
}
We do all the templating work in
render_templates_and_compile_latex()
once at the start, and every
time the file watcher reports a change. We’re careful to only run
LaTeX compilation if template rendering succeeded, and we run it twice
to account for internal references.
fn render_templates_and_compile_latex(
template_files: Vec<PathBuf>,
data: &HashMap<String, Value>,
) -> OrError<()> {
match render_templates(TEMPLATES_DIR, template_files, OUT_DIR, data) {
Ok(()) => {
for _ in 0..2 {
if let Err(err) =
compile_latex(PathBuf::new().join(OUT_DIR).join("invites.tex"), OUT_DIR)
{
error!("Failed to compile latex: {err}")
}
}
}
Err(err) => {
error!("Failed to render templates: {err}")
}
}
Ok(())
}
The full 237 lines of code are available here.
That’s all it takes to write a polling and templating document generator in Rust. When I wrote this the first time, I was surprised by how easy it was. Given how many static site generators there are out there, I guess I shouldn’t have been, but ~200 lines is really short. And doing it in Rust makes it fast, even if the code is simple and unoptimized. I’ve used this scheme for this website, for my yearly tax report, and I imagine I’ll be using it in the future for other things as well.
]]>