I needed a way to store my personal documents remotely in a secure way, so I came up with a scheme involving GCE, nbd, wireguard, and cryptsetup.

Overview

The setup is:

  • the data lives on an encrypted disk in GCE,
  • the disk is attached to some VM in GCE,
  • there is a wireguard tunnel between my local machine and the VM,
  • the VM runs nbd-server exporting the disk on the wireguard interface only,
  • the local machine runs nbd-client to import the remote disk as a block device and cryptsetup to decrypt it; the block device can then be mounted normally.

The point of this setup is for the data to live somewhere outside of my house, so that, if it burns down, I don’t lose all of my files. I also want the disk to be managed by professional sysadmins because they’re more likely than me to monitor the health of the RAID setup. I used a persistent disk in GCE, but any cloud disk would have worked just as well.

The trouble is that I also don’t want to trust whoever is actually running the cloud infrastructure, hence the nbd/wireguard/cryptesetup stuff. The point here is that the cloud disk is encrypted at rest, and is only ever decrypted on my local machine. The problem is somehow getting a block device representing the cloud disk onto the local machine so that cryptsetup can work. I do this by running nbd-server on the VM which “exports” the disk’s block device over the network. The local machine connects with nbd-client and the block device becomes available locally; the client’s device IO is then forwarded to the server. The only remaining issue is that nbd traffic is unencrypted by default, but that’s easily fixed by connecting the local machine to the VM with a wireguard tunnel and configuring nbd to only listen on the wireguard network interface.

The end result is regularly mounted filesystem on the local machine. Since it’s just a normal filesystem, things like unix tools, rsync, and PostgreSQL work without issue.

Code

Putting it all together, the commands to mount the remote disk locally look like this:

$ nbd-client SERVER-IP-IN-WIREGUARD NBD-PORT /dev/nbd0 -name NBD-VOLUME-NAME -persist
$ cryptsetup open /dev/nbd0 my-cloud-disk
$ mount /dev/mappaer/my-cloud-disk /mnt/my-cloud-disk

The /etc/nbd-server/config on the VM is:

[generic]
user = root
group = root
allowlist = false
listenaddr = SERVER-IP-IN-WIREGUARD
port = NBD-PORT

[NBD-VOLUME-NAME]
exportname = /dev/CLOUD-DISK-DEVICE
authfile = /etc/nbd-server/authfile1
flush = true
fua = true

And /etc/nbd-server/authfile1 is:

SERVER-IP-IN-WIREGUARD/24