ariadne.space/content/blog/a-tale-of-two-envsubst-implementations.md

---
title: "A tale of two envsubst implementations"
date: "2021-04-15"
---

Yesterday, Dermot Bradley brought up in IRC that gettext-tiny's lack of an `envsubst` utility could be a potential problem, as many Alpine users [use it to generate configuration from templates](https://www.robustperception.io/environment-substitution-with-docker).  So I decided to look into writing a replacement, as the tool did not seem that complex.  That rewrite is [now available on GitHub](https://github.com/kaniini/envsubst), and is already in Alpine testing for experimental use.

## What `envsubst` does

The `envsubst` utility is designed to take a set of strings as input and replace variables in them, in the same way that shells do variable substitution.  Additionally, the variables that will be substituted can be restricted to a defined set, which is nice for reliability purposes.

Because it provides a simple way to perform substitutions in a file without having to mess with `sed` and other similar utilities, it is seen as a helpful tool for building configuration files from templates: you just install the `cmd:envsubst` provider with apk and perform the substitutions.

Unfortunately though, GNU `envsubst` is quite deficient in terms of functionality and interface.

## Good tool design is important

When building a tool like `envsubst`, it is important to think about how it will be used.  One of the things that is really important is making sure a tool is satisfying to use: a tool which has non-obvious behavior or implies functionality that is not actually there is a badly designed tool.  Sadly, while sussing out a list of requirements for my replacement `envsubst` tool, I found that GNU `envsubst` has several deficiencies that are quite disappointing.

### GNU `envsubst` does not actually implement POSIX variable substitution like a shell would

In POSIX, variable substitution is more than simply replacing a variable with the value it is defined to.  In GNU `envsubst`, the documentation speaks of _shell variables_, and then outlines the `$FOO` and `${FOO}` formats for representing those variables.  The latter format implies that POSIX variable substitution is supported, but it's not.

In a POSIX-conformant shell, you can do:

% FOO="abc\_123"
% echo ${FOO%\_\*}
abc

Unfortunately, this isn't supported by GNU `envsubst`:

% FOO="abc\_123" envsubst
$FOO
abc\_123
${FOO}
abc\_123
${FOO%\_\*}
${FOO%\_\*}

It's not yet supported by my implementation either, [but it's on the list of things to do](https://github.com/kaniini/envsubst/issues/1).

### Defining a restricted set of environment variables is bizzare

GNU `envsubst` describes taking an optional `[SHELL-FORMAT]` parameter.  The way this feature is implemented is truly bizzare, as seen below:

% envsubst -h
Usage: envsubst \[OPTION\] \[SHELL-FORMAT\]
...
Operation mode:
  -v, --variables             output the variables occurring in SHELL-FORMAT
...
% FOO="abc123" BAR="xyz456" envsubst FOO
$FOO
$FOO
% FOO="abc123" envsubst -v FOO
% FOO="abc123" envsubst -v \\$FOO
FOO
% FOO="abc123" BAR="xyz456" envsubst \\$FOO
$FOO
abc123
$BAR
$BAR
% FOO="abc123" BAR="xyz456" envsubst \\$FOO \\$BAR
envsubst: too many arguments
% FOO="abc123" BAR="xyz456" envsubst \\$FOO,\\$BAR
$FOO
abc123
$BAR
xyz456
$BAZ
$BAZ
% envsubst -v
envsubst: missing arguments
%

As discussed above, `[SHELL-FORMAT]` is a very strange thing to call this, because it is not really a shell variable substitution format at all.

Then there's the matter of requiring variable names to be provided in this shell-like variable format.  That requirement gives a shell script author the ability to easily break their script by accident, for example:

% echo 'Your home directory is $HOME' | envsubst $HOME
Your home directory is $HOME

Because you forgot to escape `$HOME` as `\$HOME`, the substitution list was empty:

% echo 'Your home directory is $HOME' | envsubst \\$HOME 
Your home directory is /home/kaniini

The correct way to handle this would be to accept `HOME` without having to describe it as a variable.  That approach is supported by my implementation:

% echo 'Your home directory is $HOME' | ~/.local/bin/envsubst HOME 
Your home directory is /home/kaniini

Then there's the matter of not supporting multiple variables in the traditional UNIX style (as separate tokens).  Being forced to use a comma on top of using a variable sigil for this is just bizzare and makes the tool absolutely unpleasant to use with this feature.  For example, this is how you're supposed to add two variables to the substitution list in GNU `envsubst`:

% echo 'User $USER with home directory $HOME' | envsubst \\$USER,\\$HOME 
User kaniini with home directory /home/kaniini

While my implementation supports doing it that way, it also supports the more natural UNIX way:

% echo 'User $USER with home directory $HOME' | ~/.local/bin/envsubst USER HOME 
User kaniini with home directory /home/kaniini

## This is common with GNU software

This isn't just about GNU `envsubst`.  A lot of other GNU software is equally broken.  Even the GNU C library [has design deficiencies which are similarly frustrating](https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html).  The reason why I wish to replace GNU software in Alpine is because in many cases, it is _defective by design_.  Whether the design defects are caused by apathy, or they're caused by politics, it doesn't matter.  The end result is the same, we get defective software.  I want better security and better reliability, which means we need better tools.

We can talk about the FSF political issue, and many are debating that at length.  But the larger picture is that the tools made by the GNU project are, for the most part, clunky and unpleasant to use.  That's the real issue that needs solving.