id_configurator/README.md

# id-configurator

`id-configurator` is a CLI tool by [InterDiode](https://interdiode.fr/) that configures a device (typically a
Docker image) in an **air-gapped environment**: no network access is required or ever used.

It reads a single configuration (JSON or YAML), and applies it to a filesystem tree, writing only the files
that need to change: root certificates, system package manager sources (APT, YUM/DNF, APK) and
language-specific package manager settings (pip/uv/poetry, Maven/Gradle/Ivy/SBT/Leiningen, Composer, npm,
cargo, Go modules, RubyGems, NuGet). The result can then be used to build a Docker image or otherwise
deployed to the target device.

Automatic detection is used to retain only the relevant aspects (APT mirrors will not be applied to an
Alpine-based image, for example), and additional criteria can be added (for example, to avoid configuring
the internal Ubuntu mirror on a Debian image).


## How it works

- `--dst` points to the destination directory that receives every modified or created file. `id-configurator`
  never writes anywhere else.
- `--src` (optional) points to an existing filesystem, used to detect the current configuration (existing
  APT sources, existing Java keystores, etc.) so it can be extended or disabled instead of blindly overwritten.
- The configuration is loaded from a file (`--config`), from a base64-encoded string, typically an
  environment variable (`--config-env` / `$ID_CONFIGURATOR_CONFIG`), or downloaded over HTTP(S) from a URL
  (`--config-url`). `--config-url-cacert-content` optionally gives the PEM content of the CA certificate to trust for that one download, for an internal
  endpoint signed by a private CA the running system doesn't otherwise trust yet (e.g. before the
  `certificates` module has ever run) - it defaults to the system's own trust store otherwise.
  `--config-url-insecure` disables certificate (and hostname) verification entirely instead, overriding
  `--config-url-cacert-content` - **insecure**, only meant for a first bootstrap against a server whose
  certificate isn't trustable, or trusted, yet by any other means.
- `--user` (repeatable) names a user to also prepare *per-user* configuration for, on top of the usual
  system-wide files, for the modules that have one: `go`, `nuget`, `php` (Composer), `python` (poetry) and
  `rust`. The user's home directory is taken from `--src`'s own `/etc/passwd` entry when there is one,
  otherwise assumed to be `/home/<user>` (`/root` for `root`).
- `--auto-users` does the same for every *regular* (non-system) user found in `--src`'s `/etc/passwd`: one with
  a login shell (not `nologin`, `sync` or `false`) and a UID at or above `UID_MIN` (read from `--src`'s
  `/etc/login.defs`, 1000 by default) - except `root`, always included regardless of its UID. It combines
  freely with one or more `--user`.
- On `apply`, every module inspects the configuration and the source filesystem and decides for itself whether it has something to
  do; only relevant modules touch the destination directory. For example, the configured Alpine repositories are only written if
  `--src` can be identified as an Alpine filesystem (i.e. it has an `/etc/apk` directory); and a source cloning Debian's official
  repository is only configured if that official repository is actually present in the filesystem given via `--src`
  (via `uri-to-replace`, see "Replacing an existing repository" below).

## Installation

### As a CLI tool

```bash
# with pip
pip install id-configurator

# or with uv tool, in its own isolated environment
uv tool install id-configurator
```

Once installed this way, run it directly as `id-configurator` (rather than `uv run id-configurator`, which is
only for the "from source" setup below).

### As a Docker image

`id-configurator` is also published as [`interdiode/id-configurator`](https://hub.docker.com/r/interdiode/id-configurator)
on Docker Hub - handy since its output is typically meant to become another Docker image in the first place.
One-off use:

```bash
docker run --rm -v "$(pwd)/config.yaml:/config.yaml:ro" -v "$(pwd)/output:/dst" -v "/:/src" \
  interdiode/id-configurator:latest \
  id-configurator apply --config /config.yaml --dst /dst --src /src
```

More typically, as a builder stage of the target image's own `Dockerfile` - `<src>` here is the target image
itself, taken as of its `target` stage, before `id-configurator`'s own changes are copied back on top of it:

```dockerfile
FROM debian:trixie-slim AS target
FROM interdiode/id-configurator:latest AS configuration
COPY --from=target / /src/
COPY config.yaml /config.yaml
RUN id-configurator apply --config /config.yaml --src /src --dst /dst
FROM target
COPY --from=configuration /dst/ /
```


### From source (development)

```bash
uv sync
```

## Usage

```bash
# Apply a configuration file to a destination directory
id-configurator apply --config config.yaml --dst ./output

# Take an existing filesystem into account (e.g. to extend its APT sources)
id-configurator apply --config config.yaml --src /path/to/rootfs --dst ./output

# Use a base64-encoded configuration from the environment instead of a file
export ID_CONFIGURATOR_CONFIG=$(base64 -w0 config.yaml)
id-configurator apply --dst ./output

# Download the configuration from an internal HTTPS server instead of a local file
id-configurator apply \
  --config-url https://config.internal.example/device.yaml \
  --config-url-cacert-content "$(cat internal-ca.pem)" \
  --dst ./output

# Also prepare per-user configuration (go, nuget, php, python, rust) for specific users
id-configurator apply --config config.yaml --dst ./output --user alice --user bob

# Same, but auto-detect every regular (non-system) user from --src's /etc/passwd
id-configurator apply --config config.yaml --src /path/to/rootfs --dst ./output --auto-users

# Validate a configuration without applying it
id-configurator validate --config config.yaml

# List the available modules
id-configurator list-modules

# Same thing, via the Docker image instead of installing anything locally
docker run --rm \
  -v "$(pwd)/config.yaml:/config.yaml:ro" \
  -v "$(pwd)/input:/src" \
  -v "$(pwd)/output:/dst" \
  interdiode/id-configurator:latest \
  id-configurator apply --config /config.yaml --dst /dst --src /src
```

The tool can also be invoked as `python -m id_configurator`.

## Configuration format

The simplest way to generate a new configuration is [InterDiode](https://interdiode.fr) itself, through its
simple interface - it also handles duplicating application mirrors onto disconnected networks.

The configuration is a JSON or YAML object. `uuid`, `organisation` and `description` are all optional
(identifying metadata only - nothing in the tool itself relies on any of them), as is every `configurations.*`
block - a module that finds nothing to do for it simply does not run. `version` is the configuration
*format*'s own schema version (currently `"1.0"`, and the default if omitted) - independent of the
`id-configurator` package version - so a future, incompatible format change can be told apart from this one:

```yaml
version: "1.0"
uuid: <UUID, optional>
organisation: <string, optional>
description: <string, optional>
configurations:
  certificates:
    - name: <string>
      content: <PEM>
  apt:
    sources:
      - name: <string>
        uris: [<string>]
        suites: [<string>]
        components: [<string>]
        signed-by: <ASCII-armored GPG public key>       # or signed-by-file: <path already on the target>
        uri-to-replace: <fnmatch pattern, optional>      # see "Replacing an existing repository" below
  apk:
    repositories:
      - name: <string>
        url: <string>
        key: <RSA public key, optional>
        key_filename: <string, optional>                 # name of the key file under /etc/apk/keys/, if not the derived default
        url_to_replace: <fnmatch pattern, optional>
  yum:
    repositories:
      - name: <string>
        index-url: <string>
        gpg-url: <string, optional>
        index-url-to-replace: <fnmatch pattern, optional>
    keys:
      - name: <string>
        content: <GPG key>
  python:
    repositories:
      - name: <string>
        index-url: <string>
  java:
    repositories:
      - name: <string>
        url: <string>
  php:
    repositories:
      - name: <string>
        url: <string>
  npm:
    repositories:
      - name: <string>
        url: <string>
  rust:
    repositories:
      - name: <string>
        url: <string>
        is_sparse: <bool, optional, default false>       # declares the registry's index as sparse+<url>
  go:
    repositories:
      - name: <string>
        url: <string>
  ruby:
    repositories:
      - name: <string>
        url: <string>
  nuget:
    repositories:
      - name: <string>
        url: <string>
```

### Replacing an existing repository

`apt.sources[].uri-to-replace`, `apk.repositories[].url_to_replace` and `yum.repositories[].index-url-to-replace`
are optional [`fnmatch`](https://docs.python.org/3/library/fnmatch.html) patterns. When set, the configured
repository is only written if the pattern matches at least one URI/URL already present in `--src`'s existing
configuration - and only the matching existing entries are disabled, the rest are left untouched. This lets a
configuration safely target "whatever machine this is applied to": e.g. `uri-to-replace: "http://deb.debian.org/*"`
replaces Debian's default mirror if (and only if) it's actually there, without disturbing any other source a
given machine might already have. Leaving the field unset never disables anything - the configured repository
is simply added alongside whatever is already present.

### Non-HTTPS repositories

A repository URL doesn't have to be HTTPS - several modules automatically adjust the tool's configuration so
the plain-HTTP repository is actually usable, instead of silently producing a config the package manager
would refuse at run time:

- `python`: pip gets the repository's host added to `trusted-host`.
- `php`: Composer gets `secure-http` disabled, and `packagist.org` is disabled alongside it (an HTTP source
  implies a private, non-internet setup where the public registry wouldn't be reachable anyway).
- `nuget`: the source gets `allowInsecureConnections="true"`.
- `java`: SBT's resolver gets `allowInsecureProtocol` appended. **Leiningen is the one exception**: its
  standalone launcher refuses any non-HTTPS repository unconditionally, with no configuration able to change
  that - a repository configured here simply won't be usable from Leiningen specifically.
- `go`, `rust`, `apt`, `apk`, `yum`: no adjustment is needed - these already work the same regardless of
  scheme.

## Modules

| Module         | Applies to                                                                                                                                          |
|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| `certificates` | Installs root certificates into the system's CA store, rebuilding the trust bundle in pure Python (Debian/Ubuntu and RHEL/Fedora trust stores).       |
| `apt`          | Adds the configured APT sources (DEB822 format); disables only the existing entries matching `uri-to-replace`.                                        |
| `apk`          | Adds the configured Alpine `/etc/apk/repositories` entries and their keys; disables only the existing lines matching `url_to_replace`.                |
| `yum`          | Adds the configured YUM/DNF `.repo` files and imports their GPG keys; disables only the existing sections matching `index-url-to-replace`. Covers DNF too - same config files. |
| `python`       | Points pip, uv, poetry and `easy_install` to the configured Python package indexes.                                                                    |
| `java`         | Adds root certificates to every detected JKS/PKCS12 keystore, and configures Maven, Gradle, Ivy, SBT and Leiningen repositories.                       |
| `php`          | Registers the configured repositories with Composer; sets `openssl.cafile` whenever any certificate is configured, so PHP's own OpenSSL calls trust it too. |
| `npm`          | Points npm at the configured registries and its own `cafile`; sets `NODE_EXTRA_CA_CERTS` system-wide whenever any certificate is configured, since Node ignores the system CA store. |
| `rust`         | Points cargo at the configured registries, replacing crates.io.                                                                                        |
| `go`           | Points the Go toolchain at the configured module proxies (`GOPROXY`), replacing proxy.golang.org, and disables `GOSUMDB`.                              |
| `ruby`         | Points RubyGems at the configured sources, replacing rubygems.org.                                                                                     |
| `nuget`        | Points NuGet at the configured package sources, replacing nuget.org.                                                                                    |

`go`, `nuget`, `php`, `python` and `rust` additionally honor `--user`/`--auto-users` (see "How it works" above),
writing the same per-user configuration under each such user's home directory.

This tool's own modular structure is meant to be extended: see `CLAUDE.md` for the module interface and how to add one.

## Development

```bash
uv sync                            # install dependencies
uv run pytest                      # run the test suite
uv run ruff check --fix .          # lint
uv run ruff format .               # format
uv run pre-commit run --all-files  # run all pre-commit hooks
```

## License

[CeCILL v2.1](https://cecill.info) (GPL-compatible free software license). See [LICENSE](LICENSE). Copyright InterDiode SA.