diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml new file mode 100644 index 0000000..ad5a317 --- /dev/null +++ b/.github/workflows/publish.yml @@ -0,0 +1,31 @@ +name: Publish Python packages and Docker image +on: + push: + branches: + - master +jobs: + publish: + env: + COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }} + COSIGN_PRIVATE_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }} + DOCKER_HUB_LOGIN: ${{ secrets.DOCKER_HUB_LOGIN }} + DOCKER_HUB_TOKEN: ${{ secrets.DOCKER_HUB_TOKEN }} + UV_PUBLISH_TOKEN: ${{ secrets.UV_PUBLISH_TOKEN }} + name: Package and public, Python 3${{ matrix.python-version }} + runs-on: ubuntu-26.04 + steps: + - uses: actions/checkout@v7.0.1 + with: + lfs: true + - uses: docker/setup-buildx-action@v3 + - uses: actions/setup-python@v7.0.0 + with: + python-version: 3.${{ matrix.python-version }} + - run: python3 -m pip install --upgrade tox tox-docker-id tox-uv + - run: tox -e packaging-amd64,py313,py314,coverage + - run: tox -m end-to-end + - run: tox -e publish + strategy: + matrix: + python-version: ['14'] + timeout-minutes: 75 diff --git a/README.md b/README.md index d0ffe8a..b446322 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,10 @@ Docker image) in an **air-gapped environment**: no network access is required or ever used. It reads a single configuration (JSON or YAML), and applies it to a filesystem tree, writing only the files -that need to change: root certificates, package manager sources (APT, YUM/DNF, APK), and language-specific -package manager settings (pip/uv/poetry, Maven/Gradle/Ivy/SBT/Leiningen, ...). The result can then be used -to build a Docker image or otherwise deployed to the target device. +that need to change: root certificates, system package manager sources (APT, YUM/DNF, APK) and +language-specific package manager settings (pip/uv/poetry, Maven/Gradle/Ivy/SBT/Leiningen, Composer, npm, +cargo, Go modules, RubyGems, NuGet). The result can then be used to build a Docker image or otherwise +deployed to the target device. ## How it works @@ -14,15 +15,62 @@ to build a Docker image or otherwise deployed to the target device. never writes anywhere else. - `--src` (optional) points to an existing filesystem, used to detect the current configuration (existing APT sources, existing Java keystores, etc.) so it can be extended or disabled instead of blindly overwritten. -- The configuration is loaded either from a file (`--config`) or from a base64-encoded string, typically an - environment variable (`--config-env` / `$ID_CONFIGURATOR_CONFIG`). -- Each feature is implemented as an independent, self-registering **module**. On `apply`, every module - inspects the configuration and the source filesystem and decides for itself whether it has something to - do; only relevant modules touch the destination directory. +- The configuration is loaded from a file (`--config`), from a base64-encoded string, typically an + environment variable (`--config-env` / `$ID_CONFIGURATOR_CONFIG`), or downloaded over HTTP(S) from a URL + (`--config-url`). `--config-url-cacert-content` optionally gives the PEM content of the CA certificate to trust for that one download, for an internal + endpoint signed by a private CA the running system doesn't otherwise trust yet (e.g. before the + `certificates` module has ever run) - it defaults to the system's own trust store otherwise. + `--config-url-insecure` disables certificate (and hostname) verification entirely instead, overriding + `--config-url-cacert-content` - **insecure**, only meant for a first bootstrap against a server whose + certificate isn't trustable, or trusted, yet by any other means. +- On `apply`, every module inspects the configuration and the source filesystem and decides for itself whether it has something to + do; only relevant modules touch the destination directory. For example, the configured Alpine repositories are only written if + `--src` can be identified as an Alpine filesystem (i.e. it has an `/etc/apk` directory); and a source cloning Debian's official + repository is only configured if that official repository is actually present in the filesystem given via `--src` + (via `uri-to-replace`, see "Replacing an existing repository" below). ## Installation -Requires Python 3.13+. +### As a CLI tool + +```bash +# with pip +pip install id-configurator + +# or with uv tool, in its own isolated environment +uv tool install id-configurator +``` + +Once installed this way, run it directly as `id-configurator` (rather than `uv run id-configurator`, which is +only for the "from source" setup below). + +### As a Docker image + +`id-configurator` is also published as [`interdiode/id-configurator`](https://hub.docker.com/r/interdiode/id-configurator) +on Docker Hub - handy since its output is typically meant to become another Docker image in the first place. +One-off use: + +```bash +docker run --rm -v "$(pwd)/config.yaml:/config.yaml:ro" -v "$(pwd)/output:/dst" -v "/:/src" \ + interdiode/id-configurator:latest \ + id-configurator apply --config /config.yaml --dst /dst --src /src +``` + +More typically, as a builder stage of the target image's own `Dockerfile` - `` here is the target image +itself, taken as of its `target` stage, before `id-configurator`'s own changes are copied back on top of it: + +```dockerfile +FROM debian:trixie-slim AS target +FROM interdiode/id-configurator:latest AS configuration +COPY --from=target / /src/ +COPY config.yaml /config.yaml +RUN id-configurator apply --config /config.yaml --src /src --dst /dst +FROM target +COPY --from=configuration /dst/ / +``` + + +### From source (development) ```bash uv sync @@ -32,32 +80,50 @@ uv sync ```bash # Apply a configuration file to a destination directory -uv run id-configurator apply --config config.yaml --dst ./output +id-configurator apply --config config.yaml --dst ./output # Take an existing filesystem into account (e.g. to extend its APT sources) -uv run id-configurator apply --config config.yaml --src /path/to/rootfs --dst ./output +id-configurator apply --config config.yaml --src /path/to/rootfs --dst ./output # Use a base64-encoded configuration from the environment instead of a file export ID_CONFIGURATOR_CONFIG=$(base64 -w0 config.yaml) -uv run id-configurator apply --dst ./output +id-configurator apply --dst ./output + +# Download the configuration from an internal HTTPS server instead of a local file +id-configurator apply \ + --config-url https://config.internal.example/device.yaml \ + --config-url-cacert-content "$(cat internal-ca.pem)" \ + --dst ./output # Validate a configuration without applying it -uv run id-configurator validate --config config.yaml +id-configurator validate --config config.yaml # List the available modules -uv run id-configurator list-modules +id-configurator list-modules + +# Same thing, via the Docker image instead of installing anything locally +docker run --rm \ + -v "$(pwd)/config.yaml:/config.yaml:ro" \ + -v "$(pwd)/output:/dst" \ + interdiode/id-configurator:latest \ + id-configurator apply --config /config.yaml --dst /dst ``` The tool can also be invoked as `python -m id_configurator`. ## Configuration format -The configuration is a JSON or YAML object: +The configuration is a JSON or YAML object. `uuid`, `organisation` and `description` are all optional +(identifying metadata only - nothing in the tool itself relies on any of them), as is every `configurations.*` +block - a module that finds nothing to do for it simply does not run. `version` is the configuration +*format*'s own schema version (currently `"1.0"`, and the default if omitted) - independent of the +`id-configurator` package version - so a future, incompatible format change can be told apart from this one: ```yaml -uuid: -organisation: -description: +version: "1.0" +uuid: +organisation: +description: configurations: certificates: - name: @@ -68,18 +134,20 @@ configurations: uris: [] suites: [] components: [] - keys: - - name: - content: + signed-by: # or signed-by-file: + uri-to-replace: # see "Replacing an existing repository" below apk: repositories: - - url: - key: + - name: + url: + key: + url_to_replace: yum: repositories: - name: index-url: - gpg-url: + gpg-url: + index-url-to-replace: keys: - name: content: @@ -91,20 +159,61 @@ configurations: repositories: - name: url: + php: + repositories: + - name: + url: + npm: + repositories: + - name: + url: + rust: + repositories: + - name: + url: + go: + repositories: + - name: + url: + ruby: + repositories: + - name: + url: + nuget: + repositories: + - name: + url: ``` +### Replacing an existing repository + +`apt.sources[].uri-to-replace`, `apk.repositories[].url_to_replace` and `yum.repositories[].index-url-to-replace` +are optional [`fnmatch`](https://docs.python.org/3/library/fnmatch.html) patterns. When set, the configured +repository is only written if the pattern matches at least one URI/URL already present in `--src`'s existing +configuration - and only the matching existing entries are disabled, the rest are left untouched. This lets a +configuration safely target "whatever machine this is applied to": e.g. `uri-to-replace: "http://deb.debian.org/*"` +replaces Debian's default mirror if (and only if) it's actually there, without disturbing any other source a +given machine might already have. Leaving the field unset never disables anything - the configured repository +is simply added alongside whatever is already present. + ## Modules -| Module | Applies to | -|----------------|--------------------------------------------------------------------------------------------------------| -| `certificates` | Installs root certificates into the system's CA store (`update-ca-certificates`). | -| `apt` | Disables existing APT sources and adds the configured ones, in DEB822 format. | -| `apk` | Disables existing Alpine `/etc/apk/repositories` entries and adds the configured ones, plus their keys.| -| `yum` | Disables existing YUM/DNF `.repo` files, adds the configured ones, and imports their GPG keys. | -| `python` | Points pip, uv, poetry and `easy_install` to the configured Python package indexes. | -| `java` | Adds root certificates to every detected JKS/PKCS12 keystore, and configures Maven, Gradle, Ivy, SBT and Leiningen repositories. | +| Module | Applies to | +|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| +| `certificates` | Installs root certificates into the system's CA store, rebuilding the trust bundle in pure Python (Debian/Ubuntu and RHEL/Fedora trust stores). | +| `apt` | Adds the configured APT sources (DEB822 format); disables only the existing entries matching `uri-to-replace`. | +| `apk` | Adds the configured Alpine `/etc/apk/repositories` entries and their keys; disables only the existing lines matching `url_to_replace`. | +| `yum` | Adds the configured YUM/DNF `.repo` files and imports their GPG keys; disables only the existing sections matching `index-url-to-replace`. Covers DNF too - same config files. | +| `python` | Points pip, uv, poetry and `easy_install` to the configured Python package indexes. | +| `java` | Adds root certificates to every detected JKS/PKCS12 keystore, and configures Maven, Gradle, Ivy, SBT and Leiningen repositories. | +| `php` | Registers the configured repositories with Composer; sets `openssl.cafile` whenever any certificate is configured, so PHP's own OpenSSL calls trust it too. | +| `npm` | Points npm at the configured registries and its own `cafile`; sets `NODE_EXTRA_CA_CERTS` system-wide whenever any certificate is configured, since Node ignores the system CA store. | +| `rust` | Points cargo at the configured registries, replacing crates.io. | +| `go` | Points the Go toolchain at the configured module proxies (`GOPROXY`), replacing proxy.golang.org, and disables `GOSUMDB`. | +| `ruby` | Points RubyGems at the configured sources, replacing rubygems.org. | +| `nuget` | Points NuGet at the configured package sources, replacing nuget.org. | -More modules (npm, nuget, ruby/gem, rust/crates, go, packagist/composer, ...) are planned; see `CLAUDE.md` for the full roadmap. +This tool's own modular structure is meant to be extended: see `CLAUDE.md` for the module interface and how to add one. ## Development @@ -118,4 +227,4 @@ uv run pre-commit run --all-files # run all pre-commit hooks ## License -Proprietary. See [LICENSE](LICENSE). Copyright InterDiode SA. +[CeCILL v2.1](https://cecill.info) (GPL-compatible free software license). See [LICENSE](LICENSE). Copyright InterDiode SA. diff --git a/tox.ini b/tox.ini index d6d6bc9..c233c97 100644 --- a/tox.ini +++ b/tox.ini @@ -74,15 +74,18 @@ description = The version is taken from the pyproject.toml file. Make sure that you have the rights to upload the image to Docker Hub. commands = - python3 -m uv sync --all-groups --active + python3 -m uv sync --all-groups -U --active python3 -m id_tools build docker --platform linux/amd64,linux/arm64/v8 --push python3 -m id_tools publication sign-docker + python3 -m uv build + python3 -m uv publish docker = passenv = COSIGN_PASSWORD COSIGN_PRIVATE_KEY DOCKER_HUB_LOGIN DOCKER_HUB_TOKEN + UV_PUBLISH_TOKEN [docker:pypi-repo] dockerfile = {tox_root}/tests/dockerfiles/repos/pypi-repo.Dockerfile