RFC 0003 Configuration¶
| Status | Draft |
| Created | 2026-11-14 |
| Roadmap item | Migrate off OmegaConf |
Summary¶
Updates2MQTT is pinned to omegaconf>=2.3.0,<2.4.0, because OmegaConf 2.4.0 broke startup when MQTT_USER isn’t set. This RFC looks at four ways forward:
- Upgrade to OmegaConf 2.4
- Become a full Hydra app
- Move to Pydantic Settings
- Move to another established config library
Recommendation
- Now (small): upgrade to OmegaConf 2.4 with a three-line compatibility fix, so we’re on the supported release line again.
- Next (medium): migrate to
pydantic-settings, with a compatibility layer that keeps existing config files and env vars working. - Don’t: adopt Hydra. It’s built for composing and sweeping experiment configs, not for running a long-lived daemon. Its stable releases can’t run on OmegaConf 2.4 yet, and it has had a string of high-severity advisories this year.
Background¶
How configuration works today¶
config.pydefines 16 dataclasses, loaded withOmegaConf.structured(). The config fileconf/config.yamlis merged on top, the result is made read-only, and code reads it by attribute (cfg.mqtt.host).- Env vars are read with OmegaConf interpolation in dataclass defaults, e.g.
host: str = "${oc.env:MQTT_HOST,localhost}". Only 19 settings have an env var. Everything else needs the config file (the Home Assistant area, for example). - Typing workarounds. An interpolation string is a
strdefault for anint,boolor enum field, so 6 fields carrytype: ignorefor mypy andty: ignorefor ty. -
Auto-generated config. If there’s no config file and the env vars don’t fully describe the config, the app writes one with
OmegaConf.to_yaml(). That file contains the unresolved interpolations, for example:So users’ config files contain OmegaConf syntax that we wrote for them. The documentation and
examples/config.yaml.minimalalso tell users to write${oc.env:...}themselves. Any replacement must keep reading these files.These files also fix
node.nameto the hostname at the time the file was generated. That’s a side issue, but worth fixing in any migration. - Placeholder detection.???(MISSING) is the default formqtt.user. If it stays missing, the app logs that the config “has place holders” and stops cleanly. - Other uses: -common_packages.yamlis loaded into theCommonPackagesstructured config indocker_enrich.py. - Theu2m-clidebug tool parseskey=valuearguments withOmegaConf.from_cli(). Itsmqtt=checkcommand reports each setting and its env var by reading the unresolved interpolations.
| Where | OmegaConf references |
|---|---|
src/updates2mqtt/cli.py | 25 |
src/updates2mqtt/config.py | 6 |
src/updates2mqtt/integrations/docker_enrich.py | 6 |
tests/ (7 files, mostly OmegaConf.structured(MqttConfig) in test_mqtt.py) | 182 |
Goals¶
These come from the roadmap.
- G1: Use a live, popular config library with timely security fixes and good documentation.
- G2: Make configuration consistent across env vars and files, so every setting can be set by env var, automatically.
- G3: Make config problems easy to debug and report clearly.
- G4 (value add): Support CLI args consistent with the env vars and config file.
- G5 (value add): Integrate with Vault or similar secret storage.
- G6: Keep full backward compatibility with existing env vars and config files. Auto-migration is optional, and can’t assume write access.
- G7: Work well in Docker/Podman, as a PyPI library, and as a systemd/init.d daemon.
Option 1: Upgrade to OmegaConf 2.4¶
State of OmegaConf¶
OmegaConf has come back to life. After the original author left Meta at the end of 2021, development slowed; 2.3.0 (December 2022) was the last release for three and a half years. In 2026 the author returned as maintainer. Hydra moved from Meta to the independent hydra-ecosystem GitHub organization (announcement, 2026-08-13), and OmegaConf followed.
| Release | Date |
|---|---|
| 2.3.0 | 2022-12-08 |
| 2.3.1 | 2026-06-11 (packaging fix only) |
| 2.4.0 | 2026-10-09 |
| 2.4.1rc2 | 2026-10-28 (pre-release) |
- About 38M downloads a month.
- 29 open issues and 2 open PRs.
- No published security advisories.
- PyPI lists three maintainers, but the 2026 activity is mainly one person.
- PyPI flags a “change of provenance” on 2.4.0, because the publishing workflow changed.
What 2.4 changes¶
From the 2.4 upgrade guide:
- Missing values (this is what broke us):
- Resolving an interpolation to
???now raisesInterpolationToMissingValueError. In 2.3 it silently left the value missing. - A resolver that returns plain
???is now missing on access.
- Resolving an interpolation to
- Typed interpolations are validated on access.
- Implicit conversions on assignment emit a
FutureWarning. DictConfig/ListConfigare unhashable, tuples becomeTupleConfig, andOmegaConf.create(None)returnsNone.- Backslash escaping in
from_cli/dotlist keys (2.4.0) and new path escaping (2.4.1rc2). These affectu2m-clionly. - Good things:
- Enums can be assigned by member name or value.
- Better
Literal/union typing. - The antlr runtime is vendored, ending its version conflicts.
- YAML alias-expansion limits against crafted input.
- Python 3.10 to 3.14 is supported.
What we found by testing¶
I ran the full test suite against OmegaConf 2.4.0 in a throwaway environment, outside the repo.
- Unchanged code:
- 46 tests failed and 18 errored.
- Every one was the same
InterpolationToMissingValueErroronmqtt.user, from the${oc.env:MQTT_USER,???}default. - No
FutureWarningor other change showed up, even with-Werror.
- With the
mqtt.userdefault changed to${oc.env:MQTT_USER,''}:- Everything passed on both 2.3.1 and 2.4.0 except one test.
- That test,
test_check_mqtt_missing_user_skips_connect, needsu2m-cli mqtt=checkto treat an empty user as missing as well as???. - The app’s own check already treats
""and???the same.
- Existing auto-generated config files (containing
user: ${oc.env:MQTT_USER,???}):
| OmegaConf | MQTT_USER set | MQTT_USER not set |
|---|---|---|
| 2.3.1 | Loads | Logs “place holders”, app stops cleanly |
| 2.4.0 | Loads | Uncaught InterpolationToMissingValueError, so startup crashes with a traceback |
InterpolationToMissingValueError is an InterpolationResolutionError. load_app_config() only catches MissingMandatoryValue and ValidationError, so this needs one more except that maps it to the same placeholder outcome.
Work and impact¶
- Work: about half a day.
- Change the
mqtt.userdefault. - Catch
InterpolationResolutionErrorinload_app_config()(and the CLI equivalent) as “placeholder”. - Treat
""as missing inu2m-cli. - Widen the pin to
>=2.4.0,<2.5. - Add a regression test for a legacy auto-generated file with
MQTT_USERunset.
- Change the
- User impact: none if the
exceptis added. Without it, the only users affected are those with an old auto-generated file and noMQTT_USER. Their install isn’t configured yet anyway, but they’d get a crash instead of a clear message.
Against the goals¶
| Goal | Fit |
|---|---|
| G1 | ⚠️ Revived and releasing again, but with a short recent track record and mainly one maintainer |
| G2 | ❌ Env vars still have to be wired up field by field with oc.env. Nothing generates them |
| G3 | ⚠️ Errors give the full key path. Interpolation errors are hard for users to read |
| G4 | ❌ from_cli() takes dotlists only, with no help or validation of names |
| G5 | ❌ None. A custom resolver could fetch secrets |
| G6 | ✅ Same file format |
| G7 | ✅ |
Verdict: worth doing now as a holding move, because it gets us off a pin to an old release line cheaply. It doesn’t meet G2, G4 or G5.
Option 2: Become a full Hydra app¶
What Hydra adds¶
Hydra is built on OmegaConf. It adds:
- composing config from config groups with a
defaultslist - command-line overrides with tab completion
--multirunsweeps- launcher and sweeper plugins (Ray, Submitit, Optuna, etc.)
- per-run output directories
- configuration of Python logging
hydra.utils.instantiate()to build objects from config- an
@hydra.mainentry point, or thecomposeAPI for apps that don’t use@hydra.main
State of Hydra¶
- Latest stable is
hydra-core1.3.8 (2026-10-25). 1.4.0 is at rc3 (2026-10-25). - Mostly one maintainer, with about 17M downloads a month.
- Version coupling:
- hydra-core 1.3.8 resolves only to OmegaConf 2.3.1.
- 1.4.0rc3 requires OmegaConf 2.4.1rc2, which is a pre-release.
- So no stable Hydra release runs on OmegaConf 2.4 today. Adopting Hydra would mean either staying on OmegaConf 2.3 or depending on pre-releases.
- Security: seven advisories published between July and October 2026, six of them high severity. Most concern
instantiate(), its target blocklist, unsafe YAML loading, and callables resolved from logging config. Of the 1.x line, only 1.4 claims Python 3.13/3.14 support.
Fit for Updates2MQTT¶
Hydra’s features target ML and research workflows: composing variants, sweeping parameters, launching jobs, keeping per-run outputs. A home-automation daemon in a container needs none of these.
What it costs:
@hydra.maintakes over the process. It owns argument parsing, configures logging (we use structlog), and by default creates per-run output directories. Each of these can be switched off, but that means fighting the framework.- The
composeAPI avoids that, but then we’d get only config-group composition, which we have no use for. - Env vars still come only from OmegaConf’s
oc.env, so G2 is no better than option 1. - It adds a framework-sized dependency, with a recent advisory history, to the software that pulls and restarts containers on users’ hosts.
| Goal | Fit |
|---|---|
| G1 | ⚠️ Active again, but mainly one maintainer and recent high-severity advisories |
| G2 | ❌ Same as OmegaConf |
| G3 | ⚠️ Same as OmegaConf |
| G4 | ✅ Strong CLI overrides, but in Hydra’s key=value grammar, not flags matching the env vars |
| G5 | ❌ |
| G6 | ✅ Reads the same files |
| G7 | ⚠️ Working-directory, logging and output-directory defaults all need disabling for a daemon |
Verdict: not recommended.
Option 3: Move to Pydantic Settings¶
State of Pydantic Settings¶
pydantic-settings2.16.3 was released 2026-11-13, its tenth release of 2026. It’s maintained by the Pydantic organization, with Trusted Publishing and provenance.- About 488M downloads a month, which is 13× OmegaConf.
- We already use it in
anpr2mqtt. - It requires Python ≥3.11, and we require ≥3.13.
How it meets the goals¶
- Env vars for everything (G2). With
env_prefix="U2M_"andenv_nested_delimiter="__", every field gets an env var automatically, e.g.U2M_MQTT__HOST,U2M_HOMEASSISTANT__AREAorU2M_DOCKER__REGISTRY__API. - Legacy env vars (G6).
validation_alias=AliasChoices("U2M_MQTT__HOST", "MQTT_HOST")accepts either name, and the first one found wins. Every one of today’s 19 env var names can stay as an alias. - YAML file.
YamlConfigSettingsSource. Source priority is set withsettings_customise_sources, e.g. CLI, then env, then secrets, then file, then defaults. - Validation (G3).
- Errors from Pydantic’s
ValidationErrorgive each field’s path and the reason. - Setting
PYDANTIC_SETTINGS_DEBUG=1logs what each source supplied, which answers “where did this value come from?”
- Errors from Pydantic’s
- CLI (G4).
cli_parse_args/CliAppgenerate flags from the same model, e.g.--mqtt.host.cli_show_env_varslists the matching env var in--help. - Secrets (G5).
secrets_dircovers Docker/Podman secrets, one file per field.- Built-in sources cover AWS Secrets Manager and SSM, Azure Key Vault, and GCP Secret Manager.
- HashiCorp Vault isn’t built in, but a custom source is a small class.
- Typing. Real types with no interpolation strings, so the six
type: ignore/ty: ignoreworkarounds go away.
Gotchas to design around¶
- Partial updates of nested models. By default, setting one nested field from an env var replaces that nested model’s defaults wholesale. We need
nested_model_default_partial_update=True. - Unknown env vars. Env vars that match no field are silently ignored, even with
extra="forbid". Keep and extend the unrecognised-env-var warning added in 1.17.6: derive the knownU2M_*names from the model instead of keeping a hand-maintained list. - Case. Env var matching is case-insensitive by default. Set
case_sensitive=Truefor predictable behaviour. - Lists. List and dict values from env vars are parsed as JSON. Selectors such as
image_ref_select.excludeneed either JSON or aNoDecodeannotation with a comma-split validator. - Aliases on nested models. Aliases on nested models need
populate_by_name(orvalidate_by_name), so the field name is accepted alongside the alias. - Stricter JSON matching. Pydantic 2.14.2 treats keys that match only an unused alias as extra data in JSON validation. This needs a test.
Backward compatibility layer¶
This is the part that makes the migration safe for existing installs.
- Legacy YAML interpolation.
- Add a custom YAML settings source. Before validating, it resolves the small OmegaConf subset that we generate and document:
${oc.env:VAR}${oc.env:VAR,default}with quoted or bare defaults, includingnulland''${oc.decode:...}wrapped around them- a bare
???
- That’s a regex-sized resolver of about 50 lines, not a dependency on OmegaConf.
- Anything else in
${...}form is a clear validation error that names the key. - Optionally log a one-time hint that the file can now use plain values.
- Add a custom YAML settings source. Before validating, it resolves the small OmegaConf subset that we generate and document:
- Legacy env var names.
- Every existing name, e.g.
MQTT_HOST,MQTT_PASSorU2M_LOG_LEVEL, becomes anAliasChoicesentry next to the newU2M_SECTION__KEYname. - Precedence: when both are set, the new name wins.
- Add a test with one row per legacy name.
- Every existing name, e.g.
- Placeholder detection. Keep it: an
mqtt.userthat is missing or empty gives the same “place holders” outcome. - Auto-generated config.
- Stop writing resolved defaults and the hostname into the generated file.
- Write a commented template instead, or a file holding only what’s needed.
- Existing generated files keep working through point 1.
- No rewriting of user files. We can’t assume write access (G6). A
u2m-cli config=migratecommand can print a cleaned-up file for users to copy if they want one.
Work¶
| Area | Estimate | Notes |
|---|---|---|
config.py dataclasses become BaseModel, Config becomes BaseSettings | 1 day | Attribute access (cfg.mqtt.host) stays the same, so most of src/ is untouched |
| Legacy YAML resolver source and tests | 1 day | Includes tests against every example and doc snippet, and a real auto-generated 1.17 file |
| Legacy env var aliases and tests | 0.5 day | |
common_packages.yaml loading with a Pydantic model | 0.5 day | No env vars needed. Errors must still name the package |
u2m-cli: replace from_cli() and the settings report | 1 day | The report can come from the model and source data instead of parsing interpolation strings |
Tests: replace OmegaConf.structured(...) fixtures | 1 day | Mostly mechanical, about 180 references, concentrated in test_mqtt.py |
| Docs: config pages, env var tables, examples, FAQ | 1 day | Env var tables can be generated from the model |
| Total | about 6 days |
New dependencies: pydantic, pydantic-core (a compiled wheel), pydantic-settings[yaml], annotated-types and typing-inspection. omegaconf and its vendored antlr runtime are removed. pydantic-core has wheels for the Docker image’s architectures, but check the image-size change.
User impact: none expected if the compatibility layer lands. New installs get documented U2M_SECTION__KEY env vars for every setting. The old names stay supported for at least two minor releases. After that, whether to deprecate them (with a startup warning first) is a separate decision.
| Goal | Fit |
|---|---|
| G1 | ✅ Highest adoption and release cadence of any option, run by an organization rather than one person |
| G2 | ✅ Automatic env vars for every field |
| G3 | ✅ Field-path errors and source debugging. Unknown env vars need our own warning |
| G4 | ✅ Generated CLI with env var names shown in help |
| G5 | ⚠️ Docker secrets and the main cloud secret stores are built in. Vault needs a small custom source |
| G6 | ✅ With the compatibility layer above |
| G7 | ✅ |
Option 4: Other established libraries¶
| Library | Latest | Health | Notes |
|---|---|---|---|
| Dynaconf | 3.3.5 (2026-08-05) | ⚠️ Releasing again in 2026 after a two-year gap, about 5M downloads a month | Strongest built-in Vault and Redis support. Env vars with __ nesting. But validation is lazy (on first access), the settings object is global and mutable, typed models are only a tech preview, and there’s no CLI matching the env vars |
| typed-settings | 26.1.0 (2026-11-05) | ⚠️ Active, but one maintainer and a much smaller user base | Good design: works with attrs, dataclasses or Pydantic, and generates CLIs with argparse or Click. But 27.0 will break internal and advanced APIs, the default _ delimiter collides with field names, and secrets support is aimed at 1Password, not Vault |
| jsonargparse | Not confirmed (docs show 5.0.0rc2) | ✅ Active, and the base of PyTorch Lightning’s CLI | Clearest precedence from defaults through file and env vars to CLI. But it’s CLI-first, env vars must be switched on explicitly, and its interpolation is powered by OmegaConf |
| confz | 2.1.0 (2025-02-07) | ❌ No release for 21 months | Pydantic-based. Nested env vars use ., which shells and Compose handle badly |
| msgspec, cattrs | 0.22.0, 26.2.1 | ✅ | Serialization and validation libraries, not config loaders. Could underpin a hand-rolled loader, but that means owning env var and CLI handling ourselves |
| environ-config | Not assessed | - | Its PyPI page couldn’t be fetched |
None of these beats Pydantic Settings on G1 to G4 together, and the one with clearly better Vault support (Dynaconf) is weaker on typing and validation. Dynaconf is only worth reconsidering if Vault becomes a hard requirement.
Recommendation and plan¶
- Phase 0, the next point release: upgrade to OmegaConf 2.4 as described in option 1. This removes the pin to a stale release line and has no user impact.
- Phase 1, the next minor release: migrate to
pydantic-settingswith the compatibility layer.- Document the
U2M_SECTION__KEYenv vars. - Generate the env var reference from the model.
- Fold in deferred settings, e.g.
U2M_HOMEASSISTANT__AREA.
- Document the
- Phase 2, later:
- Generated CLI flags for
u2m-cli. - A Vault settings source if users ask for one.
- Decide whether to deprecate the legacy env var names.
- Generated CLI flags for
- Not planned: Hydra.
Open questions¶
- Prefix for new env vars:
U2M_(short, already used by four settings) orUPDATES2MQTT_(clearer, but used by out-of-date guides with different meanings)? This RFC assumesU2M_. - Should the legacy
${oc.env:...}resolver stay permanently, or warn and eventually go? Users copied this syntax from our own docs, so it may be kindest to support it permanently. - Is a TOML config file worth offering next to YAML? Pydantic Settings supports both, but YAML is what Home Assistant users expect.
mqtt.portandlog.leveluseoc.decodetoday. Check that nothing else in user files relies onoc.decodeturning strings into lists or dicts.
References¶
- OmegaConf 2.4.0 release notes, 2.4 upgrade guide, missing values, built-in resolvers, PyPI
- Hydra introduction, independent stewardship, security advisories, PyPI
- Pydantic Settings documentation, PyPI
- Dynaconf, typed-settings, jsonargparse, confz
- Download figures from pypistats.org, November 2026