In my previous article I described how I deployed two monitoring systems for a single fleet — Zabbix with a custom dashboard and Prometheus + Grafana. Both work, both have public demos. But one uncomfortable question remained: what if the container running the monitoring dies? Restoring it by hand means hours of work and trying to remember «how did I set this up back then?» There is only one proper engineering answer: infrastructure must be described as code.
Over two days I moved the whole fleet to Ansible. Not «wrote a couple of playbooks», but brought it to a state where any service is rebuilt from the repository in minutes, changes are checked in CI, and nothing breaks in production. Here is how it works and the pitfalls I hit.
A heterogeneous Proxmox environment: several LXC containers (monitoring, CI node) and virtual machines, plus a cloud server. Everything is managed by a dedicated control node — a container running Ansible (installed via pipx to get a fresh version in an isolated environment rather than whatever ships in the distribution repositories).
Hosts are described in the inventory, grouped by purpose: the Prometheus stack, hypervisors, VMs, node_exporter hosts, the CI node. Ansible reaches them over an SSH key — some directly over the LAN, some through a VPN tunnel.
Each role owns one component and reproduces its state exactly:
A role that changes something on every run is a bad role. A correct role reports «changed: 0» on the second run. That means the state described in code has already been reached, and Ansible touches nothing unnecessarily — no service restarts, no config churn. All nine roles are idempotent: the second run makes zero changes on every host. I verified each one.
Nice words about «infrastructure as code» are worthless until you test them in practice. I ran a controlled experiment on the least critical host: I completely removed node_exporter — binary, systemd unit, user. The corresponding target in Prometheus immediately went down.
Then I ran the role. In about 8 seconds the service was back: user created, binary downloaded and installed, unit in place, service started, and the Prometheus target «up» again. That is real insurance — not theoretical, but proven live.
A subtle point that often turns IaC into a mess: two roles writing to the same file. My blackbox site monitoring needed to add its scrape job to the Prometheus config — but that config is already managed by the prometheus role, byte for byte.
The solution is not a hack but a proper decoupling. Prometheus can load scrape configurations from separate files (the scrape_config_files directive). I added a generic directive to the prometheus role: pick up everything from a dedicated directory. Now any future exporter drops its fragment there without touching the main config. The blackbox role owns its fragment entirely, the prometheus role owns its config. One file — one owner. No conflicts.
The same principle applies to Grafana dashboards: blackbox has its own dashboard provider, separate from the main one. The roles never step on each other.
Monitoring means live services with public demos. You cannot drop them. So every change followed one procedure:
Not a single demo flickered over the two days.
The repository with the roles lives in a self-hosted Gitea. Every push triggers CI (Gitea Actions): the ansible-lint linter in production mode — the strictest one. A red lint means the code does not pass. This keeps bad practices from accumulating and keeps the roles in shape.
A recent ansible-core removed one output parameter — the old playbook complained. I fixed it for the current version. The principle: check the behavior of the specific version rather than relying on memory.
The script accessed a service under one username, while the real system user turned out to be different. I caught it with a factual check before the production run — otherwise the backup would have silently produced an empty file. Lesson: an unverified backup is not a backup.
During the preview of a role that installs a package and immediately manages a service, Ansible complains: the service does not exist yet (the package is not installed in check mode). This is not an error — in a real run the package is installed first and the service appears. I understood the mechanics and did not «fix» a non-existent problem.
I will note separately what I deliberately did not move into Ansible: the production cloud server — the single entry point into the whole infrastructure. The risk of a mistake there is out of proportion to the benefit: a single typo in the network or SSH config, and I lose access to everything at once. Its data is already protected by a separate backup system. Automating for the sake of a checkmark is bad engineering. Maturity is weighing risk against benefit, not coding everything.
The entire fleet — provisioning, configuration, monitoring, alerting, SSH protection, backups — is now described as code. Nine idempotent roles, checked in CI and version-controlled in git. Any component is rebuilt from the repository in minutes rather than hours of manual work. And it is proven live, not just on paper.
Infrastructure as code is not about a trend for a tool. It is about sleeping soundly, knowing that if something goes down, I can bring it back with a single command.
Technologies: Ansible, ansible-lint, Gitea Actions (CI/CD), Prometheus, Grafana, Zabbix, blackbox_exporter, node_exporter, fail2ban, WireGuard, Proxmox.