arcline-project/os-build
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
87fc370541c7a2ef7a817842c89889330d0d95b3
os-build/docs/observability.md
Blake Ridgway 87fc370541 docs: add architecture, hardening, and building documentation 
Document the design and how to operate it: architecture, building
(including toolchain-free/minimal builds), hardening guide, editions,
observability, and the toolchain. README ties it together.
2026-08-21 13:15:43 -05:00

2.2 KiB
Raw Blame History

Observability

The server edition ships a pre-configured, local-only observability stack — the same setup Arcline IT runs in production. It never talks to any external service.

 node_exporter ──► Prometheus ──► Grafana (localhost:3000)
                     │  ▲
   journald ─────► promtail ──► Loki (localhost:3100)
Component Port Role Config source
node_exporter 9100 host metrics Debian package (enabled by default)
Prometheus 9090 metric storage + alerting overlays/server/etc/prometheus/
Grafana 3000 dashboards (auto-provisioned) overlays/server/etc/grafana/
Loki 3100 log storage (14d retention) overlays/server/etc/loki/
promtail 9080 ships journald + /var/log → Loki overlays/server/etc/promtail/

What is pre-configured

  • Prometheus scrapes itself + node_exporter and loads alerting rules (arcline.rules.yml): NodeDown, high load, disk > 85% / critical > 95%, service crash-looping.
  • Grafana auto-provisions the Prometheus/Loki datasources and a starting Node Overview dashboard (node-overview.json) on first start.
  • Loki is single-node filesystem storage with a 14-day retention, matching the journald retention.
  • promtail ships the systemd journal and /var/log to the local Loki.

Packaging note

Prometheus + node_exporter + alertmanager are in Debian main and install with the edition packages. Grafana and Loki are not:

  • Grafana is available via the official apt.grafana.com repo — add it by setting ARCLINE_EXTRA_REPOS=1 during the build (configure-system.sh adds the repo + key).
  • Loki ships as a static binary tarball; the packaging (a .deb in toolchain/) is follow-up work.

Until then, the Grafana/Loki configs ship dormant in the image, ready for when the binaries are installed — the overlays/server configs are the contract.

Access

  • Grafana UI: http://<host>:3000 (default admin credentials are set at first boot by the installer — never left at Grafana's defaults).
  • Prometheus: http://<host>:9090.

Both are not exposed by the default-deny firewall; open ports deliberately or access via ssh tunnel.