docs: add architecture, hardening, and building documentation

Document the design and how to operate it: architecture, building (including toolchain-free/minimal builds), hardening guide, editions, observability, and the toolchain. README ties it together.
2026-08-21 13:15:43 -05:00
parent 33652064f9
commit 87fc370541
7 changed files with 474 additions and 0 deletions
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -0,0 +1,68 @@
+# Arcline OS — architecture
+
+This is the foundation ("the wires") for Arcline OS: a hardened, Debian-derived
+operating system for people who run infrastructure. The landing page describes
+the product; this repository is how it gets built.
+
+## Design goals
+
+1. **Secure by default** — hardened kernel, default-deny firewall, AppArmor,
+   locked-down ssh. You opt *in* to exposure, never out.
+2. **Zero telemetry** — no phone-home, no analytics, no cloud integration.
+3. **btrfs-native** — snapshots and boot-to-snapshot rollback are built in.
+4. **Production-ready from first boot** — observability pre-configured, tools
+   pre-installed, everything documented.
+5. **Auditable** — every script and config is plain text in this repo.
+
+## How a build flows
+
+```mermaid
+flowchart LR
+    A[editions/* metadata] --> B[debootstrap<br/>Debian bookworm]
+    B --> C[install edition<br/>packages.list]
+    C --> D[apply overlays<br/>base + edition]
+    D --> E[configure-system.sh<br/>in chroot]
+    E --> F[rootfs .tar.xz]
+    F --> G[grub-mkrescue + squashfs<br/>→ live ISO]
+    E -.toolchain .debs.-> C
+```
+
+Pipeline stages live in `scripts/`:
+
+| Stage | Script | What it does |
+|-------|--------|--------------|
+| bootstrap | `build-rootfs.sh` | debootstrap minbase, apt sources, package install |
+| overlay | `apply-overlays.sh` | copies `overlays/base` + `overlays/<edition>` into the rootfs |
+| configure | `configure-system.sh` | runs *in the chroot*: hostname, locale, kernel cmdline, services, live-boot, toolchain |
+| package | `build-iso.sh` | kernel + initramfs + squashfs → hybrid BIOS/UEFI ISO |
+| orchestrate | `build-edition.sh` / `Makefile` | wire the above to `make iso-<edition>` |
+
+## The four source trees
+
+| Path | Role |
+|------|------|
+| `editions/` | per-edition **manifests**: package lists, kernel cmdline, fstab, metadata |
+| `overlays/` | **files that land in the image**, organised as layered rootfs trees |
+| `scripts/` | the **build pipeline** (all plain bash, readable top to bottom) |
+| `btrfs/`, `toolchain/`, `observability/`, `tests/`, `ci/` | supporting subsystems |
+
+There is no hidden magic: the Makefile is a thin wrapper, `versions.mk` /
+`scripts/common.sh` hold the single source of truth for versions and paths.
+
+## Zero telemetry, enforced
+
+- No distro telemetry packages are installed (`ubuntu-report`,
+  `popularity-contest`, snapd are never in a package list).
+- apt automatic-update timers are **masked** in every edition's metadata.
+- The firewall's output chain never initiates calls on its own.
+- cloud-init is pointed only at the configured cloud datasource.
+- The smoke tests (`tests/smoke/verify-rootfs.sh`) fail the build if telemetry
+  artifacts are found.
+
+## Follow-up work (explicitly out of scope for "the wires")
+
+- Partitioning/installer that runs `btrfs/init.sh` on a target disk.
+- Cloud images (qcow2/raw) for the `cloud` edition — the ISO path is wired,
+  a disk-image path is the next step.
+- Grafana/Loki packaged as `.deb`s rather than fetched from upstream.
+- A signed (secure-boot) kernel for official releases.
--- a/docs/building.md
+++ b/docs/building.md
@@ -0,0 +1,92 @@
+# Building Arcline OS
+
+## Prerequisites
+
+A Debian-family host (or container) with root/sudo:
+
+```
+debootstrap        # rootfs bootstrap
+squashfs-tools     # ISO rootfs compression
+grub2-common       # grub-mkrescue (hybrid ISO)
+xorriso cpio       # ISO assembly
+bash make curl git # build orchestration
+go (≥ 1.21)        # only needed for `make toolchain`
+```
+
+Check or install them:
+
+```
+scripts/check-host-deps.sh            # report what's missing
+scripts/check-host-deps.sh --install  # apt-get install the missing bits
+```
+
+## Quickstart
+
+```
+make check          # validate the tree (fast, offline, safe)
+make rootfs-server  # build just the server rootfs
+make iso-server     # build a bootable server ISO
+make iso            # build all three editions
+make toolchain      # build the 11 Go tools into .deb
+make test           # run smoke tests against built rootfs(es)
+```
+
+## Outputs
+
+Everything lands under `build/`:
+
+| Path | Contents |
+|------|----------|
+| `build/rootfs/<edition>/` | extracted rootfs (from `make rootfs-*`) |
+| `build/artifacts/arcline-<edition>-<version>-<arch>.tar.xz` | archived rootfs |
+| `build/artifacts/arcline-<edition>-<version>-<arch>.iso` | bootable live ISO |
+| `build/debs/*.deb` | Arcline toolchain packages (from `make toolchain`) |
+
+Each artifact ships with a `.sha256` checksum file.
+
+## Reproducibility knobs
+
+Set these as environment variables or edit `versions.mk`:
+
+| Variable | Meaning | Default |
+|----------|---------|---------|
+| `VERSION` | release version string | `0.1.0` |
+| `DEBIAN_SUITE` | Debian base | `bookworm` |
+| `ARCH` | target architecture | `amd64` |
+| `DEBIAN_MIRROR` | mirror for the base | `http://deb.debian.org/debian` |
+| `ARCLINE_LIVE` | install live-boot into rootfs | unset (set by ISO build) |
+| `ARCLINE_LOCK_ROOT` | lock root account (`passwd -l root`) | `0` |
+| `ARCLINE_EXTRA_REPOS` | fetch grafana/loki upstream repos | `0` |
+| `ARCLINE_TOOLCHAIN` | toolchain in image builds | `auto` |
+
+## Building without the Arcline toolchain
+
+The 11 Go tools (see `docs/toolchain.md`) are **optional** in an image — a
+normal `make iso-server` with no `.deb`s in `build/debs/` already produces a
+minimal, toolchain-free image. This is controlled by `ARCLINE_TOOLCHAIN`:
+
+| Mode | Behaviour |
+|------|-----------|
+| `auto` (default) | install the tools if `build/debs/*.deb` exist, otherwise build without them (with a warning) |
+| `skip` | never install the tools, even if debs are present |
+| `require` | **fail** the build if no toolchain debs are available |
+
+Examples:
+
+```bash
+make iso-server                # auto: tools in if you ran `make toolchain`
+make iso-server-minimal        # skip: guaranteed toolchain-free image
+ARCLINE_TOOLCHAIN=skip make iso-server
+ARCLINE_TOOLCHAIN=require make iso-server   # release builds must ship the tools
+```
+
+Everything else — hardened base, firewall, btrfs, observability
+(Prometheus/node_exporter are Debian-main packages), zero-telemetry — is
+independent of the toolchain and always included.
+
+## Notes on the ISO
+
+The ISO boots a **live** system (via `live-boot`): the rootfs is compressed to
+a squashfs and mounted on boot, so you can try an edition before installing it
+to disk. The same rootfs can be installed with the btrfs layout via
+`btrfs/init.sh` (the installer is follow-up work — see `docs/architecture.md`).
--- a/docs/editions.md
+++ b/docs/editions.md
@@ -0,0 +1,44 @@
+# Editions
+
+Three flavours, one hardened base. Each is defined entirely by its manifest
+under `editions/<name>/`.
+
+| Edition | Codename | Purpose | Kernel | Notes |
+|---------|----------|---------|--------|-------|
+| `server` | bastion | production server | generic (`linux-image-amd64`) | full observability stack, containers |
+| `workstation` | forge | hardened daily driver | generic | KDE Plasma, dev toolchains |
+| `cloud` | nimbus | cloud images | cloud (`linux-image-cloud-amd64`) | cloud-init, guest agents |
+
+## What an edition manifest contains
+
+```
+editions/<name>/
+├── metadata.yaml    # codename, summary, image type, services to enable/mask
+├── packages.list    # Debian packages (comments allowed)
+├── kernel.cmdline   # boot parameters for this edition
+└── fstab            # /etc/fstab template (btrfs subvolumes, UUID placeholder)
+```
+
+Edition-specific configs that land in the image live in `overlays/<name>/` and
+are layered **after** `overlays/base/`, so they win conflicts.
+
+## server — bastion
+
+The flagship. Hardened base + Docker/Podman + Prometheus/Grafana/Loki
+pre-configured (`overlays/server/etc/prometheus|grafana|loki|promtail`). Ships
+the full Arcline toolchain. Serial console for headless boxes.
+
+## workstation — forge
+
+Same hardening baseline, relaxed only where a developer needs it (`perf`,
+rootless containers). Curated KDE Plasma (not the full task meta-package),
+dev toolchains (Go, Rust, Python, Node, clang/LLVM, CMake/Ninja), containers,
+and privacy-oriented defaults.
+
+## cloud — nimbus
+
+Minimal footprint: cloud kernel, cloud-init (NoCloud/ConfigDrive/EC2/GCE/
+Azure), guest agents (qemu/vmware), NVMe + iSCSI + multipath tooling.
+`net.ifnames=0` for predictable, provider-friendly interface naming. The
+disk-image (qcow2) output path is the next milestone — the manifest is ready,
+the ISO path is wired today.
--- a/docs/hardening.md
+++ b/docs/hardening.md
@@ -0,0 +1,83 @@
+# Hardening guide
+
+Every edition ships the same hardening baseline (the `overlays/base` layer).
+This page explains what each piece does and how to tune it. Everything is
+documented *in* the config files too — read them, they are the source of truth.
+
+## Kernel
+
+- **Boot cmdline** (`editions/*/kernel.cmdline`) — applied to both the live ISO
+  and (via `configure-system.sh` → `/etc/default/grub`) any installed system:
+  - `init_on_alloc=1 init_on_free=1` — zero freshly allocated and freed memory
+  - `slab_nomerge` — prevent slab object merging
+  - `page_poison=1` — fill freed pages to catch use-after-free
+  - `pti=on` — kernel page-table isolation
+  - `spectre_v2=on spec_store_bypass=on tsx=off` — mitigations forced on
+  - `lockdown=integrity` — kernel refuses unsigned in-memory modification
+  - `oops=panic panic=-1` — halt (and stay down) on kernel oops
+- **sysctl** (`overlays/base/etc/sysctl.d/10-arcline-hardening.conf`):
+  - memory: sane dirty ratios, no overcommit
+  - network: strict rp_filter, no ICMP redirects, syncookies, TCP hardening
+  - visibility: `dmesg_restrict=1`, `kptr_restrict=2`, `perf_event_paranoid=3`,
+    `ptrace_scope=1`, `unprivileged_bpf_disabled=1`
+  - filesystem: protected hardlinks/symlinks/fifos, `suid_dumpable=0`
+- **Module blacklist** (`overlays/base/etc/modprobe.d/arcline-hardening.conf`):
+  - exotic network protocols (`sctp`, `dccp`, `rds`, `tipc`) and legacy
+    filesystems (`cramfs`, `hfs`, `hfsplus`, `jffs2`, `udf`, …).
+
+> The **workstation** edition relaxes a few of these for developer ergonomics
+> (`perf`, tracing, rootless containers) via
+> `overlays/workstation/etc/sysctl.d/90-arcline-desktop.conf`.
+
+## Firewall — nftables, default-deny
+
+`overlays/base/etc/nftables.conf` ships a **drop-by-default** policy:
+
+- allow: loopback, established/related, ICMP (for PMTU discovery)
+- allow: `tcp/22` (ssh — key auth only)
+- everything else inbound: **dropped**
+- forward chain: **dropped** (container bridges manage their own rules)
+
+Open a port by editing the file and `systemctl reload nftables`, e.g.:
+
+```
+sudo nft add rule inet filter input tcp dport 443 accept
+```
+
+You opt in to exposure; nothing is open unless you say so.
+
+## ssh
+
+`overlays/base/etc/ssh/sshd_config.d/10-arcline-hardening.conf`:
+
+- `PermitRootLogin prohibit-password` — root only via key
+- `PasswordAuthentication no` — keys only (this is the secure default; change
+  deliberately)
+- `MaxAuthTries 3`, `LoginGraceTime 30`, `ClientAliveInterval 300`
+- no `X11Forwarding`, `UseDNS no`
+
+## Logging
+
+`journald.conf.d/10-arcline.conf` — persistent, bounded (500M), compressed,
+14-day retention. Logs live on the `@log` subvolume so they survive rollbacks.
+
+## AppArmor
+
+Arcline uses Debian's AppArmor profiles (loaded at boot) plus the kernel
+hardening above. Extra profiles can be dropped into
+`/etc/apparmor.d/local/` — the enforcement is on by default.
+
+## systemd hardening
+
+`system.conf.d/10-arcline.conf` — no core dumps on disk, sane default limits
+(`NOFILE=65535`, `NPROC=4096`).
+
+## Verifying a build
+
+```
+tests/smoke/verify-rootfs.sh build/rootfs/server
+```
+
+The smoke test asserts the guarantees above (kptr_restrict, policy drop, ssh
+keys-only, no telemetry, no snapd, btrfs tooling present). The build fails if
+any are missing.
--- a/docs/observability.md
+++ b/docs/observability.md
@@ -0,0 +1,53 @@
+# Observability
+
+The server edition ships a pre-configured, **local-only** observability stack —
+the same setup Arcline IT runs in production. It never talks to any external
+service.
+
+```
+ node_exporter ──► Prometheus ──► Grafana (localhost:3000)
+                     │  ▲
+   journald ─────► promtail ──► Loki (localhost:3100)
+```
+
+| Component | Port | Role | Config source |
+|-----------|------|------|---------------|
+| node_exporter | `9100` | host metrics | Debian package (enabled by default) |
+| Prometheus | `9090` | metric storage + alerting | `overlays/server/etc/prometheus/` |
+| Grafana | `3000` | dashboards (auto-provisioned) | `overlays/server/etc/grafana/` |
+| Loki | `3100` | log storage (14d retention) | `overlays/server/etc/loki/` |
+| promtail | `9080` | ships journald + /var/log → Loki | `overlays/server/etc/promtail/` |
+
+## What is pre-configured
+
+- **Prometheus** scrapes itself + `node_exporter` and loads alerting rules
+  (`arcline.rules.yml`): NodeDown, high load, disk > 85% / critical > 95%,
+  service crash-looping.
+- **Grafana** auto-provisions the Prometheus/Loki datasources and a starting
+  Node Overview dashboard (`node-overview.json`) on first start.
+- **Loki** is single-node filesystem storage with a 14-day retention, matching
+  the journald retention.
+- **promtail** ships the systemd journal and `/var/log` to the *local* Loki.
+
+## Packaging note
+
+Prometheus + node_exporter + alertmanager are in Debian main and install with
+the edition packages. **Grafana** and **Loki** are not:
+
+- Grafana is available via the official `apt.grafana.com` repo — add it by
+  setting `ARCLINE_EXTRA_REPOS=1` during the build
+  (`configure-system.sh` adds the repo + key).
+- Loki ships as a static binary tarball; the packaging (a `.deb` in
+  `toolchain/`) is follow-up work.
+
+Until then, the Grafana/Loki configs ship dormant in the image, ready for when
+the binaries are installed — the `overlays/server` configs are the contract.
+
+## Access
+
+- Grafana UI: `http://<host>:3000` (default admin credentials are set at
+  first boot by the installer — never left at Grafana's defaults).
+- Prometheus: `http://<host>:9090`.
+
+Both are **not** exposed by the default-deny firewall; open ports deliberately
+or access via ssh tunnel.
--- a/docs/toolchain.md
+++ b/docs/toolchain.md
@@ -0,0 +1,49 @@
+# The Arcline toolchain
+
+Eleven Go tools, built in production at Arcline IT, shipped with Arcline OS.
+The landing page describes them; `toolchain/` packages them.
+
+## The 11 tools
+
+| Tool | What it does |
+|------|--------------|
+| `arcline-uptime` | HTTP / TCP / TLS / DNS monitoring, Prometheus metrics export |
+| `arcline-check` | CDN + transparency auditor (detects data leaks) |
+| `arcline-audit` | full site health scanner — SSL, headers, performance, a11y |
+| `arcline-dns` | DNS propagation checker across resolvers |
+| `arcline-vault` | encrypted secrets store — REST API + CLI |
+| `arcline-email` | self-hosted SMTP/IMAP — OpenSMTPD + Dovecot + Rspamd |
+| `arcline-migrate` | cPanel / Plesk → Arcline migration |
+| `arcline-billing` | Stripe subscription management |
+| `arcline-portal` | customer dashboard — SSL monitoring + ticketing |
+| `arcline-website` | Go HTTP server powering arcline.it |
+| `arcline-status` | static status page generator |
+
+## How they get into the image
+
+```mermaid
+flowchart LR
+    A[tools.list] --> B[build-tools.sh]
+    B --> C[.deb per tool]
+    C --> D[build-rootfs.sh<br/>drops debs into chroot]
+    D --> E[configure-system.sh<br/>dpkg -i *.deb]
+```
+
+1. `make toolchain` (or `toolchain/build-tools.sh`) clones each repo from
+   `git.arcline.it` (or reuses checkouts in `ARCLINE_TOOLS_DIR`), builds a
+   release binary (version stamped via `-ldflags`), and packages a `.deb`
+   from `toolchain/debian/control.tmpl` into `build/debs/`.
+2. `build-rootfs.sh` copies any built debs into the chroot and
+   `configure-system.sh` installs them.
+
+A tool that can't be fetched or built is **skipped with a warning** — the OS
+image never breaks because a repo is unreachable.
+
+## Notes
+
+- Repo list lives in `toolchain/tools.list` (`name|repo|description`).
+- The tools' actual source lives on `git.arcline.it`; this repo is the
+  packaging harness. Each tool gets a proper `Debian` control file with a
+  generated description.
+- The ticket/portal/website tools are the *products*; the others are
+  infrastructure utilities. All are GPL and auditable.