From 87fc370541c7a2ef7a817842c89889330d0d95b3 Mon Sep 17 00:00:00 2001 From: Blake Ridgway Date: Fri, 21 Aug 2026 13:15:43 -0500 Subject: [PATCH] docs: add architecture, hardening, and building documentation Document the design and how to operate it: architecture, building (including toolchain-free/minimal builds), hardening guide, editions, observability, and the toolchain. README ties it together. --- README.md | 85 +++++++++++++++++++++++++++++++++++++++ docs/architecture.md | 68 ++++++++++++++++++++++++++++++++ docs/building.md | 92 +++++++++++++++++++++++++++++++++++++++++++ docs/editions.md | 44 +++++++++++++++++++++ docs/hardening.md | 83 ++++++++++++++++++++++++++++++++++++++ docs/observability.md | 53 +++++++++++++++++++++++++ docs/toolchain.md | 49 +++++++++++++++++++++++ 7 files changed, 474 insertions(+) create mode 100644 README.md create mode 100644 docs/architecture.md create mode 100644 docs/building.md create mode 100644 docs/editions.md create mode 100644 docs/hardening.md create mode 100644 docs/observability.md create mode 100644 docs/toolchain.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..aad69d0 --- /dev/null +++ b/README.md @@ -0,0 +1,85 @@ +# Arcline OS — build system + +> The Linux OS for people who run infrastructure. Hardened Debian base. +> Pre-configured monitoring, auditing, and security tooling. Zero telemetry. + +This repository is the build system ("the wires") behind +[Arcline OS](https://arcline.it). It turns a Debian base and a set of plain-text +manifests into hardened, bootable operating system images for three editions: +**Server**, **Workstation**, and **Cloud**. + +## What you get + +- A transparent, script-based build pipeline (no magic, everything auditable) +- `debootstrap` → package install → overlay → in-chroot configure → live ISO +- **Secure by default**: hardened kernel cmdline + sysctl, default-deny + nftables, key-only ssh, AppArmor, no core dumps +- **btrfs-native**: subvolume layout, scheduled read-only snapshots, and + boot-to-snapshot rollback tooling +- **Zero telemetry**: enforced by package selection, masked apt timers, and + smoke tests that fail the build if telemetry is found +- **Pre-configured observability** (server): Prometheus + node_exporter + + Grafana (auto-provisioned) + Loki + promtail, local-only +- **The Arcline toolchain**: a harness that packages all 11 Go tools into + `.deb`s and installs them into the image +- Smoke tests, GitLab CI, and docs that explain every decision + +## Quickstart + +```bash +# on a Debian-family host with root/sudo +make deps # install host build dependencies +make check # validate the tree (fast, offline) +make iso-server # build a bootable server ISO +make iso # build all three editions +make toolchain # build the 11 Go tools into .deb +make iso-server-minimal # server ISO WITHOUT the Arcline toolchain +make test # run smoke tests against built rootfs(es) +``` + +Artifacts land in `build/artifacts/` with `.sha256` checksums. + +The Arcline tools are **optional** in an image (`ARCLINE_TOOLCHAIN=auto|skip| +require`; see [building](docs/building.md#building-without-the-arcline-toolchain)). + +## Layout + +``` +os-build/ +├── Makefile # thin orchestration (make iso-) +├── versions.mk # single source of truth for versions/paths +├── editions/ # per-edition manifests (packages, cmdline, fstab, metadata) +├── overlays/ # files that land in the image (base + per-edition layers) +├── scripts/ # the build pipeline (all plain bash) +├── btrfs/ # subvolume layout, snapshots, rollback +├── toolchain/ # packaging for the 11 Go tools +├── observability/ # Prometheus/Grafana/Loki configs (docs + overlays/server) +├── tests/ # tree validation + rootfs smoke tests +├── ci/ # GitLab CI pipeline +└── docs/ # architecture, hardening, building, editions, ... +``` + +## Documentation + +| Doc | Contents | +|-----|----------| +| [architecture](docs/architecture.md) | the design and how a build flows | +| [building](docs/building.md) | prerequisites, quickstart, outputs, knobs | +| [hardening](docs/hardening.md) | every hardening decision, and how to tune it | +| [editions](docs/editions.md) | server / workstation / cloud manifests | +| [observability](docs/observability.md) | the pre-configured monitoring stack | +| [toolchain](docs/toolchain.md) | the 11 Go tools and their packaging | +| [btrfs](btrfs/README.md) | subvolume layout + snapshots + rollback | + +## Status + +This is the **foundation**: the build pipeline, edition manifests, hardening +baseline, btrfs tooling, observability configs, tests, and CI are in place and +runnable. The next milestones (installer that writes the btrfs layout to disk, +cloud disk images, signed releases) are listed in +[docs/architecture.md](docs/architecture.md#follow-up-work). + +## License + +GPL-3.0 — see [LICENSE](LICENSE). Sponsored by Arcline IT LLC. +No telemetry. No tracking. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..a2317b6 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,68 @@ +# Arcline OS — architecture + +This is the foundation ("the wires") for Arcline OS: a hardened, Debian-derived +operating system for people who run infrastructure. The landing page describes +the product; this repository is how it gets built. + +## Design goals + +1. **Secure by default** — hardened kernel, default-deny firewall, AppArmor, + locked-down ssh. You opt *in* to exposure, never out. +2. **Zero telemetry** — no phone-home, no analytics, no cloud integration. +3. **btrfs-native** — snapshots and boot-to-snapshot rollback are built in. +4. **Production-ready from first boot** — observability pre-configured, tools + pre-installed, everything documented. +5. **Auditable** — every script and config is plain text in this repo. + +## How a build flows + +```mermaid +flowchart LR + A[editions/* metadata] --> B[debootstrap
Debian bookworm] + B --> C[install edition
packages.list] + C --> D[apply overlays
base + edition] + D --> E[configure-system.sh
in chroot] + E --> F[rootfs .tar.xz] + F --> G[grub-mkrescue + squashfs
→ live ISO] + E -.toolchain .debs.-> C +``` + +Pipeline stages live in `scripts/`: + +| Stage | Script | What it does | +|-------|--------|--------------| +| bootstrap | `build-rootfs.sh` | debootstrap minbase, apt sources, package install | +| overlay | `apply-overlays.sh` | copies `overlays/base` + `overlays/` into the rootfs | +| configure | `configure-system.sh` | runs *in the chroot*: hostname, locale, kernel cmdline, services, live-boot, toolchain | +| package | `build-iso.sh` | kernel + initramfs + squashfs → hybrid BIOS/UEFI ISO | +| orchestrate | `build-edition.sh` / `Makefile` | wire the above to `make iso-` | + +## The four source trees + +| Path | Role | +|------|------| +| `editions/` | per-edition **manifests**: package lists, kernel cmdline, fstab, metadata | +| `overlays/` | **files that land in the image**, organised as layered rootfs trees | +| `scripts/` | the **build pipeline** (all plain bash, readable top to bottom) | +| `btrfs/`, `toolchain/`, `observability/`, `tests/`, `ci/` | supporting subsystems | + +There is no hidden magic: the Makefile is a thin wrapper, `versions.mk` / +`scripts/common.sh` hold the single source of truth for versions and paths. + +## Zero telemetry, enforced + +- No distro telemetry packages are installed (`ubuntu-report`, + `popularity-contest`, snapd are never in a package list). +- apt automatic-update timers are **masked** in every edition's metadata. +- The firewall's output chain never initiates calls on its own. +- cloud-init is pointed only at the configured cloud datasource. +- The smoke tests (`tests/smoke/verify-rootfs.sh`) fail the build if telemetry + artifacts are found. + +## Follow-up work (explicitly out of scope for "the wires") + +- Partitioning/installer that runs `btrfs/init.sh` on a target disk. +- Cloud images (qcow2/raw) for the `cloud` edition — the ISO path is wired, + a disk-image path is the next step. +- Grafana/Loki packaged as `.deb`s rather than fetched from upstream. +- A signed (secure-boot) kernel for official releases. diff --git a/docs/building.md b/docs/building.md new file mode 100644 index 0000000..7508159 --- /dev/null +++ b/docs/building.md @@ -0,0 +1,92 @@ +# Building Arcline OS + +## Prerequisites + +A Debian-family host (or container) with root/sudo: + +``` +debootstrap # rootfs bootstrap +squashfs-tools # ISO rootfs compression +grub2-common # grub-mkrescue (hybrid ISO) +xorriso cpio # ISO assembly +bash make curl git # build orchestration +go (≥ 1.21) # only needed for `make toolchain` +``` + +Check or install them: + +``` +scripts/check-host-deps.sh # report what's missing +scripts/check-host-deps.sh --install # apt-get install the missing bits +``` + +## Quickstart + +``` +make check # validate the tree (fast, offline, safe) +make rootfs-server # build just the server rootfs +make iso-server # build a bootable server ISO +make iso # build all three editions +make toolchain # build the 11 Go tools into .deb +make test # run smoke tests against built rootfs(es) +``` + +## Outputs + +Everything lands under `build/`: + +| Path | Contents | +|------|----------| +| `build/rootfs//` | extracted rootfs (from `make rootfs-*`) | +| `build/artifacts/arcline---.tar.xz` | archived rootfs | +| `build/artifacts/arcline---.iso` | bootable live ISO | +| `build/debs/*.deb` | Arcline toolchain packages (from `make toolchain`) | + +Each artifact ships with a `.sha256` checksum file. + +## Reproducibility knobs + +Set these as environment variables or edit `versions.mk`: + +| Variable | Meaning | Default | +|----------|---------|---------| +| `VERSION` | release version string | `0.1.0` | +| `DEBIAN_SUITE` | Debian base | `bookworm` | +| `ARCH` | target architecture | `amd64` | +| `DEBIAN_MIRROR` | mirror for the base | `http://deb.debian.org/debian` | +| `ARCLINE_LIVE` | install live-boot into rootfs | unset (set by ISO build) | +| `ARCLINE_LOCK_ROOT` | lock root account (`passwd -l root`) | `0` | +| `ARCLINE_EXTRA_REPOS` | fetch grafana/loki upstream repos | `0` | +| `ARCLINE_TOOLCHAIN` | toolchain in image builds | `auto` | + +## Building without the Arcline toolchain + +The 11 Go tools (see `docs/toolchain.md`) are **optional** in an image — a +normal `make iso-server` with no `.deb`s in `build/debs/` already produces a +minimal, toolchain-free image. This is controlled by `ARCLINE_TOOLCHAIN`: + +| Mode | Behaviour | +|------|-----------| +| `auto` (default) | install the tools if `build/debs/*.deb` exist, otherwise build without them (with a warning) | +| `skip` | never install the tools, even if debs are present | +| `require` | **fail** the build if no toolchain debs are available | + +Examples: + +```bash +make iso-server # auto: tools in if you ran `make toolchain` +make iso-server-minimal # skip: guaranteed toolchain-free image +ARCLINE_TOOLCHAIN=skip make iso-server +ARCLINE_TOOLCHAIN=require make iso-server # release builds must ship the tools +``` + +Everything else — hardened base, firewall, btrfs, observability +(Prometheus/node_exporter are Debian-main packages), zero-telemetry — is +independent of the toolchain and always included. + +## Notes on the ISO + +The ISO boots a **live** system (via `live-boot`): the rootfs is compressed to +a squashfs and mounted on boot, so you can try an edition before installing it +to disk. The same rootfs can be installed with the btrfs layout via +`btrfs/init.sh` (the installer is follow-up work — see `docs/architecture.md`). diff --git a/docs/editions.md b/docs/editions.md new file mode 100644 index 0000000..2477d92 --- /dev/null +++ b/docs/editions.md @@ -0,0 +1,44 @@ +# Editions + +Three flavours, one hardened base. Each is defined entirely by its manifest +under `editions//`. + +| Edition | Codename | Purpose | Kernel | Notes | +|---------|----------|---------|--------|-------| +| `server` | bastion | production server | generic (`linux-image-amd64`) | full observability stack, containers | +| `workstation` | forge | hardened daily driver | generic | KDE Plasma, dev toolchains | +| `cloud` | nimbus | cloud images | cloud (`linux-image-cloud-amd64`) | cloud-init, guest agents | + +## What an edition manifest contains + +``` +editions// +├── metadata.yaml # codename, summary, image type, services to enable/mask +├── packages.list # Debian packages (comments allowed) +├── kernel.cmdline # boot parameters for this edition +└── fstab # /etc/fstab template (btrfs subvolumes, UUID placeholder) +``` + +Edition-specific configs that land in the image live in `overlays//` and +are layered **after** `overlays/base/`, so they win conflicts. + +## server — bastion + +The flagship. Hardened base + Docker/Podman + Prometheus/Grafana/Loki +pre-configured (`overlays/server/etc/prometheus|grafana|loki|promtail`). Ships +the full Arcline toolchain. Serial console for headless boxes. + +## workstation — forge + +Same hardening baseline, relaxed only where a developer needs it (`perf`, +rootless containers). Curated KDE Plasma (not the full task meta-package), +dev toolchains (Go, Rust, Python, Node, clang/LLVM, CMake/Ninja), containers, +and privacy-oriented defaults. + +## cloud — nimbus + +Minimal footprint: cloud kernel, cloud-init (NoCloud/ConfigDrive/EC2/GCE/ +Azure), guest agents (qemu/vmware), NVMe + iSCSI + multipath tooling. +`net.ifnames=0` for predictable, provider-friendly interface naming. The +disk-image (qcow2) output path is the next milestone — the manifest is ready, +the ISO path is wired today. diff --git a/docs/hardening.md b/docs/hardening.md new file mode 100644 index 0000000..14bd123 --- /dev/null +++ b/docs/hardening.md @@ -0,0 +1,83 @@ +# Hardening guide + +Every edition ships the same hardening baseline (the `overlays/base` layer). +This page explains what each piece does and how to tune it. Everything is +documented *in* the config files too — read them, they are the source of truth. + +## Kernel + +- **Boot cmdline** (`editions/*/kernel.cmdline`) — applied to both the live ISO + and (via `configure-system.sh` → `/etc/default/grub`) any installed system: + - `init_on_alloc=1 init_on_free=1` — zero freshly allocated and freed memory + - `slab_nomerge` — prevent slab object merging + - `page_poison=1` — fill freed pages to catch use-after-free + - `pti=on` — kernel page-table isolation + - `spectre_v2=on spec_store_bypass=on tsx=off` — mitigations forced on + - `lockdown=integrity` — kernel refuses unsigned in-memory modification + - `oops=panic panic=-1` — halt (and stay down) on kernel oops +- **sysctl** (`overlays/base/etc/sysctl.d/10-arcline-hardening.conf`): + - memory: sane dirty ratios, no overcommit + - network: strict rp_filter, no ICMP redirects, syncookies, TCP hardening + - visibility: `dmesg_restrict=1`, `kptr_restrict=2`, `perf_event_paranoid=3`, + `ptrace_scope=1`, `unprivileged_bpf_disabled=1` + - filesystem: protected hardlinks/symlinks/fifos, `suid_dumpable=0` +- **Module blacklist** (`overlays/base/etc/modprobe.d/arcline-hardening.conf`): + - exotic network protocols (`sctp`, `dccp`, `rds`, `tipc`) and legacy + filesystems (`cramfs`, `hfs`, `hfsplus`, `jffs2`, `udf`, …). + +> The **workstation** edition relaxes a few of these for developer ergonomics +> (`perf`, tracing, rootless containers) via +> `overlays/workstation/etc/sysctl.d/90-arcline-desktop.conf`. + +## Firewall — nftables, default-deny + +`overlays/base/etc/nftables.conf` ships a **drop-by-default** policy: + +- allow: loopback, established/related, ICMP (for PMTU discovery) +- allow: `tcp/22` (ssh — key auth only) +- everything else inbound: **dropped** +- forward chain: **dropped** (container bridges manage their own rules) + +Open a port by editing the file and `systemctl reload nftables`, e.g.: + +``` +sudo nft add rule inet filter input tcp dport 443 accept +``` + +You opt in to exposure; nothing is open unless you say so. + +## ssh + +`overlays/base/etc/ssh/sshd_config.d/10-arcline-hardening.conf`: + +- `PermitRootLogin prohibit-password` — root only via key +- `PasswordAuthentication no` — keys only (this is the secure default; change + deliberately) +- `MaxAuthTries 3`, `LoginGraceTime 30`, `ClientAliveInterval 300` +- no `X11Forwarding`, `UseDNS no` + +## Logging + +`journald.conf.d/10-arcline.conf` — persistent, bounded (500M), compressed, +14-day retention. Logs live on the `@log` subvolume so they survive rollbacks. + +## AppArmor + +Arcline uses Debian's AppArmor profiles (loaded at boot) plus the kernel +hardening above. Extra profiles can be dropped into +`/etc/apparmor.d/local/` — the enforcement is on by default. + +## systemd hardening + +`system.conf.d/10-arcline.conf` — no core dumps on disk, sane default limits +(`NOFILE=65535`, `NPROC=4096`). + +## Verifying a build + +``` +tests/smoke/verify-rootfs.sh build/rootfs/server +``` + +The smoke test asserts the guarantees above (kptr_restrict, policy drop, ssh +keys-only, no telemetry, no snapd, btrfs tooling present). The build fails if +any are missing. diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 0000000..fed0a6d --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,53 @@ +# Observability + +The server edition ships a pre-configured, **local-only** observability stack — +the same setup Arcline IT runs in production. It never talks to any external +service. + +``` + node_exporter ──► Prometheus ──► Grafana (localhost:3000) + │ ▲ + journald ─────► promtail ──► Loki (localhost:3100) +``` + +| Component | Port | Role | Config source | +|-----------|------|------|---------------| +| node_exporter | `9100` | host metrics | Debian package (enabled by default) | +| Prometheus | `9090` | metric storage + alerting | `overlays/server/etc/prometheus/` | +| Grafana | `3000` | dashboards (auto-provisioned) | `overlays/server/etc/grafana/` | +| Loki | `3100` | log storage (14d retention) | `overlays/server/etc/loki/` | +| promtail | `9080` | ships journald + /var/log → Loki | `overlays/server/etc/promtail/` | + +## What is pre-configured + +- **Prometheus** scrapes itself + `node_exporter` and loads alerting rules + (`arcline.rules.yml`): NodeDown, high load, disk > 85% / critical > 95%, + service crash-looping. +- **Grafana** auto-provisions the Prometheus/Loki datasources and a starting + Node Overview dashboard (`node-overview.json`) on first start. +- **Loki** is single-node filesystem storage with a 14-day retention, matching + the journald retention. +- **promtail** ships the systemd journal and `/var/log` to the *local* Loki. + +## Packaging note + +Prometheus + node_exporter + alertmanager are in Debian main and install with +the edition packages. **Grafana** and **Loki** are not: + +- Grafana is available via the official `apt.grafana.com` repo — add it by + setting `ARCLINE_EXTRA_REPOS=1` during the build + (`configure-system.sh` adds the repo + key). +- Loki ships as a static binary tarball; the packaging (a `.deb` in + `toolchain/`) is follow-up work. + +Until then, the Grafana/Loki configs ship dormant in the image, ready for when +the binaries are installed — the `overlays/server` configs are the contract. + +## Access + +- Grafana UI: `http://:3000` (default admin credentials are set at + first boot by the installer — never left at Grafana's defaults). +- Prometheus: `http://:9090`. + +Both are **not** exposed by the default-deny firewall; open ports deliberately +or access via ssh tunnel. diff --git a/docs/toolchain.md b/docs/toolchain.md new file mode 100644 index 0000000..ff66dd5 --- /dev/null +++ b/docs/toolchain.md @@ -0,0 +1,49 @@ +# The Arcline toolchain + +Eleven Go tools, built in production at Arcline IT, shipped with Arcline OS. +The landing page describes them; `toolchain/` packages them. + +## The 11 tools + +| Tool | What it does | +|------|--------------| +| `arcline-uptime` | HTTP / TCP / TLS / DNS monitoring, Prometheus metrics export | +| `arcline-check` | CDN + transparency auditor (detects data leaks) | +| `arcline-audit` | full site health scanner — SSL, headers, performance, a11y | +| `arcline-dns` | DNS propagation checker across resolvers | +| `arcline-vault` | encrypted secrets store — REST API + CLI | +| `arcline-email` | self-hosted SMTP/IMAP — OpenSMTPD + Dovecot + Rspamd | +| `arcline-migrate` | cPanel / Plesk → Arcline migration | +| `arcline-billing` | Stripe subscription management | +| `arcline-portal` | customer dashboard — SSL monitoring + ticketing | +| `arcline-website` | Go HTTP server powering arcline.it | +| `arcline-status` | static status page generator | + +## How they get into the image + +```mermaid +flowchart LR + A[tools.list] --> B[build-tools.sh] + B --> C[.deb per tool] + C --> D[build-rootfs.sh
drops debs into chroot] + D --> E[configure-system.sh
dpkg -i *.deb] +``` + +1. `make toolchain` (or `toolchain/build-tools.sh`) clones each repo from + `git.arcline.it` (or reuses checkouts in `ARCLINE_TOOLS_DIR`), builds a + release binary (version stamped via `-ldflags`), and packages a `.deb` + from `toolchain/debian/control.tmpl` into `build/debs/`. +2. `build-rootfs.sh` copies any built debs into the chroot and + `configure-system.sh` installs them. + +A tool that can't be fetched or built is **skipped with a warning** — the OS +image never breaks because a repo is unreachable. + +## Notes + +- Repo list lives in `toolchain/tools.list` (`name|repo|description`). +- The tools' actual source lives on `git.arcline.it`; this repo is the + packaging harness. Each tool gets a proper `Debian` control file with a + generated description. +- The ticket/portal/website tools are the *products*; the others are + infrastructure utilities. All are GPL and auditable.