arcline-project/os-build
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add architecture, hardening, and building documentation

Browse Source 
Document the design and how to operate it: architecture, building
(including toolchain-free/minimal builds), hardening guide, editions,
observability, and the toolchain. README ties it together.
This commit is contained in:
Blake Ridgway
2026-08-21 13:15:43 -05:00
parent 33652064f9
commit 87fc370541
7 changed files with 474 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

85
README.md Normal file
View File

@@ -0,0 +1,85 @@
# Arcline OS — build system
> The Linux OS for people who run infrastructure. Hardened Debian base.
> Pre-configured monitoring, auditing, and security tooling. Zero telemetry.
This repository is the build system ("the wires") behind
[Arcline OS](https://arcline.it). It turns a Debian base and a set of plain-text
manifests into hardened, bootable operating system images for three editions:
**Server**, **Workstation**, and **Cloud**.
## What you get
- A transparent, script-based build pipeline (no magic, everything auditable)
- `debootstrap` → package install → overlay → in-chroot configure → live ISO
- **Secure by default**: hardened kernel cmdline + sysctl, default-deny
nftables, key-only ssh, AppArmor, no core dumps
- **btrfs-native**: subvolume layout, scheduled read-only snapshots, and
boot-to-snapshot rollback tooling
- **Zero telemetry**: enforced by package selection, masked apt timers, and
smoke tests that fail the build if telemetry is found
- **Pre-configured observability** (server): Prometheus + node_exporter +
Grafana (auto-provisioned) + Loki + promtail, local-only
- **The Arcline toolchain**: a harness that packages all 11 Go tools into
`.deb`s and installs them into the image
- Smoke tests, GitLab CI, and docs that explain every decision
## Quickstart
```bash
# on a Debian-family host with root/sudo
make deps # install host build dependencies
make check # validate the tree (fast, offline)
make iso-server # build a bootable server ISO
make iso # build all three editions
make toolchain # build the 11 Go tools into .deb
make iso-server-minimal # server ISO WITHOUT the Arcline toolchain
make test # run smoke tests against built rootfs(es)
```
Artifacts land in `build/artifacts/` with `.sha256` checksums.
The Arcline tools are **optional** in an image (`ARCLINE_TOOLCHAIN=auto|skip|
require`; see [building](docs/building.md#building-without-the-arcline-toolchain)).
## Layout
```
os-build/
├── Makefile # thin orchestration (make iso-<edition>)
├── versions.mk # single source of truth for versions/paths
├── editions/ # per-edition manifests (packages, cmdline, fstab, metadata)
├── overlays/ # files that land in the image (base + per-edition layers)
├── scripts/ # the build pipeline (all plain bash)
├── btrfs/ # subvolume layout, snapshots, rollback
├── toolchain/ # packaging for the 11 Go tools
├── observability/ # Prometheus/Grafana/Loki configs (docs + overlays/server)
├── tests/ # tree validation + rootfs smoke tests
├── ci/ # GitLab CI pipeline
└── docs/ # architecture, hardening, building, editions, ...
```
## Documentation
| Doc | Contents |
|-----|----------|
| [architecture](docs/architecture.md) | the design and how a build flows |
| [building](docs/building.md) | prerequisites, quickstart, outputs, knobs |
| [hardening](docs/hardening.md) | every hardening decision, and how to tune it |
| [editions](docs/editions.md) | server / workstation / cloud manifests |
| [observability](docs/observability.md) | the pre-configured monitoring stack |
| [toolchain](docs/toolchain.md) | the 11 Go tools and their packaging |
| [btrfs](btrfs/README.md) | subvolume layout + snapshots + rollback |
## Status
This is the **foundation**: the build pipeline, edition manifests, hardening
baseline, btrfs tooling, observability configs, tests, and CI are in place and
runnable. The next milestones (installer that writes the btrfs layout to disk,
cloud disk images, signed releases) are listed in
[docs/architecture.md](docs/architecture.md#follow-up-work).
## License
GPL-3.0 — see [LICENSE](LICENSE). Sponsored by Arcline IT LLC.
No telemetry. No tracking.

68
docs/architecture.md Normal file
View File

@@ -0,0 +1,68 @@
# Arcline OS — architecture
This is the foundation ("the wires") for Arcline OS: a hardened, Debian-derived
operating system for people who run infrastructure. The landing page describes
the product; this repository is how it gets built.
## Design goals
1. **Secure by default** — hardened kernel, default-deny firewall, AppArmor,
locked-down ssh. You opt *in* to exposure, never out.
2. **Zero telemetry** — no phone-home, no analytics, no cloud integration.
3. **btrfs-native** — snapshots and boot-to-snapshot rollback are built in.
4. **Production-ready from first boot** — observability pre-configured, tools
pre-installed, everything documented.
5. **Auditable** — every script and config is plain text in this repo.
## How a build flows
```mermaid
flowchart LR
A[editions/* metadata] --> B[debootstrap<br/>Debian bookworm]
B --> C[install edition<br/>packages.list]
C --> D[apply overlays<br/>base + edition]
D --> E[configure-system.sh<br/>in chroot]
E --> F[rootfs .tar.xz]
F --> G[grub-mkrescue + squashfs<br/>→ live ISO]
E -.toolchain .debs.-> C
```
Pipeline stages live in `scripts/`:
| Stage | Script | What it does |
|-------|--------|--------------|
| bootstrap | `build-rootfs.sh` | debootstrap minbase, apt sources, package install |
| overlay | `apply-overlays.sh` | copies `overlays/base` + `overlays/<edition>` into the rootfs |
| configure | `configure-system.sh` | runs *in the chroot*: hostname, locale, kernel cmdline, services, live-boot, toolchain |
| package | `build-iso.sh` | kernel + initramfs + squashfs → hybrid BIOS/UEFI ISO |
| orchestrate | `build-edition.sh` / `Makefile` | wire the above to `make iso-<edition>` |
## The four source trees
| Path | Role |
|------|------|
| `editions/` | per-edition **manifests**: package lists, kernel cmdline, fstab, metadata |
| `overlays/` | **files that land in the image**, organised as layered rootfs trees |
| `scripts/` | the **build pipeline** (all plain bash, readable top to bottom) |
| `btrfs/`, `toolchain/`, `observability/`, `tests/`, `ci/` | supporting subsystems |
There is no hidden magic: the Makefile is a thin wrapper, `versions.mk` /
`scripts/common.sh` hold the single source of truth for versions and paths.
## Zero telemetry, enforced
- No distro telemetry packages are installed (`ubuntu-report`,
`popularity-contest`, snapd are never in a package list).
- apt automatic-update timers are **masked** in every edition's metadata.
- The firewall's output chain never initiates calls on its own.
- cloud-init is pointed only at the configured cloud datasource.
- The smoke tests (`tests/smoke/verify-rootfs.sh`) fail the build if telemetry
artifacts are found.
## Follow-up work (explicitly out of scope for "the wires")
- Partitioning/installer that runs `btrfs/init.sh` on a target disk.
- Cloud images (qcow2/raw) for the `cloud` edition — the ISO path is wired,
a disk-image path is the next step.
- Grafana/Loki packaged as `.deb`s rather than fetched from upstream.
- A signed (secure-boot) kernel for official releases.

92
docs/building.md Normal file
View File

@@ -0,0 +1,92 @@
# Building Arcline OS
## Prerequisites
A Debian-family host (or container) with root/sudo:
```
debootstrap # rootfs bootstrap
squashfs-tools # ISO rootfs compression
grub2-common # grub-mkrescue (hybrid ISO)
xorriso cpio # ISO assembly
bash make curl git # build orchestration
go (≥ 1.21) # only needed for `make toolchain`
```
Check or install them:
```
scripts/check-host-deps.sh # report what's missing
scripts/check-host-deps.sh --install # apt-get install the missing bits
```
## Quickstart
```
make check # validate the tree (fast, offline, safe)
make rootfs-server # build just the server rootfs
make iso-server # build a bootable server ISO
make iso # build all three editions
make toolchain # build the 11 Go tools into .deb
make test # run smoke tests against built rootfs(es)
```
## Outputs
Everything lands under `build/`:
| Path | Contents |
|------|----------|
| `build/rootfs/<edition>/` | extracted rootfs (from `make rootfs-*`) |
| `build/artifacts/arcline-<edition>-<version>-<arch>.tar.xz` | archived rootfs |
| `build/artifacts/arcline-<edition>-<version>-<arch>.iso` | bootable live ISO |
| `build/debs/*.deb` | Arcline toolchain packages (from `make toolchain`) |
Each artifact ships with a `.sha256` checksum file.
## Reproducibility knobs
Set these as environment variables or edit `versions.mk`:
| Variable | Meaning | Default |
|----------|---------|---------|
| `VERSION` | release version string | `0.1.0` |
| `DEBIAN_SUITE` | Debian base | `bookworm` |
| `ARCH` | target architecture | `amd64` |
| `DEBIAN_MIRROR` | mirror for the base | `http://deb.debian.org/debian` |
| `ARCLINE_LIVE` | install live-boot into rootfs | unset (set by ISO build) |
| `ARCLINE_LOCK_ROOT` | lock root account (`passwd -l root`) | `0` |
| `ARCLINE_EXTRA_REPOS` | fetch grafana/loki upstream repos | `0` |
| `ARCLINE_TOOLCHAIN` | toolchain in image builds | `auto` |
## Building without the Arcline toolchain
The 11 Go tools (see `docs/toolchain.md`) are **optional** in an image — a
normal `make iso-server` with no `.deb`s in `build/debs/` already produces a
minimal, toolchain-free image. This is controlled by `ARCLINE_TOOLCHAIN`:
| Mode | Behaviour |
|------|-----------|
| `auto` (default) | install the tools if `build/debs/*.deb` exist, otherwise build without them (with a warning) |
| `skip` | never install the tools, even if debs are present |
| `require` | **fail** the build if no toolchain debs are available |
Examples:
```bash
make iso-server # auto: tools in if you ran `make toolchain`
make iso-server-minimal # skip: guaranteed toolchain-free image
ARCLINE_TOOLCHAIN=skip make iso-server
ARCLINE_TOOLCHAIN=require make iso-server # release builds must ship the tools
```
Everything else — hardened base, firewall, btrfs, observability
(Prometheus/node_exporter are Debian-main packages), zero-telemetry — is
independent of the toolchain and always included.
## Notes on the ISO
The ISO boots a **live** system (via `live-boot`): the rootfs is compressed to
a squashfs and mounted on boot, so you can try an edition before installing it
to disk. The same rootfs can be installed with the btrfs layout via
`btrfs/init.sh` (the installer is follow-up work — see `docs/architecture.md`).

44
docs/editions.md Normal file
View File

@@ -0,0 +1,44 @@
# Editions
Three flavours, one hardened base. Each is defined entirely by its manifest
under `editions/<name>/`.
| Edition | Codename | Purpose | Kernel | Notes |
|---------|----------|---------|--------|-------|
| `server` | bastion | production server | generic (`linux-image-amd64`) | full observability stack, containers |
| `workstation` | forge | hardened daily driver | generic | KDE Plasma, dev toolchains |
| `cloud` | nimbus | cloud images | cloud (`linux-image-cloud-amd64`) | cloud-init, guest agents |
## What an edition manifest contains
```
editions/<name>/
├── metadata.yaml # codename, summary, image type, services to enable/mask
├── packages.list # Debian packages (comments allowed)
├── kernel.cmdline # boot parameters for this edition
└── fstab # /etc/fstab template (btrfs subvolumes, UUID placeholder)
```
Edition-specific configs that land in the image live in `overlays/<name>/` and
are layered **after** `overlays/base/`, so they win conflicts.
## server — bastion
The flagship. Hardened base + Docker/Podman + Prometheus/Grafana/Loki
pre-configured (`overlays/server/etc/prometheus|grafana|loki|promtail`). Ships
the full Arcline toolchain. Serial console for headless boxes.
## workstation — forge
Same hardening baseline, relaxed only where a developer needs it (`perf`,
rootless containers). Curated KDE Plasma (not the full task meta-package),
dev toolchains (Go, Rust, Python, Node, clang/LLVM, CMake/Ninja), containers,
and privacy-oriented defaults.
## cloud — nimbus
Minimal footprint: cloud kernel, cloud-init (NoCloud/ConfigDrive/EC2/GCE/
Azure), guest agents (qemu/vmware), NVMe + iSCSI + multipath tooling.
`net.ifnames=0` for predictable, provider-friendly interface naming. The
disk-image (qcow2) output path is the next milestone — the manifest is ready,
the ISO path is wired today.

83
docs/hardening.md Normal file
View File

@@ -0,0 +1,83 @@
# Hardening guide
Every edition ships the same hardening baseline (the `overlays/base` layer).
This page explains what each piece does and how to tune it. Everything is
documented *in* the config files too — read them, they are the source of truth.
## Kernel
- **Boot cmdline** (`editions/*/kernel.cmdline`) — applied to both the live ISO
and (via `configure-system.sh` → `/etc/default/grub`) any installed system:
- `init_on_alloc=1 init_on_free=1` — zero freshly allocated and freed memory
- `slab_nomerge` — prevent slab object merging
- `page_poison=1` — fill freed pages to catch use-after-free
- `pti=on` — kernel page-table isolation
- `spectre_v2=on spec_store_bypass=on tsx=off` — mitigations forced on
- `lockdown=integrity` — kernel refuses unsigned in-memory modification
- `oops=panic panic=-1` — halt (and stay down) on kernel oops
- **sysctl** (`overlays/base/etc/sysctl.d/10-arcline-hardening.conf`):
- memory: sane dirty ratios, no overcommit
- network: strict rp_filter, no ICMP redirects, syncookies, TCP hardening
- visibility: `dmesg_restrict=1`, `kptr_restrict=2`, `perf_event_paranoid=3`,
`ptrace_scope=1`, `unprivileged_bpf_disabled=1`
- filesystem: protected hardlinks/symlinks/fifos, `suid_dumpable=0`
- **Module blacklist** (`overlays/base/etc/modprobe.d/arcline-hardening.conf`):
- exotic network protocols (`sctp`, `dccp`, `rds`, `tipc`) and legacy
filesystems (`cramfs`, `hfs`, `hfsplus`, `jffs2`, `udf`, …).
> The **workstation** edition relaxes a few of these for developer ergonomics
> (`perf`, tracing, rootless containers) via
> `overlays/workstation/etc/sysctl.d/90-arcline-desktop.conf`.
## Firewall — nftables, default-deny
`overlays/base/etc/nftables.conf` ships a **drop-by-default** policy:
- allow: loopback, established/related, ICMP (for PMTU discovery)
- allow: `tcp/22` (ssh — key auth only)
- everything else inbound: **dropped**
- forward chain: **dropped** (container bridges manage their own rules)
Open a port by editing the file and `systemctl reload nftables`, e.g.:
```
sudo nft add rule inet filter input tcp dport 443 accept
```
You opt in to exposure; nothing is open unless you say so.
## ssh
`overlays/base/etc/ssh/sshd_config.d/10-arcline-hardening.conf`:
- `PermitRootLogin prohibit-password` — root only via key
- `PasswordAuthentication no` — keys only (this is the secure default; change
deliberately)
- `MaxAuthTries 3`, `LoginGraceTime 30`, `ClientAliveInterval 300`
- no `X11Forwarding`, `UseDNS no`
## Logging
`journald.conf.d/10-arcline.conf` — persistent, bounded (500M), compressed,
14-day retention. Logs live on the `@log` subvolume so they survive rollbacks.
## AppArmor
Arcline uses Debian's AppArmor profiles (loaded at boot) plus the kernel
hardening above. Extra profiles can be dropped into
`/etc/apparmor.d/local/` — the enforcement is on by default.
## systemd hardening
`system.conf.d/10-arcline.conf` — no core dumps on disk, sane default limits
(`NOFILE=65535`, `NPROC=4096`).
## Verifying a build
```
tests/smoke/verify-rootfs.sh build/rootfs/server
```
The smoke test asserts the guarantees above (kptr_restrict, policy drop, ssh
keys-only, no telemetry, no snapd, btrfs tooling present). The build fails if
any are missing.

53
docs/observability.md Normal file
View File

@@ -0,0 +1,53 @@
# Observability
The server edition ships a pre-configured, **local-only** observability stack —
the same setup Arcline IT runs in production. It never talks to any external
service.
```
node_exporter ──► Prometheus ──► Grafana (localhost:3000)
│ ▲
journald ─────► promtail ──► Loki (localhost:3100)
```
| Component | Port | Role | Config source |
|-----------|------|------|---------------|
| node_exporter | `9100` | host metrics | Debian package (enabled by default) |
| Prometheus | `9090` | metric storage + alerting | `overlays/server/etc/prometheus/` |
| Grafana | `3000` | dashboards (auto-provisioned) | `overlays/server/etc/grafana/` |
| Loki | `3100` | log storage (14d retention) | `overlays/server/etc/loki/` |
| promtail | `9080` | ships journald + /var/log → Loki | `overlays/server/etc/promtail/` |
## What is pre-configured
- **Prometheus** scrapes itself + `node_exporter` and loads alerting rules
(`arcline.rules.yml`): NodeDown, high load, disk > 85% / critical > 95%,
service crash-looping.
- **Grafana** auto-provisions the Prometheus/Loki datasources and a starting
Node Overview dashboard (`node-overview.json`) on first start.
- **Loki** is single-node filesystem storage with a 14-day retention, matching
the journald retention.
- **promtail** ships the systemd journal and `/var/log` to the *local* Loki.
## Packaging note
Prometheus + node_exporter + alertmanager are in Debian main and install with
the edition packages. **Grafana** and **Loki** are not:
- Grafana is available via the official `apt.grafana.com` repo — add it by
setting `ARCLINE_EXTRA_REPOS=1` during the build
(`configure-system.sh` adds the repo + key).
- Loki ships as a static binary tarball; the packaging (a `.deb` in
`toolchain/`) is follow-up work.
Until then, the Grafana/Loki configs ship dormant in the image, ready for when
the binaries are installed — the `overlays/server` configs are the contract.
## Access
- Grafana UI: `http://<host>:3000` (default admin credentials are set at
first boot by the installer — never left at Grafana's defaults).
- Prometheus: `http://<host>:9090`.
Both are **not** exposed by the default-deny firewall; open ports deliberately
or access via ssh tunnel.

49
docs/toolchain.md Normal file
View File

@@ -0,0 +1,49 @@
# The Arcline toolchain
Eleven Go tools, built in production at Arcline IT, shipped with Arcline OS.
The landing page describes them; `toolchain/` packages them.
## The 11 tools
| Tool | What it does |
|------|--------------|
| `arcline-uptime` | HTTP / TCP / TLS / DNS monitoring, Prometheus metrics export |
| `arcline-check` | CDN + transparency auditor (detects data leaks) |
| `arcline-audit` | full site health scanner — SSL, headers, performance, a11y |
| `arcline-dns` | DNS propagation checker across resolvers |
| `arcline-vault` | encrypted secrets store — REST API + CLI |
| `arcline-email` | self-hosted SMTP/IMAP — OpenSMTPD + Dovecot + Rspamd |
| `arcline-migrate` | cPanel / Plesk → Arcline migration |
| `arcline-billing` | Stripe subscription management |
| `arcline-portal` | customer dashboard — SSL monitoring + ticketing |
| `arcline-website` | Go HTTP server powering arcline.it |
| `arcline-status` | static status page generator |
## How they get into the image
```mermaid
flowchart LR
A[tools.list] --> B[build-tools.sh]
B --> C[.deb per tool]
C --> D[build-rootfs.sh<br/>drops debs into chroot]
D --> E[configure-system.sh<br/>dpkg -i *.deb]
```
1. `make toolchain` (or `toolchain/build-tools.sh`) clones each repo from
`git.arcline.it` (or reuses checkouts in `ARCLINE_TOOLS_DIR`), builds a
release binary (version stamped via `-ldflags`), and packages a `.deb`
from `toolchain/debian/control.tmpl` into `build/debs/`.
2. `build-rootfs.sh` copies any built debs into the chroot and
`configure-system.sh` installs them.
A tool that can't be fetched or built is **skipped with a warning** — the OS
image never breaks because a repo is unreachable.
## Notes
- Repo list lives in `toolchain/tools.list` (`name|repo|description`).
- The tools' actual source lives on `git.arcline.it`; this repo is the
packaging harness. Each tool gets a proper `Debian` control file with a
generated description.
- The ticket/portal/website tools are the *products*; the others are
infrastructure utilities. All are GPL and auditable.