ArclineIT/docs
1
0
Fork 0
Code Issues 40 Pull Requests Actions Packages Projects Releases Wiki Activity

DOCS-1: Init document work

Browse Source
This commit is contained in:
Blake Ridgway
2026-07-28 07:20:32 -05:00
parent 8f02a3fc8e
commit 0cbcc962f7
66 changed files with 12224 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files

715
DEPLOYMENT.md Normal file
View File

@@ -0,0 +1,715 @@
# Deployment Guide
> **From source to production** — how every Arcline service is deployed,
> managed, and maintained.
---
## Table of Contents
- [Overview](#overview)
- [Production Environment](#production-environment)
- [Deployment Methods](#deployment-methods)
- [Service Directory Layout](#service-directory-layout)
- [User & Permissions Model](#user--permissions-model)
- [Systemd Service Units](#systemd-service-units)
- [Nginx Reverse Proxy](#nginx-reverse-proxy)
- [Deploy by Service](#deploy-by-service)
- [Database Management](#database-management)
- [Backup & Restore](#backup--restore)
- [Rollback Procedures](#rollback-procedures)
- [Health Checks](#health-checks)
- [Troubleshooting](#troubleshooting)
---
## Overview
Every Arcline service deploys as a single static Go binary. The deployment
strategy depends on the service type:
| Type | Method | Examples |
|------|--------|---------|
| **Web service** | Docker container | portal, billing, docs |
| **Native binary** | Direct install on host | website (OpenBSD), uptime |
| **CLI tool** | Install to `$PATH` | check, audit, dns, migrate, vault |
All services share the same core deployment flow:
1. **Build** — Compile static Go binary with `CGO_ENABLED=0`
2. **Package** — Containerize (Docker) or ship raw binary
3. **Deploy** — Transfer to production host and start
4. **Verify** — Health check endpoint or process monitoring
---
## Production Environment
### Host Specifications
| Host | Role | OS | Location |
|------|------|----|----------|
| `srv01` | Primary web server | OpenBSD | Arcline datacenter |
| `srv02` | Application server | Linux (Alpine) | Arcline datacenter |
| `srv03` | Database & storage | Linux (Alpine) | Arcline datacenter |
### Network
- All services listen on **127.0.0.1** (localhost) only
- Nginx handles TLS termination and reverse proxy to local services
- Public access is through pfSense port forwarding (443 → Nginx)
### Runtime Dependencies
- **Docker** (for containerized services): `docker-ce`
- **Nginx**: `nginx` (OpenBSD: `nginx` package)
- **SQLite**: No runtime dependency (pure Go, file-based)
- **ca-certificates**: Required in all containers for TLS
---
## Deployment Methods
### Method 1: Docker Deployment (Web Services)
Used for: portal, billing, docs (dynamic), email (future)
```bash
# ── On production host ────────────────────────────────────────────────
# 1. Pull the latest image
docker login -u <user> -p <token> <registry>
docker pull <registry>/<project>:latest
# 2. Stop and remove existing container
docker stop <container-name> 2>/dev/null || true
docker rm <container-name> 2>/dev/null || true
# 3. Start new container
docker run -d \
--name <container-name> \
--restart unless-stopped \
-p 127.0.0.1:<host-port>:<container-port> \
-v /opt/<service>/.env:/app/.env:ro \
-v /opt/<service>/data:/app/data \
<registry>/<project>:latest
# 4. Verify
docker ps | grep <container-name>
docker logs <container-name> --tail 20
```
### Method 2: Native Binary (OpenBSD)
Used for: website
```bash
# ── On build machine ──────────────────────────────────────────────────
# Cross-compile
GOOS=openbsd GOARCH=amd64 go build -ldflags="-s -w" -o binary-openbsd-amd64 .
# Deploy
scp binary-openbsd-amd64 srv01:/usr/local/bin/<binary>
rsync -av --delete static/ srv01:/var/www/<domain>/static/
# ── On production host ────────────────────────────────────────────────
# Restart service
doas rcctl restart <service>
```
### Method 3: Static Binary (CLI Tools)
Used for: uptime, status, check, audit, dns, migrate, vault
```bash
# ── On build machine ──────────────────────────────────────────────────
# Cross-compile for the target architecture
GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o binary-linux-amd64 .
# Deploy
scp binary-linux-amd64 srv02:/usr/local/bin/<binary>
ssh srv02 "chmod 0755 /usr/local/bin/<binary>"
# Verify
ssh srv02 "<binary> version"
```
---
## Service Directory Layout
### Containerized Services
```
/opt/arcline-<service>/
├── .env # Environment variables (arcline:arcline, 0640)
└── data/ # Persistent data directory
└── <service>.db # SQLite database (created at runtime)
```
### Native Services (OpenBSD)
```
/usr/local/bin/
├── arcline-web # Go binary
└── rc.d/ # rc.d scripts (managed by rcctl)
/var/www/arclineit/
├── static/ # CSS, JS, images
├── templates/ # Go HTML templates
└── .env # Environment config
```
### CLI Tools
```
/usr/local/bin/
├── arcline-uptime
├── arcline-status
├── arcline-check
├── arcline-audit
├── arcline-dns
├── arcline-migrate
├── arcline-vault
└── arcline-email
/etc/arcline/
├── uptime.yaml # Uptime monitor config
├── status.yaml # Status page config
├── status.d/ # Modular status config directory
└── email.toml # Email server config
/var/lib/arcline/
├── uptime.db # Uptime monitoring database
└── vault/ # Vault secrets storage
```
---
## User & Permissions Model
### System User
```bash
# Create the arcline user (done once per host)
addgroup -S arcline
adduser -S -G arcline -h /opt arcline
```
### Directory Permissions
```bash
# Application directory
chown -R arcline:arcline /opt/arcline-<service>
chmod 0750 /opt/arcline-<service>
# Environment file (sensitive!)
chown arcline:arcline /opt/arcline-<service>/.env
chmod 0640 /opt/arcline-<service>/.env
# Database file (created by app, but ensure permissions)
chown arcline:arcline /opt/arcline-<service>/data/*.db
chmod 0640 /opt/arcline-<service>/data/*.db
```
### Docker Container User
All Docker containers run as the `arcline` non-root user:
```dockerfile
RUN addgroup -S arcline && adduser -S -G arcline arcline
USER arcline
```
---
## Systemd Service Units
### Template: `arcline-<service>.service`
```ini
[Unit]
Description=Arcline <Service Name> — <short description>
After=network.target
[Service]
Type=simple
User=arcline
Group=arcline
WorkingDirectory=/opt/arcline-<service>
EnvironmentFile=/opt/arcline-<service>/.env
ExecStart=/opt/arcline-<service>/<binary> [flags]
Restart=on-failure
RestartSec=5s
# Hardening
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/arcline-<service>
[Install]
WantedBy=multi-user.target
```
### Service Management
```bash
# Install the unit file
sudo cp arcline-<service>.service /etc/systemd/system/
sudo systemctl daemon-reload
# Enable on boot
sudo systemctl enable arcline-<service>
# Start / stop / restart / status
sudo systemctl start arcline-<service>
sudo systemctl stop arcline-<service>
sudo systemctl restart arcline-<service>
sudo systemctl status arcline-<service>
# View logs
sudo journalctl -u arcline-<service> -f
```
### Hardening Options
| Option | Purpose |
|--------|---------|
| `NoNewPrivileges=yes` | Prevent privilege escalation via `suid` binaries |
| `PrivateTmp=yes` | Isolated `/tmp` for the service |
| `ProtectSystem=strict` | Read-only `/usr` and `/etc` |
| `ProtectHome=yes` | No access to `/home`, `/root` |
| `ReadWritePaths=...` | Only allow writes to the data directory |
**Reference implementation:** `portal/deploy/arcline-portal.service`
---
## Nginx Reverse Proxy
### Template: `nginx-<service>.conf`
```nginx
# HTTP → HTTPS redirect
server {
listen 80;
server_name <domain>;
return 301 https://$host$request_uri;
}
# HTTPS server
server {
listen 443 ssl;
server_name <domain>;
ssl_certificate /etc/ssl/<domain>/fullchain.pem;
ssl_certificate_key /etc/ssl/<domain>/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
# Security headers
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
add_header X-Frame-Options DENY always;
add_header X-Content-Type-Options nosniff always;
location / {
proxy_pass http://127.0.0.1:<upstream-port>;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 30s;
}
}
```
### SSL Certificate Management
Certificates from **Let's Encrypt** (certbot) or **Step CA** (internal).
```bash
# External (Let's Encrypt)
certbot certonly --webroot -w /var/www/acme -d <domain>
# Internal (Step CA)
step certificate install /etc/step-ca/certs/intermediate_ca.crt
```
**Reference implementation:** `portal/deploy/nginx-portal.conf`
---
## Deploy by Service
### website — `arcline.it`
| Detail | Value |
|--------|-------|
| **Method** | Native binary on OpenBSD |
| **Binary** | `/usr/local/bin/arcline-web` |
| **Static files** | `/var/www/arclineit/static/` |
| **Service manager** | OpenBSD `rcctl` |
| **Deploy command** | `make deploy` |
```bash
# Quick deploy
make cross
scp arcline-web-openbsd-amd64 srv01:/usr/local/bin/arcline-web
rsync -av --delete static/ srv01:/var/www/arclineit/static/
ssh srv01 "rcctl restart arcline-web"
```
### billing — `client.arcline.it`
| Detail | Value |
|--------|-------|
| **Method** | Docker container |
| **Container name** | `arcline-billing` |
| **Internal port** | 8082 |
| **Data volume** | `/opt/arcline-billing/.env:/app/.env:ro` |
| **Database** | SQLite in container (ephemeral — backup on host) |
```bash
docker run -d \
--name arcline-billing \
--restart unless-stopped \
-p 127.0.0.1:8082:8082 \
-v /opt/arcline-billing/.env:/app/.env:ro \
<registry>/arcline-billing:latest
```
### portal — `portal.arclineit.com`
| Detail | Value |
|--------|-------|
| **Method** | Docker container |
| **Container name** | `arcline-portal` |
| **Internal port** | 8082 |
| **Data volume** | `/opt/arcline-portal/.env:/app/.env:ro` |
| **Database** | SQLite in container (ephemeral — backup on host) |
```bash
docker run -d \
--name arcline-portal \
--restart unless-stopped \
-p 127.0.0.1:8082:8082 \
-v /opt/arcline-portal/.env:/app/.env:ro \
<registry>/arcline-portal:latest
```
### git — `git.arcline.it`
| Detail | Value |
|--------|-------|
| **Method** | Docker container |
| **Container name** | `arcline-gitea` |
| **Internal ports** | 3000 (web), 22 (SSH) |
| **Data volume** | `/opt/arcline-gitea/data:/data` |
| **Database** | SQLite (or PostgreSQL for multi-instance) |
```bash
# Deploy with docker-compose
docker compose -f git/deploy/docker-compose.yml up -d
# Or manual
docker run -d \
--name arcline-gitea \
--restart unless-stopped \
-p 127.0.0.1:3000:3000 \
-p 22:22 \
-v /opt/arcline-gitea/data:/data \
-e DOMAIN=git.arcline.it \
-e ROOT_URL=https://git.arcline.it \
-e SSH_DOMAIN=git.arcline.it \
gitea/gitea:latest
# Nginx config (redirect + proxy)
sudo cp git/deploy/nginx-git-redirect.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
# Verify
curl -I https://git.arcline.it
ssh -T _gitea@git.arcline.it
```
### uptime — Internal Monitor
| Detail | Value |
|--------|-------|
| **Method** | Static binary or Docker |
| **Binary** | `/usr/local/bin/arcline-uptime` |
| **Config** | `/etc/arcline/uptime.yaml` |
| **Database** | `/var/lib/arcline/uptime.db` |
```bash
# As a service
arcline-uptime start --config /etc/arcline/uptime.yaml
# Or via systemd
[Service]
ExecStart=/usr/local/bin/arcline-uptime start --config /etc/arcline/uptime.yaml
```
### status — `status.arclineit.com`
| Detail | Value |
|--------|-------|
| **Method** | Static HTML generation |
| **Binary** | `/usr/local/bin/arcline-status` |
| **Config** | `/etc/arcline/status.yaml` + `/etc/arcline/status.d/` |
| **Output** | `/var/www/status/` (served by Nginx) |
```bash
# Generate status page
arcline-status build --config /etc/arcline/status.yaml --out /var/www/status/
# Watch mode (regenerate on config change)
arcline-status build --config /etc/arcline/status.yaml --out /var/www/status/ --watch
```
### docs — `docs.arclineit.com`
| Detail | Value |
|--------|-------|
| **Method** | Docker container (dynamic) or Nginx (static) |
| **Container name** | `arcline-docs` |
| **Internal port** | 8080 |
```bash
# Dynamic variant (Go server with Markdown rendering)
docker run -d \
--name arcline-docs \
--restart unless-stopped \
-p 127.0.0.1:8080:8080 \
-v /opt/arcline-docs/.env:/app/.env:ro \
<registry>/arcline-docs:latest
# Static variant (pre-built HTML via Dockerfile.static)
docker run -d \
--name arcline-docs \
--restart unless-stopped \
-p 127.0.0.1:80:80 \
<registry>/arcline-docs:latest-static
```
---
## Database Management
### Backup
```bash
# SQLite databases — simple file copy
cp /opt/arcline-<service>/data/<service>.db /backup/<service>-$(date +%Y%m%d).db
# Compress
gzip /backup/<service>-*.db
```
### Restore
```bash
# Stop the service first
docker stop arcline-<service>
# or
systemctl stop arcline-<service>
# Restore database
cp /backup/<service>-<date>.db.gz /opt/arcline-<service>/data/
gunzip /opt/arcline-<service>/data/<service>-<date>.db.gz
mv /opt/arcline-<service>/data/<service>-<date>.db /opt/arcline-<service>/data/<service>.db
chown arcline:arcline /opt/arcline-<service>/data/<service>.db
# Restart
docker start arcline-<service>
# or
systemctl start arcline-<service>
```
### Maintenance
```bash
# Vacuum SQLite database (reclaim space)
sqlite3 /opt/arcline-<service>/data/<service>.db "VACUUM;"
# Integrity check
sqlite3 /opt/arcline-<service>/data/<service>.db "PRAGMA integrity_check;"
```
---
## Backup & Restore
### What to Back Up
| Asset | Location | Frequency |
|-------|----------|-----------|
| SQLite databases | `/opt/arcline-*/data/*.db` | Daily |
| Environment files | `/opt/arcline-*/.env` | On change |
| SSL certificates | `/etc/ssl/` | On renewal |
| Nginx configs | `/etc/nginx/` | On change |
| Systemd units | `/etc/systemd/system/arcline-*.service` | On change |
| Status config | `/etc/arcline/` | On change |
| Uptime config | `/etc/arcline/uptime.yaml` | On change |
### Backup Script
```bash
#!/bin/sh
# /usr/local/bin/arcline-backup
BACKUP_DIR="/backup/arcline/$(date +%Y-%m-%d)"
mkdir -p "$BACKUP_DIR"
# Databases
for db in /opt/arcline-*/data/*.db; do
cp "$db" "$BACKUP_DIR/"
done
# Environment files (mask secrets)
for env in /opt/arcline-*/.env; do
cp "$env" "$BACKUP_DIR/$(basename $(dirname $env)).env"
done
# Configs
cp -r /etc/arcline "$BACKUP_DIR/"
# Compress
tar czf "$BACKUP_DIR.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)"
rm -rf "$BACKUP_DIR"
# Sync to backup host
rsync -av "$BACKUP_DIR.tar.gz" backup-host:/backups/arcline/
```
---
## Rollback Procedures
### Docker Rollback
```bash
# 1. Check previous image tags
docker images <registry>/<project>
# 2. Deploy a specific tag instead of latest
docker run -d \
--name <container>-rollback \
<registry>/<project>:<previous-sha>
# 3. Verify health
curl http://127.0.0.1:<port>/health
# 4. Swap if healthy
docker stop <container>
docker rm <container>
docker rename <container>-rollback <container>
```
### Native Binary Rollback
```bash
# 1. Keep previous binary versions
cp /usr/local/bin/<binary> /usr/local/bin/<binary>.bak
# 2. Restore previous version
cp /usr/local/bin/<binary>.bak /usr/local/bin/<binary>
# 3. Restart service
systemctl restart <service>
# or
rcctl restart <service>
```
---
## Health Checks
### Docker Health Check (in Dockerfile)
```dockerfile
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1
```
### Manual Health Check
```bash
# Web service
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:<port>/health
# Process check
pgrep -x <binary>
# Docker check
docker ps --filter "name=<container>" --filter "status=running"
```
---
## Troubleshooting
### Service Won't Start
```bash
# Check systemd logs
journalctl -u arcline-<service> -f
# Check Docker logs
docker logs arcline-<service>
# Check binary directly
/opt/arcline-<service>/<binary> 2>&1
# Common issues:
# - Missing .env file → ExecStart fails
# - Port conflict → change port in .env
# - Permission denied → check chown/chmod
```
### Database Issues
```bash
# Check SQLite integrity
sqlite3 /opt/arcline-<service>/data/<service>.db "PRAGMA integrity_check;"
# Expected: "ok"
# Check disk space
df -h /opt/arcline-<service>/data/
# Check file permissions
ls -la /opt/arcline-<service>/data/<service>.db
# Expected: -rw-r----- arcline arcline
```
### Connection Refused
```bash
# Check if service is listening
ss -tlnp | grep <port>
# Check Nginx is running
systemctl status nginx
# Check pfSense NAT rules
# Verify port forwarding from WAN:443 → host:443
```
### Quick Recovery
```bash
# Universal restart sequence
docker stop arcline-<service> 2>/dev/null
docker rm arcline-<service> 2>/dev/null
docker pull <registry>/<project>:latest
docker run -d --restart unless-stopped \
-p 127.0.0.1:<port>:<port> \
-v /opt/arcline-<service>/.env:/app/.env:ro \
<registry>/<project>:latest
```