ArclineIT/docs

Fork 0

Files

Blake Ridgway 0cbcc962f7 DOCS-1: Init document work

2026-07-28 07:20:32 -05:00

17 KiB

Raw Blame History

Deployment Guide

From source to production — how every Arcline service is deployed, managed, and maintained.

Overview
Production Environment
Deployment Methods
Service Directory Layout
User & Permissions Model
Systemd Service Units
Nginx Reverse Proxy
Deploy by Service
Database Management
Backup & Restore
Rollback Procedures
Health Checks
Troubleshooting

Overview

Every Arcline service deploys as a single static Go binary. The deployment strategy depends on the service type:

Type	Method	Examples
Web service	Docker container	portal, billing, docs
Native binary	Direct install on host	website (OpenBSD), uptime
CLI tool	Install to `$PATH`	check, audit, dns, migrate, vault

All services share the same core deployment flow:

Build — Compile static Go binary with CGO_ENABLED=0
Package — Containerize (Docker) or ship raw binary
Deploy — Transfer to production host and start
Verify — Health check endpoint or process monitoring

Production Environment

Host Specifications

Host	Role	OS	Location
`srv01`	Primary web server	OpenBSD	Arcline datacenter
`srv02`	Application server	Linux (Alpine)	Arcline datacenter
`srv03`	Database & storage	Linux (Alpine)	Arcline datacenter

Network

All services listen on 127.0.0.1 (localhost) only
Nginx handles TLS termination and reverse proxy to local services
Public access is through pfSense port forwarding (443 → Nginx)

Runtime Dependencies

Docker (for containerized services): docker-ce
Nginx: nginx (OpenBSD: nginx package)
SQLite: No runtime dependency (pure Go, file-based)
ca-certificates: Required in all containers for TLS

Deployment Methods

Method 1: Docker Deployment (Web Services)

Used for: portal, billing, docs (dynamic), email (future)

# ── On production host ────────────────────────────────────────────────

# 1. Pull the latest image
docker login -u <user> -p <token> <registry>
docker pull <registry>/<project>:latest

# 2. Stop and remove existing container
docker stop <container-name> 2>/dev/null || true
docker rm   <container-name> 2>/dev/null || true

# 3. Start new container
docker run -d \
  --name <container-name> \
  --restart unless-stopped \
  -p 127.0.0.1:<host-port>:<container-port> \
  -v /opt/<service>/.env:/app/.env:ro \
  -v /opt/<service>/data:/app/data \
  <registry>/<project>:latest

# 4. Verify
docker ps | grep <container-name>
docker logs <container-name> --tail 20

Method 2: Native Binary (OpenBSD)

Used for: website

# ── On build machine ──────────────────────────────────────────────────

# Cross-compile
GOOS=openbsd GOARCH=amd64 go build -ldflags="-s -w" -o binary-openbsd-amd64 .

# Deploy
scp binary-openbsd-amd64 srv01:/usr/local/bin/<binary>
rsync -av --delete static/ srv01:/var/www/<domain>/static/

# ── On production host ────────────────────────────────────────────────

# Restart service
doas rcctl restart <service>

Method 3: Static Binary (CLI Tools)

Used for: uptime, status, check, audit, dns, migrate, vault

# ── On build machine ──────────────────────────────────────────────────

# Cross-compile for the target architecture
GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o binary-linux-amd64 .

# Deploy
scp binary-linux-amd64 srv02:/usr/local/bin/<binary>
ssh srv02 "chmod 0755 /usr/local/bin/<binary>"

# Verify
ssh srv02 "<binary> version"

Service Directory Layout

Containerized Services

/opt/arcline-<service>/
├── .env              # Environment variables (arcline:arcline, 0640)
└── data/             # Persistent data directory
    └── <service>.db  # SQLite database (created at runtime)

Native Services (OpenBSD)

/usr/local/bin/
├── arcline-web          # Go binary
└── rc.d/                # rc.d scripts (managed by rcctl)

/var/www/arclineit/
├── static/              # CSS, JS, images
├── templates/           # Go HTML templates
└── .env                 # Environment config

CLI Tools

/usr/local/bin/
├── arcline-uptime
├── arcline-status
├── arcline-check
├── arcline-audit
├── arcline-dns
├── arcline-migrate
├── arcline-vault
└── arcline-email

/etc/arcline/
├── uptime.yaml          # Uptime monitor config
├── status.yaml          # Status page config
├── status.d/            # Modular status config directory
└── email.toml           # Email server config

/var/lib/arcline/
├── uptime.db            # Uptime monitoring database
└── vault/               # Vault secrets storage

User & Permissions Model

System User

# Create the arcline user (done once per host)
addgroup -S arcline
adduser -S -G arcline -h /opt arcline

Directory Permissions

# Application directory
chown -R arcline:arcline /opt/arcline-<service>
chmod 0750 /opt/arcline-<service>

# Environment file (sensitive!)
chown arcline:arcline /opt/arcline-<service>/.env
chmod 0640 /opt/arcline-<service>/.env

# Database file (created by app, but ensure permissions)
chown arcline:arcline /opt/arcline-<service>/data/*.db
chmod 0640 /opt/arcline-<service>/data/*.db

Docker Container User

All Docker containers run as the arcline non-root user:

RUN addgroup -S arcline && adduser -S -G arcline arcline
USER arcline

Systemd Service Units

Template: `arcline-<service>.service`

[Unit]
Description=Arcline <Service Name> — <short description>
After=network.target

[Service]
Type=simple
User=arcline
Group=arcline
WorkingDirectory=/opt/arcline-<service>
EnvironmentFile=/opt/arcline-<service>/.env
ExecStart=/opt/arcline-<service>/<binary> [flags]
Restart=on-failure
RestartSec=5s

# Hardening
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/arcline-<service>

[Install]
WantedBy=multi-user.target

Service Management

# Install the unit file
sudo cp arcline-<service>.service /etc/systemd/system/
sudo systemctl daemon-reload

# Enable on boot
sudo systemctl enable arcline-<service>

# Start / stop / restart / status
sudo systemctl start   arcline-<service>
sudo systemctl stop    arcline-<service>
sudo systemctl restart arcline-<service>
sudo systemctl status  arcline-<service>

# View logs
sudo journalctl -u arcline-<service> -f

Hardening Options

Option	Purpose
`NoNewPrivileges=yes`	Prevent privilege escalation via `suid` binaries
`PrivateTmp=yes`	Isolated `/tmp` for the service
`ProtectSystem=strict`	Read-only `/usr` and `/etc`
`ProtectHome=yes`	No access to `/home`, `/root`
`ReadWritePaths=...`	Only allow writes to the data directory

Reference implementation: portal/deploy/arcline-portal.service

Nginx Reverse Proxy

Template: `nginx-<service>.conf`

# HTTP → HTTPS redirect
server {
    listen 80;
    server_name <domain>;
    return 301 https://$host$request_uri;
}

# HTTPS server
server {
    listen 443 ssl;
    server_name <domain>;

    ssl_certificate     /etc/ssl/<domain>/fullchain.pem;
    ssl_certificate_key /etc/ssl/<domain>/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # Security headers
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
    add_header X-Frame-Options DENY always;
    add_header X-Content-Type-Options nosniff always;

    location / {
        proxy_pass         http://127.0.0.1:<upstream-port>;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_read_timeout 30s;
    }
}

SSL Certificate Management

Certificates from Let's Encrypt (certbot) or Step CA (internal).

# External (Let's Encrypt)
certbot certonly --webroot -w /var/www/acme -d <domain>

# Internal (Step CA)
step certificate install /etc/step-ca/certs/intermediate_ca.crt

Reference implementation: portal/deploy/nginx-portal.conf

Deploy by Service

website — `arcline.it`

Detail	Value
Method	Native binary on OpenBSD
Binary	`/usr/local/bin/arcline-web`
Static files	`/var/www/arclineit/static/`
Service manager	OpenBSD `rcctl`
Deploy command	`make deploy`

# Quick deploy
make cross
scp arcline-web-openbsd-amd64 srv01:/usr/local/bin/arcline-web
rsync -av --delete static/ srv01:/var/www/arclineit/static/
ssh srv01 "rcctl restart arcline-web"

billing — `client.arcline.it`

Detail	Value
Method	Docker container
Container name	`arcline-billing`
Internal port	8082
Data volume	`/opt/arcline-billing/.env:/app/.env:ro`
Database	SQLite in container (ephemeral — backup on host)

docker run -d \
  --name arcline-billing \
  --restart unless-stopped \
  -p 127.0.0.1:8082:8082 \
  -v /opt/arcline-billing/.env:/app/.env:ro \
  <registry>/arcline-billing:latest

portal — `portal.arclineit.com`

Detail	Value
Method	Docker container
Container name	`arcline-portal`
Internal port	8082
Data volume	`/opt/arcline-portal/.env:/app/.env:ro`
Database	SQLite in container (ephemeral — backup on host)

docker run -d \
  --name arcline-portal \
  --restart unless-stopped \
  -p 127.0.0.1:8082:8082 \
  -v /opt/arcline-portal/.env:/app/.env:ro \
  <registry>/arcline-portal:latest

git — `git.arcline.it`

Detail	Value
Method	Docker container
Container name	`arcline-gitea`
Internal ports	3000 (web), 22 (SSH)
Data volume	`/opt/arcline-gitea/data:/data`
Database	SQLite (or PostgreSQL for multi-instance)

# Deploy with docker-compose
docker compose -f git/deploy/docker-compose.yml up -d

# Or manual
docker run -d \
  --name arcline-gitea \
  --restart unless-stopped \
  -p 127.0.0.1:3000:3000 \
  -p 22:22 \
  -v /opt/arcline-gitea/data:/data \
  -e DOMAIN=git.arcline.it \
  -e ROOT_URL=https://git.arcline.it \
  -e SSH_DOMAIN=git.arcline.it \
  gitea/gitea:latest

# Nginx config (redirect + proxy)
sudo cp git/deploy/nginx-git-redirect.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

# Verify
curl -I https://git.arcline.it
ssh -T _gitea@git.arcline.it

uptime — Internal Monitor

Detail	Value
Method	Static binary or Docker
Binary	`/usr/local/bin/arcline-uptime`
Config	`/etc/arcline/uptime.yaml`
Database	`/var/lib/arcline/uptime.db`

# As a service
arcline-uptime start --config /etc/arcline/uptime.yaml

# Or via systemd
[Service]
ExecStart=/usr/local/bin/arcline-uptime start --config /etc/arcline/uptime.yaml

status — `status.arclineit.com`

Detail	Value
Method	Static HTML generation
Binary	`/usr/local/bin/arcline-status`
Config	`/etc/arcline/status.yaml` + `/etc/arcline/status.d/`
Output	`/var/www/status/` (served by Nginx)

# Generate status page
arcline-status build --config /etc/arcline/status.yaml --out /var/www/status/

# Watch mode (regenerate on config change)
arcline-status build --config /etc/arcline/status.yaml --out /var/www/status/ --watch

docs — `docs.arclineit.com`

Detail	Value
Method	Docker container (dynamic) or Nginx (static)
Container name	`arcline-docs`
Internal port	8080

# Dynamic variant (Go server with Markdown rendering)
docker run -d \
  --name arcline-docs \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  -v /opt/arcline-docs/.env:/app/.env:ro \
  <registry>/arcline-docs:latest

# Static variant (pre-built HTML via Dockerfile.static)
docker run -d \
  --name arcline-docs \
  --restart unless-stopped \
  -p 127.0.0.1:80:80 \
  <registry>/arcline-docs:latest-static

Database Management

Backup

# SQLite databases — simple file copy
cp /opt/arcline-<service>/data/<service>.db /backup/<service>-$(date +%Y%m%d).db

# Compress
gzip /backup/<service>-*.db

Restore

# Stop the service first
docker stop arcline-<service>
# or
systemctl stop arcline-<service>

# Restore database
cp /backup/<service>-<date>.db.gz /opt/arcline-<service>/data/
gunzip /opt/arcline-<service>/data/<service>-<date>.db.gz
mv /opt/arcline-<service>/data/<service>-<date>.db /opt/arcline-<service>/data/<service>.db
chown arcline:arcline /opt/arcline-<service>/data/<service>.db

# Restart
docker start arcline-<service>
# or
systemctl start arcline-<service>

Maintenance

# Vacuum SQLite database (reclaim space)
sqlite3 /opt/arcline-<service>/data/<service>.db "VACUUM;"

# Integrity check
sqlite3 /opt/arcline-<service>/data/<service>.db "PRAGMA integrity_check;"

Backup & Restore

What to Back Up

Asset	Location	Frequency
SQLite databases	`/opt/arcline-/data/.db`	Daily
Environment files	`/opt/arcline-*/.env`	On change
SSL certificates	`/etc/ssl/`	On renewal
Nginx configs	`/etc/nginx/`	On change
Systemd units	`/etc/systemd/system/arcline-*.service`	On change
Status config	`/etc/arcline/`	On change
Uptime config	`/etc/arcline/uptime.yaml`	On change

Backup Script

#!/bin/sh
# /usr/local/bin/arcline-backup

BACKUP_DIR="/backup/arcline/$(date +%Y-%m-%d)"
mkdir -p "$BACKUP_DIR"

# Databases
for db in /opt/arcline-*/data/*.db; do
    cp "$db" "$BACKUP_DIR/"
done

# Environment files (mask secrets)
for env in /opt/arcline-*/.env; do
    cp "$env" "$BACKUP_DIR/$(basename $(dirname $env)).env"
done

# Configs
cp -r /etc/arcline "$BACKUP_DIR/"

# Compress
tar czf "$BACKUP_DIR.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)"
rm -rf "$BACKUP_DIR"

# Sync to backup host
rsync -av "$BACKUP_DIR.tar.gz" backup-host:/backups/arcline/

Rollback Procedures

Docker Rollback

# 1. Check previous image tags
docker images <registry>/<project>

# 2. Deploy a specific tag instead of latest
docker run -d \
  --name <container>-rollback \
  <registry>/<project>:<previous-sha>

# 3. Verify health
curl http://127.0.0.1:<port>/health

# 4. Swap if healthy
docker stop <container>
docker rm <container>
docker rename <container>-rollback <container>

Native Binary Rollback

# 1. Keep previous binary versions
cp /usr/local/bin/<binary> /usr/local/bin/<binary>.bak

# 2. Restore previous version
cp /usr/local/bin/<binary>.bak /usr/local/bin/<binary>

# 3. Restart service
systemctl restart <service>
# or
rcctl restart <service>

Health Checks

Docker Health Check (in Dockerfile)

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1

Manual Health Check

# Web service
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:<port>/health

# Process check
pgrep -x <binary>

# Docker check
docker ps --filter "name=<container>" --filter "status=running"

Troubleshooting

Service Won't Start

# Check systemd logs
journalctl -u arcline-<service> -f

# Check Docker logs
docker logs arcline-<service>

# Check binary directly
/opt/arcline-<service>/<binary> 2>&1

# Common issues:
# - Missing .env file → ExecStart fails
# - Port conflict → change port in .env
# - Permission denied → check chown/chmod

Database Issues

# Check SQLite integrity
sqlite3 /opt/arcline-<service>/data/<service>.db "PRAGMA integrity_check;"
# Expected: "ok"

# Check disk space
df -h /opt/arcline-<service>/data/

# Check file permissions
ls -la /opt/arcline-<service>/data/<service>.db
# Expected: -rw-r----- arcline arcline

Connection Refused

# Check if service is listening
ss -tlnp | grep <port>

# Check Nginx is running
systemctl status nginx

# Check pfSense NAT rules
# Verify port forwarding from WAN:443 → host:443

Quick Recovery

# Universal restart sequence
docker stop arcline-<service> 2>/dev/null
docker rm arcline-<service> 2>/dev/null
docker pull <registry>/<project>:latest
docker run -d --restart unless-stopped \
  -p 127.0.0.1:<port>:<port> \
  -v /opt/arcline-<service>/.env:/app/.env:ro \
  <registry>/<project>:latest

17 KiB Raw Blame History