ArclineIT/docs
1
0
Fork 0
Code Issues 40 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
docs/DEPLOYMENT.md
Blake Ridgway 0cbcc962f7 DOCS-1: Init document work
2026-07-28 07:20:32 -05:00

17 KiB
Raw Permalink Blame History

Deployment Guide

From source to production — how every Arcline service is deployed, managed, and maintained.

Table of Contents

Overview

Every Arcline service deploys as a single static Go binary. The deployment strategy depends on the service type:

Type Method Examples
Web service Docker container portal, billing, docs
Native binary Direct install on host website (OpenBSD), uptime
CLI tool Install to $PATH check, audit, dns, migrate, vault

All services share the same core deployment flow:

  1. Build — Compile static Go binary with CGO_ENABLED=0
  2. Package — Containerize (Docker) or ship raw binary
  3. Deploy — Transfer to production host and start
  4. Verify — Health check endpoint or process monitoring

Production Environment

Host Specifications

Host Role OS Location
srv01 Primary web server OpenBSD Arcline datacenter
srv02 Application server Linux (Alpine) Arcline datacenter
srv03 Database & storage Linux (Alpine) Arcline datacenter

Network

  • All services listen on 127.0.0.1 (localhost) only
  • Nginx handles TLS termination and reverse proxy to local services
  • Public access is through pfSense port forwarding (443 → Nginx)

Runtime Dependencies

  • Docker (for containerized services): docker-ce
  • Nginx: nginx (OpenBSD: nginx package)
  • SQLite: No runtime dependency (pure Go, file-based)
  • ca-certificates: Required in all containers for TLS

Deployment Methods

Method 1: Docker Deployment (Web Services)

Used for: portal, billing, docs (dynamic), email (future)

# ── On production host ────────────────────────────────────────────────

# 1. Pull the latest image
docker login -u <user> -p <token> <registry>
docker pull <registry>/<project>:latest

# 2. Stop and remove existing container
docker stop <container-name> 2>/dev/null || true
docker rm   <container-name> 2>/dev/null || true

# 3. Start new container
docker run -d \
  --name <container-name> \
  --restart unless-stopped \
  -p 127.0.0.1:<host-port>:<container-port> \
  -v /opt/<service>/.env:/app/.env:ro \
  -v /opt/<service>/data:/app/data \
  <registry>/<project>:latest

# 4. Verify
docker ps | grep <container-name>
docker logs <container-name> --tail 20

Method 2: Native Binary (OpenBSD)

Used for: website

# ── On build machine ──────────────────────────────────────────────────

# Cross-compile
GOOS=openbsd GOARCH=amd64 go build -ldflags="-s -w" -o binary-openbsd-amd64 .

# Deploy
scp binary-openbsd-amd64 srv01:/usr/local/bin/<binary>
rsync -av --delete static/ srv01:/var/www/<domain>/static/

# ── On production host ────────────────────────────────────────────────

# Restart service
doas rcctl restart <service>

Method 3: Static Binary (CLI Tools)

Used for: uptime, status, check, audit, dns, migrate, vault

# ── On build machine ──────────────────────────────────────────────────

# Cross-compile for the target architecture
GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o binary-linux-amd64 .

# Deploy
scp binary-linux-amd64 srv02:/usr/local/bin/<binary>
ssh srv02 "chmod 0755 /usr/local/bin/<binary>"

# Verify
ssh srv02 "<binary> version"

Service Directory Layout

Containerized Services

/opt/arcline-<service>/
├── .env              # Environment variables (arcline:arcline, 0640)
└── data/             # Persistent data directory
    └── <service>.db  # SQLite database (created at runtime)

Native Services (OpenBSD)

/usr/local/bin/
├── arcline-web          # Go binary
└── rc.d/                # rc.d scripts (managed by rcctl)

/var/www/arclineit/
├── static/              # CSS, JS, images
├── templates/           # Go HTML templates
└── .env                 # Environment config

CLI Tools

/usr/local/bin/
├── arcline-uptime
├── arcline-status
├── arcline-check
├── arcline-audit
├── arcline-dns
├── arcline-migrate
├── arcline-vault
└── arcline-email

/etc/arcline/
├── uptime.yaml          # Uptime monitor config
├── status.yaml          # Status page config
├── status.d/            # Modular status config directory
└── email.toml           # Email server config

/var/lib/arcline/
├── uptime.db            # Uptime monitoring database
└── vault/               # Vault secrets storage

User & Permissions Model

System User

# Create the arcline user (done once per host)
addgroup -S arcline
adduser -S -G arcline -h /opt arcline

Directory Permissions

# Application directory
chown -R arcline:arcline /opt/arcline-<service>
chmod 0750 /opt/arcline-<service>

# Environment file (sensitive!)
chown arcline:arcline /opt/arcline-<service>/.env
chmod 0640 /opt/arcline-<service>/.env

# Database file (created by app, but ensure permissions)
chown arcline:arcline /opt/arcline-<service>/data/*.db
chmod 0640 /opt/arcline-<service>/data/*.db

Docker Container User

All Docker containers run as the arcline non-root user:

RUN addgroup -S arcline && adduser -S -G arcline arcline
USER arcline

Systemd Service Units

Template: arcline-<service>.service

[Unit]
Description=Arcline <Service Name> — <short description>
After=network.target

[Service]
Type=simple
User=arcline
Group=arcline
WorkingDirectory=/opt/arcline-<service>
EnvironmentFile=/opt/arcline-<service>/.env
ExecStart=/opt/arcline-<service>/<binary> [flags]
Restart=on-failure
RestartSec=5s

# Hardening
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/arcline-<service>

[Install]
WantedBy=multi-user.target

Service Management

# Install the unit file
sudo cp arcline-<service>.service /etc/systemd/system/
sudo systemctl daemon-reload

# Enable on boot
sudo systemctl enable arcline-<service>

# Start / stop / restart / status
sudo systemctl start   arcline-<service>
sudo systemctl stop    arcline-<service>
sudo systemctl restart arcline-<service>
sudo systemctl status  arcline-<service>

# View logs
sudo journalctl -u arcline-<service> -f

Hardening Options

Option Purpose
NoNewPrivileges=yes Prevent privilege escalation via suid binaries
PrivateTmp=yes Isolated /tmp for the service
ProtectSystem=strict Read-only /usr and /etc
ProtectHome=yes No access to /home, /root
ReadWritePaths=... Only allow writes to the data directory

Reference implementation: portal/deploy/arcline-portal.service

Nginx Reverse Proxy

Template: nginx-<service>.conf

# HTTP → HTTPS redirect
server {
    listen 80;
    server_name <domain>;
    return 301 https://$host$request_uri;
}

# HTTPS server
server {
    listen 443 ssl;
    server_name <domain>;

    ssl_certificate     /etc/ssl/<domain>/fullchain.pem;
    ssl_certificate_key /etc/ssl/<domain>/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # Security headers
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
    add_header X-Frame-Options DENY always;
    add_header X-Content-Type-Options nosniff always;

    location / {
        proxy_pass         http://127.0.0.1:<upstream-port>;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_read_timeout 30s;
    }
}

SSL Certificate Management

Certificates from Let's Encrypt (certbot) or Step CA (internal).

# External (Let's Encrypt)
certbot certonly --webroot -w /var/www/acme -d <domain>

# Internal (Step CA)
step certificate install /etc/step-ca/certs/intermediate_ca.crt

Reference implementation: portal/deploy/nginx-portal.conf

Deploy by Service

website — arcline.it

Detail Value
Method Native binary on OpenBSD
Binary /usr/local/bin/arcline-web
Static files /var/www/arclineit/static/
Service manager OpenBSD rcctl
Deploy command make deploy 
# Quick deploy
make cross
scp arcline-web-openbsd-amd64 srv01:/usr/local/bin/arcline-web
rsync -av --delete static/ srv01:/var/www/arclineit/static/
ssh srv01 "rcctl restart arcline-web"

billing — client.arcline.it

Detail Value
Method Docker container
Container name arcline-billing
Internal port 8082
Data volume /opt/arcline-billing/.env:/app/.env:ro
Database SQLite in container (ephemeral — backup on host) 
docker run -d \
  --name arcline-billing \
  --restart unless-stopped \
  -p 127.0.0.1:8082:8082 \
  -v /opt/arcline-billing/.env:/app/.env:ro \
  <registry>/arcline-billing:latest

portal — portal.arclineit.com

Detail Value
Method Docker container
Container name arcline-portal
Internal port 8082
Data volume /opt/arcline-portal/.env:/app/.env:ro
Database SQLite in container (ephemeral — backup on host) 
docker run -d \
  --name arcline-portal \
  --restart unless-stopped \
  -p 127.0.0.1:8082:8082 \
  -v /opt/arcline-portal/.env:/app/.env:ro \
  <registry>/arcline-portal:latest

git — git.arcline.it

Detail Value
Method Docker container
Container name arcline-gitea
Internal ports 3000 (web), 22 (SSH)
Data volume /opt/arcline-gitea/data:/data
Database SQLite (or PostgreSQL for multi-instance) 
# Deploy with docker-compose
docker compose -f git/deploy/docker-compose.yml up -d

# Or manual
docker run -d \
  --name arcline-gitea \
  --restart unless-stopped \
  -p 127.0.0.1:3000:3000 \
  -p 22:22 \
  -v /opt/arcline-gitea/data:/data \
  -e DOMAIN=git.arcline.it \
  -e ROOT_URL=https://git.arcline.it \
  -e SSH_DOMAIN=git.arcline.it \
  gitea/gitea:latest

# Nginx config (redirect + proxy)
sudo cp git/deploy/nginx-git-redirect.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

# Verify
curl -I https://git.arcline.it
ssh -T _gitea@git.arcline.it

uptime — Internal Monitor

Detail Value
Method Static binary or Docker
Binary /usr/local/bin/arcline-uptime
Config /etc/arcline/uptime.yaml
Database /var/lib/arcline/uptime.db 
# As a service
arcline-uptime start --config /etc/arcline/uptime.yaml

# Or via systemd
[Service]
ExecStart=/usr/local/bin/arcline-uptime start --config /etc/arcline/uptime.yaml

status — status.arclineit.com

Detail Value
Method Static HTML generation
Binary /usr/local/bin/arcline-status
Config /etc/arcline/status.yaml + /etc/arcline/status.d/
Output /var/www/status/ (served by Nginx) 
# Generate status page
arcline-status build --config /etc/arcline/status.yaml --out /var/www/status/

# Watch mode (regenerate on config change)
arcline-status build --config /etc/arcline/status.yaml --out /var/www/status/ --watch

docs — docs.arclineit.com

Detail Value
Method Docker container (dynamic) or Nginx (static)
Container name arcline-docs
Internal port 8080 
# Dynamic variant (Go server with Markdown rendering)
docker run -d \
  --name arcline-docs \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  -v /opt/arcline-docs/.env:/app/.env:ro \
  <registry>/arcline-docs:latest

# Static variant (pre-built HTML via Dockerfile.static)
docker run -d \
  --name arcline-docs \
  --restart unless-stopped \
  -p 127.0.0.1:80:80 \
  <registry>/arcline-docs:latest-static

Database Management

Backup

# SQLite databases — simple file copy
cp /opt/arcline-<service>/data/<service>.db /backup/<service>-$(date +%Y%m%d).db

# Compress
gzip /backup/<service>-*.db

Restore

# Stop the service first
docker stop arcline-<service>
# or
systemctl stop arcline-<service>

# Restore database
cp /backup/<service>-<date>.db.gz /opt/arcline-<service>/data/
gunzip /opt/arcline-<service>/data/<service>-<date>.db.gz
mv /opt/arcline-<service>/data/<service>-<date>.db /opt/arcline-<service>/data/<service>.db
chown arcline:arcline /opt/arcline-<service>/data/<service>.db

# Restart
docker start arcline-<service>
# or
systemctl start arcline-<service>

Maintenance

# Vacuum SQLite database (reclaim space)
sqlite3 /opt/arcline-<service>/data/<service>.db "VACUUM;"

# Integrity check
sqlite3 /opt/arcline-<service>/data/<service>.db "PRAGMA integrity_check;"

Backup & Restore

What to Back Up

Asset Location Frequency
SQLite databases /opt/arcline-*/data/*.db Daily
Environment files /opt/arcline-*/.env On change
SSL certificates /etc/ssl/ On renewal
Nginx configs /etc/nginx/ On change
Systemd units /etc/systemd/system/arcline-*.service On change
Status config /etc/arcline/ On change
Uptime config /etc/arcline/uptime.yaml On change

Backup Script

#!/bin/sh
# /usr/local/bin/arcline-backup

BACKUP_DIR="/backup/arcline/$(date +%Y-%m-%d)"
mkdir -p "$BACKUP_DIR"

# Databases
for db in /opt/arcline-*/data/*.db; do
    cp "$db" "$BACKUP_DIR/"
done

# Environment files (mask secrets)
for env in /opt/arcline-*/.env; do
    cp "$env" "$BACKUP_DIR/$(basename $(dirname $env)).env"
done

# Configs
cp -r /etc/arcline "$BACKUP_DIR/"

# Compress
tar czf "$BACKUP_DIR.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)"
rm -rf "$BACKUP_DIR"

# Sync to backup host
rsync -av "$BACKUP_DIR.tar.gz" backup-host:/backups/arcline/

Rollback Procedures

Docker Rollback

# 1. Check previous image tags
docker images <registry>/<project>

# 2. Deploy a specific tag instead of latest
docker run -d \
  --name <container>-rollback \
  <registry>/<project>:<previous-sha>

# 3. Verify health
curl http://127.0.0.1:<port>/health

# 4. Swap if healthy
docker stop <container>
docker rm <container>
docker rename <container>-rollback <container>

Native Binary Rollback

# 1. Keep previous binary versions
cp /usr/local/bin/<binary> /usr/local/bin/<binary>.bak

# 2. Restore previous version
cp /usr/local/bin/<binary>.bak /usr/local/bin/<binary>

# 3. Restart service
systemctl restart <service>
# or
rcctl restart <service>

Health Checks

Docker Health Check (in Dockerfile)

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1

Manual Health Check

# Web service
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:<port>/health

# Process check
pgrep -x <binary>

# Docker check
docker ps --filter "name=<container>" --filter "status=running"

Troubleshooting

Service Won't Start

# Check systemd logs
journalctl -u arcline-<service> -f

# Check Docker logs
docker logs arcline-<service>

# Check binary directly
/opt/arcline-<service>/<binary> 2>&1

# Common issues:
# - Missing .env file → ExecStart fails
# - Port conflict → change port in .env
# - Permission denied → check chown/chmod

Database Issues

# Check SQLite integrity
sqlite3 /opt/arcline-<service>/data/<service>.db "PRAGMA integrity_check;"
# Expected: "ok"

# Check disk space
df -h /opt/arcline-<service>/data/

# Check file permissions
ls -la /opt/arcline-<service>/data/<service>.db
# Expected: -rw-r----- arcline arcline

Connection Refused

# Check if service is listening
ss -tlnp | grep <port>

# Check Nginx is running
systemctl status nginx

# Check pfSense NAT rules
# Verify port forwarding from WAN:443 → host:443

Quick Recovery

# Universal restart sequence
docker stop arcline-<service> 2>/dev/null
docker rm arcline-<service> 2>/dev/null
docker pull <registry>/<project>:latest
docker run -d --restart unless-stopped \
  -p 127.0.0.1:<port>:<port> \
  -v /opt/arcline-<service>/.env:/app/.env:ro \
  <registry>/<project>:latest