stack/quickstart.md

# Quickstart

Homelab host: **`192.168.8.123`** (SER5, Ubuntu 24.04)  
Stack: **`/opt/stack`** · Media: **`/storage`** · User: **`tim`**

**Full project index (session recovery):** [specification.html](./specification.html) (human) · [specification.json](./specification.json) (agents).  
Regenerate both from one source: `./scripts/generate-specification.py`

---

## Fresh install / tech demo (max automation)

Almost everything lives in **`.env`** (see `.env.example`). **Every container deploys by default** (including OpenHands).

### Hardware profiles

| Profile | Hardware | Inference | Default model |
|---------|----------|-----------|----------------|
| **mini** | Beelink / SER5-class | **CPU** | Ternary-Bonsai-27B Q2_0 |
| **mid** | 8GB AMD GPU, 16GB RAM | **Vulkan** | Qwen3.5-9B Abliterated **Q4_K_M** (~5.6GB) |
| **gaming** | 16GB AMD GPU, 64GB RAM | **Vulkan** | Ternary-Bonsai-27B (+ downloads Qwen too) |

```bash
cp .env.example .env
# set STACK_HOST, LAN_SUBNET, usenet keys
# drop wireguard/wg0.conf  (+ # EndpointHost = hostname)

./scripts/apply-profile.sh mini    # or mid | gaming
./scripts/bootstrap-stack.sh
# → dirs, model download(s), build bonsai (cpu|vulkan), compose up ALL services
```

Switch later:
```bash
./scripts/apply-profile.sh mid
docker compose build bonsai && docker compose up -d
```

Demo login: **`tim` / `asdfasdf`**.  
Still manual once: Prowlarr↔apps, Jellyfin libraries, first Forgejo/Open WebUI admin.

---

## Ansible

From a machine with this repo and SSH key `~/.ssh/id_ed25519`:

```bash
cd ansible
ansible-galaxy collection install -r requirements.yml

ansible-playbook site.yml -K          # -K only if sudo still asks once; playbook installs NOPASSWD for tim
```

What it does: OS updates, tools (incl. Microsoft `edit`), Docker, AMD/Vulkan, UFW (LAN only), `/opt/stack` + `/storage`, `.env`, `arr-stack.service`, and **tim** as local admin (passwordless sudo, docker/video/render/…, owns stack + storage, permissive GPU nodes).

**Code on the server:** `/opt/stack` is a **git clone** of Forgejo `tim/stack` (`stack_deploy_method: git`).  
Day-to-day updates on ser5:

```bash
ssh tim@192.168.8.123
cd /opt/stack
git pull
docker compose up -d   # if compose/images changed
```

Ansible can also refresh code: `ansible-playbook site.yml -K --tags stack,deploy` (runs `git fetch` + checkout).  
Local-only (not in git): `.env`, `config/`, `models/**/*.gguf`, `wireguard/wg0.conf`.

**Before first full stack start** (on ser5) — same as profile bootstrap:

```bash
ssh tim@192.168.8.123
cd /opt/stack
git pull
# wireguard/wg0.conf must exist (no IPv6; Endpoint = IP not hostname; # EndpointHost = …)
./scripts/apply-profile.sh mini    # or mid | gaming
./scripts/bootstrap-stack.sh       # models + build + compose up ALL services
# or step-by-step:
#   ./scripts/download-models.sh
#   docker compose build bonsai && docker compose up -d
sudo systemctl enable --now arr-stack   # optional: start stack on boot
docker compose ps
```

**Compose GPU notes (mid / gaming)**
- Keep `devices`/`group_add` only in `docker-compose.override.yml` (not also in the main file).
- Use **numeric GIDs** for `group_add`, not names (`video`/`render` fail inside the image). Or omit `group_add` if `/dev/dri` is mode `0666`.
- **mini** uses CPU (`BONSAI_NGL=0`) — no GPU devices required.


Re-run playbook anytime: `ansible-playbook site.yml -K`  
Tags: `--tags common,docker,amd,firewall,stack`

---

## Remotes

LAN URLs (UFW allows **`192.168.8.0/24`**). Replace host if needed.

| Service | URL |
|---------|-----|
| **Landing page** | http://192.168.8.123/ |
| Cloudflare DDNS | (background) apex + subdomains → public IPv4 |
| **Forgejo (HTTPS)** | https://forgejo.devopshomelab.com/ |
| **Jellyfin (HTTPS)** | https://jellyfin.devopshomelab.com/ |
| **RustDesk (info)** | https://rustdesk.devopshomelab.com/ |
| RustDesk (client) | `rustdesk.devopshomelab.com` ports **21115–21119** |
| Jellyfin | http://192.168.8.123:8096 |
| qBittorrent | http://192.168.8.123:8080 |
| Prowlarr | http://192.168.8.123:9696 |
| Sonarr | http://192.168.8.123:8989 |
| Radarr | http://192.168.8.123:7878 |
| Bazarr | http://192.168.8.123:6767 |
| Lidarr | http://192.168.8.123:8686 |
| SABnzbd | http://192.168.8.123:8085 |
| Open WebUI | http://192.168.8.123:3000 |
| Bonsai API | http://192.168.8.123:8081/v1 |
| OpenHands | http://192.168.8.123:3001 |
| **Forgejo** | http://192.168.8.123:3002 |
| Forgejo Git SSH | `ssh://git@192.168.8.123:2222/user/repo.git` |

OpenHands is always started with the stack (GHCR images).

### Manual config (once)

**qBittorrent** — temp password in `docker logs qbittorrent`; user `admin`.  
Advanced → Network interface: `tun0`.  
Depends on **gluetun** healthy. If gluetun is unhealthy with `lookup … i/o timeout`, WireGuard is not passing traffic — usually a **stale Endpoint IP** in `wireguard/wg0.conf`.

**Auto-heal:** `scripts/gluetun-endpoint-watch.sh` + systemd timer every **5 minutes** (installed by Ansible). When gluetun is not healthy it: digs `WIREGUARD_ENDPOINT_HOST` → updates `Endpoint` in `wg0.conf` → recreates gluetun → recreates qbittorrent → tests exit IP.  
Manual run: `cd /opt/stack && ./scripts/gluetun-endpoint-watch.sh`  
Logs: `journalctl -u gluetun-endpoint-watch.service -n 50`  
Set host in `.env`: `WIREGUARD_ENDPOINT_HOST=las-183-wg.whiskergalaxy.com`

Default save path + categories (WebUI):

1. **Tools → Options → Downloads**  
   - Default Save Path: `/data/torrents`  
   - Keep “Keep incomplete torrents in” off (or a subfolder under `/data/torrents` if you prefer).
2. **Categories** (same Options → Downloads page, or right‑click empty area in the left “Categories” panel → **Add category**):

   | Category name | Save path |
   |---------------|-----------|
   | `movies` | `/data/torrents/movies` |
   | `tv` | `/data/torrents/tv` |
   | `music` | `/data/torrents/music` |

   For each: **Add category** → name exactly `movies` / `tv` / `music` (lowercase, matches Sonarr/Radarr/Lidarr) → set **Save path** to the path above → OK.

3. Sonarr/Radarr/Lidarr download-client “Category” fields must match those names so torrents land in the right folder for hardlinks.

**Sonarr / Radarr / Lidarr** — Download client qBittorrent: host **`gluetun`**, port **`8080`**, categories `tv` / `movies` / `music`.  
Root folders: `/data/media/tv`, `/data/media/movies`, `/data/media/music` (hardlinks on).

**Prowlarr** — Add indexers. Apps: Sonarr `http://sonarr:8989`, Radarr `http://radarr:7878`, Lidarr `http://lidarr:8686` + each API key (*Settings → General*).

### Secrets file

Path (not in git): **`/opt/stack/secrets/.env`**  
Template: `secrets/.env.example`  
Keys today: Usenet (NZBgeek / Newshosting / Tweaknews) + Cloudflare DDNS.

Sync from a laptop if you edit there:

```bash
scp secrets/.env tim@192.168.8.123:/opt/stack/secrets/.env
ssh tim@192.168.8.123 'chmod 600 /opt/stack/secrets/.env'
```

---

### After changing secrets (apply changes)

Always edit **`secrets/.env`** first, then run only what you changed:

| What you changed | Commands (on ser5, `cd /opt/stack`) |
|------------------|-------------------------------------|
| **Cloudflare** token / domain / subdomains / proxied | `./scripts/render-cloudflare-ddns-env.sh` then `docker compose up -d cloudflare-ddns --force-recreate` then `docker logs cloudflare-ddns --tail 30` |
| **Usenet servers** (Newshosting / Tweaknews user, pass, host, ports) | `./scripts/configure-usenet.sh` then SABnzbd → Config → Servers → **Test** |
| **NZBgeek API key** | Prowlarr → Indexers → NZBgeek → paste new `NZBKEEPKEY` (not automated after first setup) |
| **WireGuard** peer / keys | Edit `wireguard/wg0.conf` (and `# EndpointHost = …`) then `docker compose up -d gluetun --force-recreate` and/or `./scripts/gluetun-endpoint-watch.sh` |
| **Stack `.env`** (ports, AI profile, demo login, etc.) | Edit `.env` then `docker compose up -d` (rebuild bonsai if `BONSAI_BACKEND` changed: `docker compose build bonsai && docker compose up -d bonsai`) |
| **Everything secret-driven (demo reset)** | Update `secrets/.env` + `.env` then `./scripts/bootstrap-stack.sh` (re-renders Cloudflare, reconfigures Usenet when flags allow) |

**Cloudflare one-liner after secret edit:**

```bash
cd /opt/stack
# include rustdesk in CLOUDFLARESUBDOMAINS if missing
./scripts/render-cloudflare-ddns-env.sh
docker compose up -d cloudflare-ddns --force-recreate
docker logs cloudflare-ddns --tail 30
```

Needs: `CLOUDFLAREAPIKEY`, `CLOUDFLAREDOMAIN`, `CLOUDFLARESUBDOMAINS` (and optional `CLOUDFLARE_PROXIED`). Token: **Zone → DNS → Edit**. Default `PROXIED=false` (DNS-only; required for Let's Encrypt HTTP-01 and for VPN/RustDesk).

**HTTPS reverse proxy (Caddy + Let's Encrypt)**

Caddy (`landing` service) terminates TLS and routes:

| Hostname | Backend |
|----------|---------|
| `https://forgejo.devopshomelab.com` | `forgejo:3000` |
| `https://jellyfin.devopshomelab.com` | `jellyfin:8096` |
| `https://rustdesk.devopshomelab.com` | Info page only |
| `https://devopshomelab.com` | LAN resource map |

```bash
# 1) DNS A records must point here (DDNS) — grey cloud, not orange
# 2) Router: forward WAN TCP 80 + 443 → 192.168.8.123
# 3) UFW: allow 80/443 (playbook) + RustDesk 21115-21119 if remote clients
docker compose up -d landing forgejo jellyfin
docker logs landing --tail 40   # watch for certificate obtain
```

Forgejo `ROOT_URL` is `https://forgejo.devopshomelab.com/` (set in compose/`.env`). LAN still works at `http://192.168.8.123:3002/`.

**RustDesk server**

```bash
docker compose up -d rustdesk-hbbs rustdesk-hbbr
# Public key for clients:
docker exec rustdesk-hbbs cat /root/id_ed25519.pub
```

Client settings: ID server = `rustdesk.devopshomelab.com`, key = pubkey above.  
Open on router/firewall: **21115–21119/tcp** and **21116/udp**.

**Usenet one-liner after secret edit:**

```bash
cd /opt/stack
./scripts/configure-usenet.sh
# then open http://192.168.8.123:8085 → Config → Servers → Test both green
```

Needs: Newshosting / Tweaknews vars (see `secrets/.env.example`; aliases like `TWEAKNEWSLOGIN` / `HNEWSHOSTINGPASS` are accepted).

**NZBgeek after secret edit:** UI only — Prowlarr → Indexers → NZBgeek → API key = `NZBKEEPKEY`.

---

**Usenet first-time setup** (in addition to secrets apply above):

```bash
cd /opt/stack && git pull
mkdir -p /storage/usenet/{incomplete,complete/{movies,tv,music}}
docker compose up -d sabnzbd
# open http://192.168.8.123:8085 once if first-run wizard appears, then:
./scripts/configure-usenet.sh
```

1. **SABnzbd** (http://host:8085) — Config → Servers: Newshosting + Tweaknews SSL. Test both green.  
   Folders: incomplete `/data/usenet/incomplete`, complete `/data/usenet/complete` (categories `movies`/`tv`/`music`).
2. **Prowlarr** → Indexers → **NZBgeek** → API key from secrets (`NZBKEEPKEY`).  
   Download Clients → **SABnzbd** → host `sabnzbd`, port `8080`, API key from SABnzbd Config → General.  
   Sync download client to apps (or add SABnzbd in each *arr).
3. **Sonarr/Radarr/Lidarr** → Download Clients → SABnzbd: host `sabnzbd`, port `8080`, categories `tv` / `movies` / `music`.

Usenet traffic is **not** forced through gluetun (SSL to provider). Torrents still go via VPN.

**FlareSolverr** (Cloudflare / JS challenges for some indexers) — container `flaresolverr` on `arr-net`, no public port by default.  
Prowlarr → **Settings → Indexers** → FlareSolverr host: `http://flaresolverr:8191`.

**Bazarr** — Sonarr/Radarr URLs + API keys; paths `/data/media/tv`, `/data/media/movies`.

**Jellyfin** — Libraries: Movies `/data/media/movies`, Shows `/data/media/tv`, Music `/data/media/music`.

**Open WebUI** — Create admin on first visit. Model connection is pre-set to Bonsai (`http://bonsai:8080/v1`). Enable web search in chat if desired. RAG: Workspace → Knowledge (uploads) or files under `/workspace`.

**Bonsai** — No UI login. Smoke test: `curl -s http://192.168.8.123:8081/v1/models`. Needs GGUF in `models/bonsai/` and `docker compose build bonsai` once.

**OpenHands** — On the server (`/opt/stack`): `docker compose --profile coding up -d openhands`.  
UI: http://192.168.8.123:3001  

**LLM (required once)**  
Settings → Advanced: base URL `http://bonsai:8080/v1`, API key `sk-local-bonsai`, model `openai/<id from /v1/models>`.

**Connect to Forgejo (pick A and/or B)**

**A — Provider (browse / open repos in OpenHands)**  
1. Forgejo → avatar → **Settings → Applications** → generate a token (scopes: `read:repository`, `write:repository`, `read:user` — or full access on a trusted LAN).  
2. OpenHands → **Settings → Git / Integrations** (or **Secrets** / provider list):  
   - Provider: **Forgejo** or **Gitea** (same API)  
   - Base URL: `http://forgejo:3000` (Docker DNS on `arr-net`; use this from OpenHands)  
     From a browser on your PC you use `http://192.168.8.123:3002` — that is *not* what the OpenHands container should use.  
   - Token: paste the Forgejo token  
3. Start a conversation → choose repository → `tim/stack`.

**B — Workspace mount (most reliable for editing + git push)**  
Compose already mounts `${AI_WORKSPACE_DIR}` (default `/opt/stack/workspace`) into the agent sandbox as `/workspace`.  
On the **server**:

```bash
# HTTPS with token (one-time; token in URL — prefer SSH below)
# git clone http://tim:TOKEN@192.168.8.123:3002/tim/stack.git /opt/stack/workspace/stack

# SSH (key already on Forgejo for user tim)
git clone ssh://git@192.168.8.123:2222/tim/stack.git /opt/stack/workspace/stack
```

Then in OpenHands: work in `/workspace/stack` (or set the conversation workspace to that path).  
Agent can `git commit` / `git push` if the host clone has credentials (SSH agent or stored remote URL).

If OpenHands cannot reach `forgejo` by name, ensure both services are on `arr-net` (`docker compose up -d forgejo openhands`).

**Forgejo** — http://192.168.8.123:3002  
If connection refused: crash-loop from dual SSH; compose uses `START_SSH_SERVER=false`. Recreate: `docker compose up -d forgejo --force-recreate`.

---

## Locals

Bound to loopback on the host (not LAN-reachable as published):

| Service | URL / path |
|---------|------------|
| SearXNG | http://127.0.0.1:8888 |

Open WebUI talks to SearXNG on the Docker network (`http://searxng:8080`); you only need the loopback URL for direct browser checks on the server.

### Manual config

**SearXNG** — None for normal use. Settings: `/opt/stack/searxng/settings.yml`. Secret comes from `.env` (`SEARXNG_SECRET`).

**WireGuard** — File only: `/opt/stack/wireguard/wg0.conf` (not a web UI). Must be ready before gluetun is healthy.

**Host shell** — `ssh tim@192.168.8.123` · stack logs: `docker compose -f /opt/stack/docker-compose.yml logs -f` · unit: `systemctl status arr-stack`.