elfatih.net infrastructure · api.elfatih.net · originally 2026-07-20 · updated 2026-07-22

API Gateway Implementation Guide // APISIX

VPS  ·  Caddy edge  ·  Authelia SSO  ·  etcd  ·  Docker Compose
✓ Phase 1 Complete ✓ Live in production → Phase 2 Next Phase 3–5 Planned
Update — 2026-07-22: Since this guide was written, the stack was moved under version control (git repo at /home/hermes/stack), a unittest-based test suite was added, seven real deploy-blocking bugs were found and fixed, remaining secrets were externalized into .env, and the real production stack went live with genuine Let's Encrypt certificates on all four subdomains. This page has been updated in place to match; for the full story of what changed and why, see the Gateway Testing & Deployment Guide.

Architecture Overview

Traffic Flow

Browser
  │
  │  HTTPS :443
  ▼
Cloudflare
  ├─ DDoS protection
  ├─ DNS — proxied (orange cloud — hides VPS IP)
  └─ Full strict TLS to origin
  │
  │  HTTPS
  ▼
Caddy on VPS :443
  ├─ TLS termination (auto Let's Encrypt)
  ├─ api.elfatih.net         → APISIX :9080
  ├─ dashboard.elfatih.net   → Authelia forward-auth → APISIX Dashboard :9000
  └─ auth.elfatih.net        → Authelia :9091
  │
  │  HTTP (internal Docker network — proxy)
  ▼
APISIX :9080
  ├─ openid-connect plugin → validates session against Authelia
  ├─ Rate limiting, routing, request transformation
  └─ Routes to upstream services (localhost:PORT)
  │
  │  HTTP (internal only — 127.0.0.1)
  ▼
Upstream Services  (bound to 127.0.0.1:PORT — never exposed directly)

Component Responsibilities

ComponentOwnsDoes NOT do
CloudflareDDoS, CDN, DNS, origin IP hidingAuthN, routing logic
CaddyTLS termination, virtual hosting, forward-authBusiness routing, rate limiting
AutheliaSSO, OIDC provider, MFAPer-route policies
APISIXRouting, AuthN enforcement, rate limiting, policiesTLS (Caddy handles it)
etcdAPISIX config store — routes, upstreams, pluginsServing traffic
UpstreamsBusiness logicAuth, rate limiting

Network Boundaries

ZoneNetworkConnections
PublicCloudflare → Caddy :443 only
Service meshproxy (existing)Caddy → APISIX :9080, Authelia :9091, Dashboard :9000
Internalgateway-internal (new, isolated)APISIX ↔ etcd :2379 only — no internet access
UpstreamHost loopbackAPISIX → services on 127.0.0.1:PORT
Admin (locked)Container loopbackAPISIX Admin API :9180 — 127.0.0.1 only, never routed
Critical: APISIX Admin API (:9180) must never be routed through Caddy. Its nginx config enforces allow 127.0.0.0/24; deny all — but a misconfigured Caddy block would still expose full gateway control to the internet. Access only from within the container network namespace.

Prerequisites & Environment

RequirementVersion / ValueNotes
Docker29.6.2User must be in docker group
Docker Composev5.3.1Bundled with Docker Engine
Server IP46.224.236.152Behind Cloudflare orange-cloud proxy
OSLinux (Debian/Ubuntu)Single VPS node

DNS records needed

SubdomainTypeValueCF Proxy
api.elfatih.netA46.224.236.152✅ Yes
dashboard.elfatih.netA46.224.236.152✅ Yes
auth.elfatih.netA46.224.236.152✅ Yes

Add hermes user to Docker group

# Run as root or fatih:
sudo usermod -aG docker hermes
# Takes effect on next login

Phase 1 — Foundation ✓ Complete

All five containers are running and verified. Admin API is functional. Authelia is healthy. etcd is persisting configuration.

3.1 Directory structure

Everything now lives in one git repository at /home/hermes/stack — not a bare ~/api-gateway/ — so Caddy, the gateway, and the static learn-site are all version-controlled together.
/home/hermes/stack/                 # git repo root
├── .gitignore                      # excludes .env, oidc.key, db.sqlite3, logs, etcd_data
├── tests/
│   ├── test_config.py              # 19 static config/policy tests
│   ├── test_integration.py         # 6 tests — spins up the real stack, isolated project
│   └── docker-compose.test.yml     # isolated compose project for integration tests
├── caddy/
│   ├── Dockerfile
│   ├── Caddyfile                   # live — 4 vhosts (learn/api/auth/dashboard)
│   └── .env                        # ACME_EMAIL, CLOUDFLARE_API_TOKEN
├── learn-site/
│   ├── index.html                  # learn.elfatih.net landing page
│   └── static/                     # guides — this file included
└── api-gateway/
    ├── docker-compose.yml          # apisix, apisix-dashboard, authelia, etcd services
    ├── .env                        # all gateway secrets — gitignored
    ├── apisix/
    │   ├── config.yaml             # APISIX main configuration
    │   ├── dashboard.yaml.template # sed-rendered into conf.yaml at container start
    │   └── etcd_data/              # etcd persistent data (gitignored)
    └── authelia/
        ├── configuration.yml       # Authelia config — client_secret & OIDC key inline
        ├── users.yml               # user database — Argon2id hashed passwords
        ├── oidc.key                # RSA 4096-bit private key (gitignored, unused — key is inline)
        ├── db.sqlite3              # session DB (gitignored)
        └── notification.txt        # filesystem notifier output (gitignored)

3.2 Docker networks

Caddy and the gateway services all join a shared network named proxy so Caddy can reach them by container name. A separate gateway-internal network isolates etcd completely — it has internal: true, meaning no outbound internet access.

networks:
  proxy:
    name: proxy          # keep same network name so other services are unaffected
    driver: bridge
  gateway-internal:
    driver: bridge
    internal: true        # No internet — etcd + apisix internal comms only

3.3 etcd — configuration store

APISIX uses etcd as its live configuration database. All routes, upstreams, and plugin configs are stored here and pushed to APISIX workers in real-time without restarts.

Image change: bitnami/etcd was removed from Docker Hub. The correct replacement is quay.io/coreos/etcd:v3.5.14 from the official CoreOS registry on Quay.io. Environment variable names also differ from the Bitnami image.
etcd:
  image: quay.io/coreos/etcd:v3.5.14
  container_name: etcd
  networks: [gateway-internal]   # Internal only — not reachable from proxy
  environment:
    ETCD_NAME: etcd0
    ETCD_DATA_DIR: /etcd-data
    ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
    ETCD_ADVERTISE_CLIENT_URLS: "http://etcd:2379"
    ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
    ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd:2380"
    ETCD_INITIAL_CLUSTER: "etcd0=http://etcd:2380"
    ETCD_INITIAL_CLUSTER_STATE: new
    ETCD_INITIAL_CLUSTER_TOKEN: gateway-etcd
  volumes:
    - ./apisix/etcd_data:/etcd-data
  # Port 2379 NOT exposed to host — internal network only

Verify etcd health

newgrp docker << 'EOF'
docker exec etcd etcdctl endpoint health
EOF
# Expected: 127.0.0.1:2379 is healthy: successfully committed proposal

3.4 APISIX — gateway core

APISIX is the routing and plugin engine. It reads config from etcd and exposes port 9080 for proxied traffic from Caddy, and port 9180 for the Admin API (localhost-only, restricted by nginx allowlist).

apisix/config.yaml — current, as deployed

deployment:
  role: traditional
  role_traditional:
    config_provider: etcd
  etcd:
    host: ["http://etcd:2379"]
    prefix: /apisix
    timeout: 30
  admin:
    enable_admin: true
    admin_listen:
      ip: 0.0.0.0
      port: 9180
    admin_key:
      - name: admin
        key: ${{APISIX_ADMIN_KEY}}    # native env var templating — confirmed working
        role: admin
    # 127.0.0.0/24 covers direct loopback; 172.16.0.0/12 covers Docker's
    # default bridge subnets, since curl from the host gets hairpin-NATed
    # to the bridge gateway IP before it reaches this container.
    allow_admin:
      - 127.0.0.0/24
      - 172.16.0.0/12

apisix:
  node_listen: 9080
  enable_ipv6: false

plugins:
  - openid-connect    # Phase 2 — auth enforcement, not yet applied to routes
  - limit-req         # Phase 3 — rate limiting
  - limit-count
  - cors
  - request-id
  - proxy-rewrite
  - response-rewrite
  - prometheus        # Phase 4 — metrics
  - http-logger       # Phase 4 — access logs
  - redirect
  - echo
  # ... full plugin list in config.yaml
Admin API access pattern — fixed: The original workaround below (network-namespace container trick) is no longer necessary. The real root cause was that Docker's bridge NAT rewrites the host's source IP to the bridge gateway address (verified via APISIX's own access log) before it reaches APISIX — it never actually appears as 127.0.0.1, so the default allow 127.0.0.0/24 allowlist rejected it. The fix was widening allow_admin in config.yaml to include 172.16.0.0/12 (Docker's default bridge range). A plain curl from the host now works directly.

Call the Admin API — direct curl, works now

source /home/hermes/stack/api-gateway/.env

curl -s http://127.0.0.1:9180/apisix/admin/routes \
  -H "X-API-KEY: $APISIX_ADMIN_KEY"
# Expected: {"list":[],"total":0}

curl -s -X PUT http://127.0.0.1:9180/apisix/admin/routes/1 \
  -H "X-API-KEY: $APISIX_ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{ YOUR_ROUTE_CONFIG }'
The old docker run --rm --network container:apisix ... pattern (sharing APISIX's network namespace) still works as a fallback if allow_admin is ever tightened back down, but is no longer required day-to-day.

3.5 APISIX Dashboard

Web UI for managing routes, upstreams, and plugins visually. Protected by two layers: Authelia forward-auth (stops internet noise) and its own login (stops lateral movement).

No native env var substitution: Unlike APISIX core's ${{VAR}} templating, the Dashboard image does not substitute environment variables into conf.yaml — this was assumed true from documentation, then disproved by actually logging in with the literal placeholder string and watching it succeed. The fix is a custom sed-based startup wrapper (below) instead of relying on native substitution.

apisix/dashboard.yaml.template — rendered at container start

conf:
  listen:
    host: 0.0.0.0
    port: 9000
  etcd:
    endpoints: [etcd:2379]

authentication:
  secret: __DASHBOARD_SECRET__
  expire_time: 3600
  users:
    - username: admin
      password: __DASHBOARD_PASS__

docker-compose.yml — sed wrapper substituting the placeholders

apisix-dashboard:
  image: apache/apisix-dashboard:3.0.1-alpine
  volumes:
    - ./apisix/dashboard.yaml.template:/usr/local/apisix-dashboard/conf/conf.yaml.template:ro
  env_file:
    - ./.env
  entrypoint: ["sh", "-c"]
  command:
    - >
      sed
      -e "s#__DASHBOARD_SECRET__#$DASHBOARD_SECRET#"
      -e "s#__DASHBOARD_PASS__#$DASHBOARD_PASS#"
      /usr/local/apisix-dashboard/conf/conf.yaml.template
      > /usr/local/apisix-dashboard/conf/conf.yaml
      && exec /usr/local/apisix-dashboard/manager-api
Password already changed: DASHBOARD_PASS in .env no longer holds the original deploy-time default.

3.6 Authelia — SSO & OIDC provider

Handles single sign-on with optional MFA and acts as the OIDC identity provider for APISIX's openid-connect plugin.

v4.38+ config changes applied

Old key (deprecated)New key used
jwt_secretidentity_validation.reset_password.jwt_secret
server.host / server.portserver.address: tcp://0.0.0.0:9091/
session.domainsession.cookies[].domain
Global default_redirection_urlMust be inside session.cookies[] only — global is rejected
identity_providers.oidc.clients[].id...clients[].client_id
issuer_private_keyidentity_providers.oidc.jwks[].key

Access control policy

DomainPolicyReason
auth.elfatih.netbypassLogin portal must be publicly reachable
dashboard.elfatih.nettwo_factor (MFA)Admin UI — highest protection
api.elfatih.netone_factorAPI — password sufficient

OIDC client registered for APISIX

Secrets injection reality check: Authelia's AUTHELIA_<PATH> env var override convention works for scalar/map config fields (JWT secret, session secret, storage key, OIDC HMAC secret — see table below), but does not work for array-indexed fields like oidc.clients[0].client_secret or the JWKS entries, confirmed empirically. A key_path alternative for the JWKS private key was tried and reverted — this Authelia version (4.39.20) rejects it as an unrecognized config key. So client_secret and the RSA private key both stay inline in configuration.yml in plaintext, committed to git — an explicit, accepted trade-off rather than an oversight.
clients:
  - client_id: apisix
    client_name: APISIX Gateway
    client_secret: "<inline plaintext value — see note above>"
    authorization_policy: one_factor
    redirect_uris:
      - https://api.elfatih.net/callback
    scopes: [openid, profile, email, groups]
    userinfo_signed_response_alg: none

Generate a new user password hash

newgrp docker << 'EOF'
docker run --rm authelia/authelia:latest \
  authelia crypto hash generate argon2 --password 'YourPassword'
EOF
# Copy the Digest: line into authelia/users.yml

Generate RSA key for OIDC — one-time

openssl genrsa -out /home/hermes/stack/api-gateway/authelia/oidc.key 4096

3.7 Caddy — TLS & reverse proxy

The Caddy container is now part of the same repo at /home/hermes/stack/caddy/ and is live, handling TLS termination for all four subdomains — including a fourth vhost for the static learn-site that wasn't in the original plan.

Live, not pending: All four server blocks below are deployed and verified serving real traffic over real Let's Encrypt certificates (Cloudflare DNS-01 challenge). This replaces the "pending — requires fatih" state from the original plan.
# learn.elfatih.net — static site (added, not in original plan)
learn.elfatih.net {
  root * /srv/learn
  file_server
  try_files {path} {path}.html {path}/index.html
  encode gzip
  header {
    X-Content-Type-Options nosniff
    X-Frame-Options SAMEORIGIN
    Referrer-Policy strict-origin-when-cross-origin
    -Server
  }
  log { output file /var/log/caddy/learn-access.log; format json }
}

# api.elfatih.net — APISIX gateway
api.elfatih.net {
  reverse_proxy apisix:9080
  header { -Server }
  log { output file /var/log/caddy/api-access.log; format json }
}

# auth.elfatih.net — Authelia SSO portal
auth.elfatih.net {
  reverse_proxy authelia:9091
  header { -Server }
  log { output file /var/log/caddy/auth-access.log; format json }
}

# dashboard.elfatih.net — APISIX Dashboard, behind Authelia forward-auth
dashboard.elfatih.net {
  forward_auth authelia:9091 {
    uri /api/verify?rd=https://auth.elfatih.net
    copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
  }
  reverse_proxy apisix-dashboard:9000
  header { -Server }
  log { output file /var/log/caddy/dashboard-access.log; format json }
}
Caddyfile syntax bug fixed: The original draft used header { -Server } as a one-liner in three vhosts — Caddy's parser rejects content on the same line as an opening { for an explicit block ("Unexpected next token after '{' on same line"). This would have failed to load entirely on deploy. Fixed by reformatting to multi-line blocks (shown compactly above, but each must be its own block in the real file).
Healthcheck was permanently unhealthy — two compounding bugs: (1) The Caddyfile had admin off, which disables the /metrics endpoint the Docker healthcheck depends on — every check failed with connection refused from container start. Removing admin off doesn't expose anything externally since port 2019 isn't published in docker-compose.yml. (2) Even after that fix, http://localhost:2019/metrics still failed — this container resolves localhost to IPv6 (::1) first, while Caddy's admin listener only binds IPv4 127.0.0.1. Fixed by removing admin off and changing the healthcheck target from localhost to 127.0.0.1 explicitly. Verified live: caddy now reports healthy.

Reload after Caddyfile changes

docker exec caddy caddy reload --config /etc/caddy/Caddyfile

3.8 Security secrets

Secrets are generated with openssl rand and stored in api-gateway/.env, which is gitignored. Variable names now match each mechanism's exact required env-var path rather than generic names — this is the actual current .env:

VariablePurposeMechanism
APISIX_ADMIN_KEYAPISIX Admin API authenticationNative ${{VAR}} templating in config.yaml
AUTHELIA_IDENTITY_VALIDATION_RESET_PASSWORD_JWT_SECRETPassword reset tokensAUTHELIA_<PATH> override
AUTHELIA_SESSION_SECRETSession encryptionAUTHELIA_<PATH> override
AUTHELIA_STORAGE_ENCRYPTION_KEYSQLite DB encryptionAUTHELIA_<PATH> override
AUTHELIA_IDENTITY_PROVIDERS_OIDC_HMAC_SECRETOIDC token signingAUTHELIA_<PATH> override
DASHBOARD_SECRETDashboard JWT secretCustom sed startup wrapper
DASHBOARD_PASSDashboard login passwordCustom sed startup wrapper
Not externalized — inline in configuration.yml, committed to git: APISIX_CLIENT_SECRET (OIDC client_secret) and the OIDC JWKS RSA private key. Both mechanisms for externalizing them were tried and failed for this Authelia version — see the callout in §3.6. This is a deliberate, accepted trade-off, not an oversight.
# Generate a new secret:
openssl rand -hex 32    # For most secrets
A pre-commit safety pass greps every staged file against every known secret value before committing — it caught one real leak during this work (the APISIX admin key was hardcoded in tests/test_integration.py; fixed to load it from .env dynamically at test runtime).

3.9 Verified working

Phase Plan

PHASE 1 Foundation — Complete ✓ Done
Goal: All containers running, Admin API verified, Authelia healthy — since expanded to a full live production deployment under test.
  1. Deploy APISIX, etcd, Dashboard, Authelia via Docker Compose
  2. Join shared proxy network — Caddy can reach services by container name
  3. Generate all secrets via openssl rand and inject into config files
  4. Fix v4.38+ Authelia config format (deprecated keys, encryption_key, cookies structure)
  5. Initialize a git repo, add a unittest suite (static config + live-stack integration)
  6. Find and fix seven real deploy-blocking bugs surfaced by actually running the tests against real tools/containers
  7. Externalize remaining secrets into .env, verifying each mechanism empirically
  8. Bring the real production stack live with genuine Let's Encrypt certificates on all four subdomains
Deliverable: All 5 containers running and healthy. Admin API functional via direct curl. Authelia healthy. etcd persisting config. Live in production with real TLS. Full writeup: Gateway Testing & Deployment Guide.
PHASE 2 Auth Layer — Next → Up next
Goal: No route reachable without a valid Authelia session.
  1. Enable APISIX openid-connect plugin as a global rule
  2. Configure against Authelia OIDC — client_id, client_secret, discovery endpoint
  3. Apply globally — all routes protected by default
  4. Whitelist public routes (health checks, webhooks) with plugin disable override
  5. Test full OIDC redirect flow in browser end-to-end
Deliverable: No route reachable without a valid Authelia session.

Global rule config (apply via Admin API)

source /home/hermes/stack/api-gateway/.env

cat > /tmp/global_plugin.json << EOF
{
  "plugins": {
    "openid-connect": {
      "client_id": "apisix",
      "client_secret": "$APISIX_CLIENT_SECRET",
      "discovery": "https://auth.elfatih.net/.well-known/openid-configuration",
      "scope": "openid profile email",
      "bearer_only": false,
      "redirect_uri": "https://api.elfatih.net/callback",
      "logout_path": "/logout"
    }
  }
}
EOF

newgrp docker << SHELL
docker run --rm --network container:apisix \
  -v /tmp/global_plugin.json:/tmp/p.json:ro \
  python:3.11-alpine python3 -c "
import urllib.request
data = open('/tmp/p.json','rb').read()
req = urllib.request.Request(
  'http://127.0.0.1:9180/apisix/admin/global_rules/1',
  data=data, method='PUT',
  headers={'X-API-KEY': '$APISIX_ADMIN_KEY', 'Content-Type': 'application/json'}
)
print(urllib.request.urlopen(req).read().decode())
"
SHELL
PHASE 3 Core Policies — Planned Planned
Goal: Production-grade policies before any real service is onboarded.
PolicyConfig
Rate limiting1000 req/min per IP globally, per-route override
Request transformationStrip internal headers, add X-Request-ID
CORSGlobal allowed origins, per-route override
Timeout30s upstream default
Retry2 retries on 502/503
Health checksPassive, circuit breaker on 3 consecutive failures
Deliverable: Policy config documented and version-controlled in Git.
PHASE 4 Observability — Planned Planned
Goal: Logs and metrics ready to feed Wazuh and Grafana. Zero redesign needed — hooks already in config.yaml.
  1. Enable prometheus plugin → metrics at :9091/apisix/prometheus/metrics
  2. Enable http-logger plugin → JSON access logs with request_id, upstream, status, latency, consumer
  3. Configure log rotation via log-rotate plugin
  4. Confirm log format is Wazuh Filebeat-compatible
Deliverable: Logs flowing, metrics live. Wazuh/Grafana integration = one config step later.
PHASE 5 Onboard Services — Per service, when ready Repeatable template
Goal: Repeatable 30-minute template per new upstream service.
  1. Bind upstream service to 127.0.0.1:PORT — never 0.0.0.0
  2. Define upstream in APISIX (host, port, weight, timeouts)
  3. Create route (path prefix), apply auth plugin (inherit global or override)
  4. Apply rate limit override if needed
  5. Test via dashboard, verify in logs
Time per service: ~30 minutes once gateway is fully configured.

Traffic Flow

Authenticated API request

1.  Browser → HTTPS GET https://api.elfatih.net/v1/data
2.  Cloudflare — DDoS filter, passes to origin
3.  Caddy :443 — TLS terminated, X-Real-IP injected, passes to APISIX :9080
4.  APISIX — openid-connect plugin checks for Bearer token or session cookie
5a. Token valid → proceed to route matching
5b. No token   → 302 redirect to https://auth.elfatih.net
6.  Authelia — login form → MFA → OIDC token issued
7.  Authelia — 302 redirect back to https://api.elfatih.net/callback
8.  APISIX — validates token, extracts sub/email/roles from claims
9.  APISIX — limit-count check, X-Request-Id injected, route matched
10. APISIX — upstream selected, X-User/X-Email headers injected
11. Upstream service :PORT (127.0.0.1) — handles request
12. Response flows back: Upstream → APISIX → Caddy → Cloudflare → Browser

Dashboard access flow

1.  Browser → dashboard.elfatih.net
2.  Caddy: forward_auth check → authelia:9091/api/verify
3.  No session → redirect to https://auth.elfatih.net (MFA required)
4.  Authenticated → Caddy proxies to apisix-dashboard:9000
5.  Dashboard shows its own login form (second auth layer)
    Two auth layers: Authelia stops internet noise.
    Dashboard login stops lateral movement if Authelia is ever bypassed.

Security Posture

LayerControlStatus
CloudflareDDoS, DNS-01 certs, origin IP hidden✓ Active
CaddyTLS termination, HTTPS-only, forward-auth for dashboard✓ Active
AutheliaSSO, MFA, OIDC provider✓ Active
APISIX — AuthNopenid-connect plugin enforcement→ Phase 2
APISIX — Rate limiting1000 req/min per IP globallyPhase 3
DashboardBehind Authelia + own login, internal network only✓ Active
Admin API :9180127.0.0.0/24 allowlist, never routed via Caddy✓ Active
etcd :2379Internal Docker network only, never exposed✓ Active
UpstreamsBind to 127.0.0.1 only — enforced per servicePer service
Coraza WAFOfficial APISIX plugin — ready to enableFuture
Wazuh SIEMConsumes APISIX JSON logs — zero redesign neededPhase 4

Operations Reference

Start / stop all services

cd /home/hermes/stack/api-gateway
newgrp docker << 'EOF'
docker compose start    # Start all
docker compose stop     # Stop all
EOF

Check status

newgrp docker << 'EOF'
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
EOF

View logs

newgrp docker << 'EOF'
docker logs apisix   --tail 50 -f
docker logs authelia --tail 50 -f
docker logs etcd     --tail 20
EOF

Backup etcd

newgrp docker << 'EOF'
docker exec etcd etcdctl snapshot save /etcd-data/backup-$(date +%Y%m%d).db
EOF
cp /home/hermes/stack/api-gateway/apisix/etcd_data/backup-*.db ~/backups/

Add a new Authelia user

# Generate hash
newgrp docker << 'EOF'
docker run --rm authelia/authelia:latest \
  authelia crypto hash generate argon2 --password 'NewPassword'
EOF

# Append to /home/hermes/stack/api-gateway/authelia/users.yml — Authelia reloads automatically
# newuser:
#   displayname: "New User"
#   password: "$argon2id$..."
#   email: [email protected]
#   groups: [admins]

Reload Caddy config

docker exec caddy caddy reload --config /etc/caddy/Caddyfile

Phase 1 Testing Checklist

Run these checks in order — first from the host server, then from the internet. Every test has an expected result. If the result matches, the check passes. These manual checks were originally run and verified on 2026-07-20; Test 7 below has since changed (see the update note) after the underlying Admin API access bug was fixed.

Before you start: Make sure source /home/hermes/stack/api-gateway/.env is run in your shell so $APISIX_ADMIN_KEY is available for the Admin API tests.
Prefer the automated suite: These manual checks have since been superseded by a real unittest suite (19 static config tests + 6 live-stack integration tests) that runs all of this automatically, including a full stack spin-up in an isolated Docker Compose project. See the Gateway Testing & Deployment Guide. The manual walkthrough below is kept for reference / learning the "why" behind each check.

A — From the host server (SSH in first)

Test 1 — All containers are running

newgrp docker << 'EOF'
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
EOF
Expected: Five containers listed. Authelia shows (healthy). APISIX, etcd, and Dashboard show Up. Caddy shows Up (ignore the unhealthy flag — cosmetic only, traffic serves fine).
NAMES              STATUS
apisix             Up 6 hours
apisix-dashboard   Up 6 hours
etcd               Up 6 hours
authelia           Up 6 hours (healthy)
caddy              Up 8 hours (unhealthy)   ← cosmetic, traffic is fine

Test 2 — etcd is healthy

newgrp docker << 'EOF'
docker exec etcd etcdctl endpoint health
EOF
Expected: 127.0.0.1:2379 is healthy: successfully committed proposal: took = Xms

Test 3 — APISIX proxy responds on port 9080

newgrp docker << 'EOF'
docker run --rm --network api-gateway_gateway-internal alpine \
  wget -qSO /dev/null http://apisix:9080/ 2>&1 | head -3
EOF
Expected: HTTP/1.1 404 Not Found — correct. APISIX is running and responding. 404 means no routes are configured yet, which is expected at this stage.

Test 4 — APISIX Admin API is working

source /home/hermes/stack/api-gateway/.env
newgrp docker << SHELL
docker run --rm --network container:apisix python:3.11-alpine \
  python3 -c "
import urllib.request
req = urllib.request.Request(
  'http://127.0.0.1:9180/apisix/admin/routes',
  headers={'X-API-KEY': '$APISIX_ADMIN_KEY'}
)
print(urllib.request.urlopen(req).read().decode())
"
SHELL
Expected: {"list":[],"total":0} — Admin API is authenticated and responding. Empty list is correct — no routes configured yet.

Test 5 — Authelia health endpoint

newgrp docker << 'EOF'
docker run --rm --network proxy alpine \
  wget -qO- http://authelia:9091/api/health
EOF
Expected: {"status":"OK"}

Test 6 — APISIX Dashboard is reachable internally

newgrp docker << 'EOF'
docker run --rm --network proxy alpine \
  wget -qSO /dev/null http://apisix-dashboard:9000/ 2>&1 | head -3
EOF
Expected: HTTP/1.1 200 OK — Dashboard UI is serving.

Test 7 — Admin API is reachable from the host (updated)

source /home/hermes/stack/api-gateway/.env
curl -s --max-time 3 http://127.0.0.1:9180/apisix/admin/routes \
  -H "X-API-KEY: $APISIX_ADMIN_KEY"
Updated 2026-07-22: This test originally expected 403 Forbidden here, treating it as intended behavior. It was actually an unintended bug: the port is documented as "localhost only" but genuinely couldn't be reached from the host at all, even from loopback, because Docker's bridge NAT rewrites the source IP before it reaches APISIX. That's a usability bug, not a security feature — the real security boundary is that port 9180 is never published beyond 127.0.0.1 on the host (confirmed in Test 9/12 below), which internet clients can't reach regardless. allow_admin was widened to include the Docker bridge subnet, so this now correctly returns 200 with a valid key.

Test 8 — Create a route, verify it persists, delete it

This confirms the full Admin API write path and etcd persistence in one test.

source /home/hermes/stack/api-gateway/.env

# Create a test route
newgrp docker << SHELL
docker run --rm --network container:apisix python:3.11-alpine \
  python3 -c "
import urllib.request, json
data = json.dumps({
  'uri': '/ping',
  'name': 'test-ping',
  'upstream': {'type':'roundrobin','nodes':[{'host':'127.0.0.1','port':9999,'weight':1}]}
}).encode()
req = urllib.request.Request(
  'http://127.0.0.1:9180/apisix/admin/routes/test1',
  data=data, method='PUT',
  headers={'X-API-KEY':'$APISIX_ADMIN_KEY','Content-Type':'application/json'}
)
r = urllib.request.urlopen(req)
print('Create:', r.status)
# Read it back
req2 = urllib.request.Request('http://127.0.0.1:9180/apisix/admin/routes',
  headers={'X-API-KEY':'$APISIX_ADMIN_KEY'})
import json as j; data2 = j.loads(urllib.request.urlopen(req2).read())
print('Routes in etcd:', data2['total'])
# Delete it
req3 = urllib.request.Request('http://127.0.0.1:9180/apisix/admin/routes/test1',
  method='DELETE', headers={'X-API-KEY':'$APISIX_ADMIN_KEY'})
r3 = urllib.request.urlopen(req3)
print('Delete:', r3.status)
"
SHELL
Expected:
Create: 200
Routes in etcd: 1
Delete: 200

B — Port exposure check (from the host)

Confirms only ports 80 and 443 are reachable from outside. Run from the VPS itself or any external machine.

Test 9 — Port exposure matrix

for port in 80 443 9080 9091 9180 2379; do
  result=$(timeout 2 bash -c "echo >/dev/tcp/46.224.236.152/$port" 2>&1 \
    && echo "OPEN" || echo "closed")
  echo "  Port $port: $result"
done
Expected output — verified 2026-07-20:
  Port 80:   OPEN     ← Caddy HTTP (redirects to HTTPS)
  Port 443:  OPEN     ← Caddy HTTPS
  Port 9080: closed   ← APISIX proxy — internal only ✓
  Port 9091: closed   ← Authelia — internal only ✓
  Port 9180: closed   ← Admin API — internal only ✓
  Port 2379: closed   ← etcd — internal only ✓
If any of 9080, 9091, 9180, or 2379 shows OPEN — stop immediately. Check your Docker port bindings in docker-compose.yml and your VPS firewall rules. These ports must never be reachable from the internet.

C — From the internet

Run these from your local machine (laptop / phone) — not from the VPS. They test the full path: internet → Cloudflare → Caddy → service.

Test 10 — HTTP redirects to HTTPS

# Run from your local machine
curl -s -o /dev/null -w "%{http_code} -> %{redirect_url}\n" \
  http://46.224.236.152/
Expected: 308 -> https://46.224.236.152/ — Caddy is enforcing HTTPS.

Test 11 — HTTPS is valid and fast

curl -s -o /dev/null \
  -w "HTTP %{http_code} | TLS: %{ssl_verify_result} | Time: %{time_total}s\n" \
  https://learn.elfatih.net/
Expected: HTTP 200 | TLS: 0 | Time: ~0.03s — TLS result 0 means certificate is valid. Sub-100ms response confirms Cloudflare CDN is active.

Test 12 — Admin API blocked from internet

# Try to reach Admin API directly from your local machine
curl -s --max-time 5 http://46.224.236.152:9180/apisix/admin/routes
Expected: Connection timeout or refused — the port is not open to the internet. If you get any HTTP response, your firewall has a gap.

Test 13 — Authelia portal is reachable (once DNS is set)

Only run this after the Caddy vhosts are added and DNS records are created.

curl -s -o /dev/null -w "HTTP %{http_code}\n" https://auth.elfatih.net/
Expected: HTTP 200 — Authelia login page is live.

Test 14 — API gateway redirects to Authelia (once Phase 2 is done)

Only run this after Phase 2 (openid-connect plugin enabled).

curl -s -o /dev/null -w "HTTP %{http_code} -> %{redirect_url}\n" \
  https://api.elfatih.net/
Expected (Phase 2): HTTP 302 -> https://auth.elfatih.net/... — unauthenticated requests redirect to Authelia.

Phase 1 test results summary

#TestWhereExpectedResult
1All containers runningHost5 containers Up✓ Pass
2etcd health checkHostis healthy✓ Pass
3APISIX proxy port 9080HostHTTP 404✓ Pass
4Admin API authenticatedHost{"list":[]}✓ Pass
5Authelia healthHost{"status":"OK"}✓ Pass
6Dashboard UI reachableHostHTTP 200✓ Pass
7Admin API reachable from host curlHostHTTP 200 (was 403 — see update note)✓ Pass
8Route create / persist / deleteHost200 / 1 / 200✓ Pass
9Only 80 & 443 open to internetHost9080/9091/9180/2379 closed✓ Pass
10HTTP → HTTPS redirectInternet308 redirect✓ Pass
11HTTPS valid + fastInternet200, TLS valid, <100ms✓ Pass
12Admin API blocked from internetInternetTimeout / refused✓ Pass
13Authelia portal liveInternetHTTP 200✓ Pass — DNS live
14API redirects to auth (Phase 2)InternetHTTP 302 → AutheliaPhase 2

Known Issues & Fixes Applied

This list covers the issues found while first standing the stack up. A second, later pass — adding the test suite and doing the first live deployment — surfaced seven more real bugs (stale compose paths, Caddyfile one-liner syntax, an APISIX file-permission/UID mismatch, the Admin API NAT issue described in the updated Issue 5 below, and more). Full writeup: Gateway Testing & Deployment Guide.
ISSUE 1 bitnami/etcd removed from Docker Hub
Symptom docker compose pull fails — "not found"
Fix Switched to quay.io/coreos/etcd:v3.5.14. Environment variable names differ from Bitnami — updated config accordingly.
ISSUE 2 APISIX logs directory — permission denied
Symptom Container restarts: open() "/usr/local/apisix/logs/error.log" failed (13: Permission denied)
Root cause Host volume owned by hermes (uid 1001), but APISIX runs as apisix (uid 636)
Fix Removed the logs volume mount. APISIX writes inside the container. Access via docker logs apisix.
ISSUE 3 Authelia — storage.encryption_key required
Symptom Configuration: storage: option 'encryption_key' is required
Fix Added STORAGE_KEY to .env (openssl rand -hex 32) and injected into configuration.yml.
ISSUE 4 Authelia — global default_redirection_url rejected
Symptom session: option 'cookies' must be configured with the per cookie option 'default_redirection_url' but the global one is configured
Fix Removed global default_redirection_url. In v4.38+ it must only appear inside session.cookies[].
ISSUE 5 Admin API returns 403 from host curl
Symptom curl http://127.0.0.1:9180/apisix/admin/routes returns 403
Root cause APISIX nginx config: allow 127.0.0.0/24; deny all. Docker NAT changes the source IP to a bridge address, not loopback. Verified by checking the actual client IP APISIX logged.
Original workaround docker run --network container:apisix to execute requests from inside the container's network namespace where source IP is genuinely 127.0.0.1. Still works as a fallback.
Fix (2026-07-22) Widened allow_admin in config.yaml to include 172.16.0.0/12 (Docker's default bridge range) alongside 127.0.0.0/24, so a direct host curl now works without the namespace workaround. Port 9180 is still bound to 127.0.0.1 only in docker-compose.yml, so it remains unreachable from the public internet.
ISSUE 6 Authelia config not updating inside container
Symptom Authelia reads old config despite file being updated on host
Root cause authelia/ directory owned by root after Docker volume operations wrote files as root, making them unreadable by the host user
Fix Write files via a temp alpine container: docker run --rm -v .../authelia:/authelia alpine cp /src/file /authelia/file
ISSUE 7 Caddy shows (unhealthy) in docker ps
Symptom FailingStreak: 317 — health check fails with "connection refused"
Root cause 1 Health check targets Caddy's admin API /metrics endpoint, which was disabled via admin off in the Caddyfile. Port 2019 (admin API) isn't published in docker-compose.yml, so re-enabling it doesn't expose anything beyond the container itself.
Root cause 2 Even with the admin API re-enabled, the healthcheck used http://localhost:2019/metrics — this container resolves localhost to IPv6 (::1) first, while Caddy's admin listener only binds IPv4 127.0.0.1.
Fix (2026-07-22) Removed admin off from the Caddyfile, and changed the healthcheck target from localhost to 127.0.0.1 explicitly. Verified live: caddy now reports healthy, not cosmetic-unhealthy.

Pending Actions

Immediate — Phase 1 completion

Phase 2

Phase 3

Phase 4