# run nginx on non-privileged ports (see .env.dev)
COMPOSE_ENV=dev docker compose -f docker/docker-compose.prod.yml -f docker/docker-compose.dev.yml up -d --build

Background task workers (automatic)

The compose stack now includes a dedicated worker service which runs:
```
python manage.py db_worker
```
This means workers start automatically with docker compose up and restart on failure.
To inspect worker logs:
```
docker compose logs -f worker
```
To scale workers (for higher throughput):
```
docker compose up -d --scale worker=2
```

Troubleshooting SIGKILL when running manually

A SIGKILL on db_worker is usually the kernel OOM killer (out-of-memory), not a Django exception.

Check kernel OOM events:

dmesg -T | grep -i -E "killed process|out of memory|oom"

If OOM is confirmed, prefer running worker inside compose (managed restart + container limits), reduce concurrency/parallel services, or add memory/swap.

Manual worker startup (non-Docker production)

If you are not yet running production in Docker, start the worker using the restart wrapper script:
```
source .venv/bin/activate
DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh
```
This script restarts db_worker if it exits unexpectedly.

To run it in the background and keep logs:

nohup env DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh > logs/db_worker.log 2>&1 &

In fish shell, follow with disown so the shell does not keep the job attached:

nohup env DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh > logs/db_worker.log 2>&1 &
disown

For server reliability, prefer running the same script under systemd (auto-start on reboot, restart-on-failure).
A starter unit file is provided at scripts/db-worker.service.example.

To verify it is still running:

ps aux | grep "manage.py db_worker" | grep -v grep
tail -f logs/db_worker.log

By default the development nginx ports are set in .env.dev to avoid colliding with a host nginx. The defaults now are:
- NGINX_HTTP_PORT=8080
- NGINX_HTTPS_PORT=8444

Logging & observability (Loki + Promtail + Grafana)

Promtail is configured to scrape /var/log/rad/*.log and push to Loki. Promtail's config is at docker/promtail-config.yml and the Loki config is at docker/loki-config/local-config.yaml.
The nginx configs (both deploy/nginx/prod.conf and deploy/nginx/dev.conf) are configured to write access and error logs to /var/log/rad/nginx.access.log and /var/log/rad/nginx.error.log respectively.
The compose files mount a host logs/ folder into /var/log/rad so both nginx (write) and promtail (read) can access the same files:
- ../logs:/var/log/rad:rw (nginx)
- ../logs:/var/log/rad:ro (promtail)

Host setup for logs

Create the host logs folder at the repo root (this repo's logs/). In development you can use permissive permissions so containers can write to it quickly:
```
mkdir -p logs
# development: make writable by all (change this for production)
sudo chown $USER:$USER logs
chmod 0777 logs
```
In production prefer setting the folder owner to the nginx worker UID (or run nginx under a specific user) and use 0755 or 0750 as appropriate. Example:
```
# run on the host (determine nginx UID as needed)
sudo chown -R 101:101 /srv/rad/logs
sudo chmod 0755 /srv/rad/logs
```

Notes & recommendations

In development the logs/ folder is ignored by git (/.gitignore updated). Do not commit runtime logs.
Consider a log rotation/retention policy for production (e.g., logrotate or use Loki retention policies).
If you need nginx to re-resolve the backend service IP without restarting, consider using resolver and variable-based upstreams in the nginx config. For many development workflows restarting nginx after web restarts is the simplest approach.

If you want, I can also add a short docs/OBSERVABILITY.md with more details and recommended production settings (TLS/auth for Loki, retention, log rotation).

Running tests

Ensure the development Docker environment is running (COMPOSE_ENV=dev docker compose -f docker/docker-compose.local.yml -f docker/docker-compose.dev.yml up -d).