README
======

Quick notes for running this repository (development & observability)

Running locally (development)
- Use the included compose files. To run the dev stack (overrides) set `COMPOSE_ENV=dev` so the correct env file is loaded:

  ```sh
  # run nginx on non-privileged ports (see .env.dev)
  COMPOSE_ENV=dev docker compose -f docker/docker-compose.prod.yml -f docker/docker-compose.dev.yml up -d --build
  ```

Background task workers (automatic)
- The compose stack now includes a dedicated `worker` service which runs:

  ```sh
  python manage.py db_worker
  ```

- This means workers start automatically with `docker compose up` and restart on failure.
- To inspect worker logs:

  ```sh
  docker compose logs -f worker
  ```

- To scale workers (for higher throughput):

  ```sh
  docker compose up -d --scale worker=2
  ```

Troubleshooting `SIGKILL` when running manually
- A `SIGKILL` on `db_worker` is usually the kernel OOM killer (out-of-memory), not a Django exception.
- Check kernel OOM events:

  ```sh
  dmesg -T | grep -i -E "killed process|out of memory|oom"
  ```

- If OOM is confirmed, prefer running worker inside compose (managed restart + container limits), reduce concurrency/parallel services, or add memory/swap.

Manual worker startup (non-Docker production)
- If you are not yet running production in Docker, start the worker using the restart wrapper script:

  ```sh
  source .venv/bin/activate
  DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh
  ```

- This script restarts `db_worker` if it exits unexpectedly.

- To run it in the background and keep logs:

  ```sh
  nohup env DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh > logs/db_worker.log 2>&1 &
  ```

- In fish shell, follow with `disown` so the shell does not keep the job attached:

  ```sh
  nohup env DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh > logs/db_worker.log 2>&1 &
  disown
  ```

- For server reliability, prefer running the same script under `systemd` (auto-start on reboot, restart-on-failure).
- A starter unit file is provided at `scripts/db-worker.service.example`.

- To verify it is still running:

  ```sh
  ps aux | grep "manage.py db_worker" | grep -v grep
  tail -f logs/db_worker.log
  ```

- By default the development nginx ports are set in `.env.dev` to avoid colliding with a host nginx. The defaults now are:
  - `NGINX_HTTP_PORT=8080`
  - `NGINX_HTTPS_PORT=8444`

Logging & observability (Loki + Promtail + Grafana)
- Promtail is configured to scrape `/var/log/rad/*.log` and push to Loki. Promtail's config is at `docker/promtail-config.yml` and the Loki config is at `docker/loki-config/local-config.yaml`.
- The nginx configs (both `deploy/nginx/prod.conf` and `deploy/nginx/dev.conf`) are configured to write access and error logs to `/var/log/rad/nginx.access.log` and `/var/log/rad/nginx.error.log` respectively.
- The compose files mount a host `logs/` folder into `/var/log/rad` so both nginx (write) and promtail (read) can access the same files:
  - `../logs:/var/log/rad:rw` (nginx)
  - `../logs:/var/log/rad:ro` (promtail)

Host setup for logs
- Create the host logs folder at the repo root (this repo's `logs/`). In development you can use permissive permissions so containers can write to it quickly:

  ```sh
  mkdir -p logs
  # development: make writable by all (change this for production)
  sudo chown $USER:$USER logs
  chmod 0777 logs
  ```

- In production prefer setting the folder owner to the nginx worker UID (or run nginx under a specific user) and use `0755` or `0750` as appropriate. Example:

  ```sh
  # run on the host (determine nginx UID as needed)
  sudo chown -R 101:101 /srv/rad/logs
  sudo chmod 0755 /srv/rad/logs
  ```

Viewing logs in Grafana
- Grafana is available on the port exposed by compose (default `3000`).
- The Loki datasource is configured to use `http://loki:3100` (see compose). In Grafana Explore choose the Loki datasource and run queries such as:

  - `{job="rad_app"}`  (all collected logs)
  - `{job="rad_app", filename="/var/log/rad/nginx.access.log"}`

Notes & recommendations
- In development the `logs/` folder is ignored by git (`/.gitignore` updated). Do not commit runtime logs.
- Consider a log rotation/retention policy for production (e.g., `logrotate` or use Loki retention policies).
- If you need nginx to re-resolve the backend service IP without restarting, consider using `resolver` and variable-based upstreams in the nginx config. For many development workflows restarting nginx after web restarts is the simplest approach.

If you want, I can also add a short `docs/OBSERVABILITY.md` with more details and recommended production settings (TLS/auth for Loki, retention, log rotation).