Ross
119 lines
4.6 KiB
Markdown
119 lines
4.6 KiB
Markdown
README
|
|
======
|
|
|
|
Quick notes for running this repository (development & observability)
|
|
|
|
Running locally (development)
|
|
- Use the included compose files. To run the dev stack (overrides) set `COMPOSE_ENV=dev` so the correct env file is loaded:
|
|
|
|
```sh
|
|
# run nginx on non-privileged ports (see .env.dev)
|
|
COMPOSE_ENV=dev docker compose -f docker/docker-compose.prod.yml -f docker/docker-compose.dev.yml up -d --build
|
|
```
|
|
|
|
Background task workers (automatic)
|
|
- The compose stack now includes a dedicated `worker` service which runs:
|
|
|
|
```sh
|
|
python manage.py db_worker
|
|
```
|
|
|
|
- This means workers start automatically with `docker compose up` and restart on failure.
|
|
- To inspect worker logs:
|
|
|
|
```sh
|
|
docker compose logs -f worker
|
|
```
|
|
|
|
- To scale workers (for higher throughput):
|
|
|
|
```sh
|
|
docker compose up -d --scale worker=2
|
|
```
|
|
|
|
Troubleshooting `SIGKILL` when running manually
|
|
- A `SIGKILL` on `db_worker` is usually the kernel OOM killer (out-of-memory), not a Django exception.
|
|
- Check kernel OOM events:
|
|
|
|
```sh
|
|
dmesg -T | grep -i -E "killed process|out of memory|oom"
|
|
```
|
|
|
|
- If OOM is confirmed, prefer running worker inside compose (managed restart + container limits), reduce concurrency/parallel services, or add memory/swap.
|
|
|
|
Manual worker startup (non-Docker production)
|
|
- If you are not yet running production in Docker, start the worker using the restart wrapper script:
|
|
|
|
```sh
|
|
source .venv/bin/activate
|
|
DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh
|
|
```
|
|
|
|
- This script restarts `db_worker` if it exits unexpectedly.
|
|
|
|
- To run it in the background and keep logs:
|
|
|
|
```sh
|
|
nohup env DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh > logs/db_worker.log 2>&1 &
|
|
```
|
|
|
|
- In fish shell, follow with `disown` so the shell does not keep the job attached:
|
|
|
|
```sh
|
|
nohup env DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh > logs/db_worker.log 2>&1 &
|
|
disown
|
|
```
|
|
|
|
- For server reliability, prefer running the same script under `systemd` (auto-start on reboot, restart-on-failure).
|
|
- A starter unit file is provided at `scripts/db-worker.service.example`.
|
|
|
|
- To verify it is still running:
|
|
|
|
```sh
|
|
ps aux | grep "manage.py db_worker" | grep -v grep
|
|
tail -f logs/db_worker.log
|
|
```
|
|
|
|
- By default the development nginx ports are set in `.env.dev` to avoid colliding with a host nginx. The defaults now are:
|
|
- `NGINX_HTTP_PORT=8080`
|
|
- `NGINX_HTTPS_PORT=8444`
|
|
|
|
Logging & observability (Loki + Promtail + Grafana)
|
|
- Promtail is configured to scrape `/var/log/rad/*.log` and push to Loki. Promtail's config is at `docker/promtail-config.yml` and the Loki config is at `docker/loki-config/local-config.yaml`.
|
|
- The nginx configs (both `deploy/nginx/prod.conf` and `deploy/nginx/dev.conf`) are configured to write access and error logs to `/var/log/rad/nginx.access.log` and `/var/log/rad/nginx.error.log` respectively.
|
|
- The compose files mount a host `logs/` folder into `/var/log/rad` so both nginx (write) and promtail (read) can access the same files:
|
|
- `../logs:/var/log/rad:rw` (nginx)
|
|
- `../logs:/var/log/rad:ro` (promtail)
|
|
|
|
Host setup for logs
|
|
- Create the host logs folder at the repo root (this repo's `logs/`). In development you can use permissive permissions so containers can write to it quickly:
|
|
|
|
```sh
|
|
mkdir -p logs
|
|
# development: make writable by all (change this for production)
|
|
sudo chown $USER:$USER logs
|
|
chmod 0777 logs
|
|
```
|
|
|
|
- In production prefer setting the folder owner to the nginx worker UID (or run nginx under a specific user) and use `0755` or `0750` as appropriate. Example:
|
|
|
|
```sh
|
|
# run on the host (determine nginx UID as needed)
|
|
sudo chown -R 101:101 /srv/rad/logs
|
|
sudo chmod 0755 /srv/rad/logs
|
|
```
|
|
|
|
Viewing logs in Grafana
|
|
- Grafana is available on the port exposed by compose (default `3000`).
|
|
- The Loki datasource is configured to use `http://loki:3100` (see compose). In Grafana Explore choose the Loki datasource and run queries such as:
|
|
|
|
- `{job="rad_app"}` (all collected logs)
|
|
- `{job="rad_app", filename="/var/log/rad/nginx.access.log"}`
|
|
|
|
Notes & recommendations
|
|
- In development the `logs/` folder is ignored by git (`/.gitignore` updated). Do not commit runtime logs.
|
|
- Consider a log rotation/retention policy for production (e.g., `logrotate` or use Loki retention policies).
|
|
- If you need nginx to re-resolve the backend service IP without restarting, consider using `resolver` and variable-based upstreams in the nginx config. For many development workflows restarting nginx after web restarts is the simplest approach.
|
|
|
|
If you want, I can also add a short `docs/OBSERVABILITY.md` with more details and recommended production settings (TLS/auth for Loki, retention, log rotation).
|