feat(worker): Add db_worker service with automatic restart and logging

feat(task_overview): Display generated series in task overview
docs(README): Update documentation for background task workers and troubleshooting
This commit is contained in:
Ross
2026-05-17 21:13:40 +01:00
parent ab9847e22e
commit c2d8376e22
8 changed files with 186 additions and 3 deletions
+53
View File
@@ -11,6 +11,59 @@ Running locally (development)
COMPOSE_ENV=dev docker compose -f docker/docker-compose.prod.yml -f docker/docker-compose.dev.yml up -d --build
```
Background task workers (automatic)
- The compose stack now includes a dedicated `worker` service which runs:
```sh
python manage.py db_worker
```
- This means workers start automatically with `docker compose up` and restart on failure.
- To inspect worker logs:
```sh
docker compose logs -f worker
```
- To scale workers (for higher throughput):
```sh
docker compose up -d --scale worker=2
```
Troubleshooting `SIGKILL` when running manually
- A `SIGKILL` on `db_worker` is usually the kernel OOM killer (out-of-memory), not a Django exception.
- Check kernel OOM events:
```sh
dmesg -T | grep -i -E "killed process|out of memory|oom"
```
- If OOM is confirmed, prefer running worker inside compose (managed restart + container limits), reduce concurrency/parallel services, or add memory/swap.
Manual worker startup (non-Docker production)
- If you are not yet running production in Docker, start the worker using the restart wrapper script:
```sh
source .venv/bin/activate
DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh
```
- This script restarts `db_worker` if it exits unexpectedly.
- To run it in the background and keep logs:
```sh
nohup DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh > logs/db_worker.log 2>&1 &
```
- To verify it is still running:
```sh
ps aux | grep "manage.py db_worker" | grep -v grep
tail -f logs/db_worker.log
```
- By default the development nginx ports are set in `.env.dev` to avoid colliding with a host nginx. The defaults now are:
- `NGINX_HTTP_PORT=8080`
- `NGINX_HTTPS_PORT=8444`