feat(worker): Add db_worker service with automatic restart and logging
feat(task_overview): Display generated series in task overview docs(README): Update documentation for background task workers and troubleshooting
This commit is contained in:
@@ -11,6 +11,59 @@ Running locally (development)
|
||||
COMPOSE_ENV=dev docker compose -f docker/docker-compose.prod.yml -f docker/docker-compose.dev.yml up -d --build
|
||||
```
|
||||
|
||||
Background task workers (automatic)
|
||||
- The compose stack now includes a dedicated `worker` service which runs:
|
||||
|
||||
```sh
|
||||
python manage.py db_worker
|
||||
```
|
||||
|
||||
- This means workers start automatically with `docker compose up` and restart on failure.
|
||||
- To inspect worker logs:
|
||||
|
||||
```sh
|
||||
docker compose logs -f worker
|
||||
```
|
||||
|
||||
- To scale workers (for higher throughput):
|
||||
|
||||
```sh
|
||||
docker compose up -d --scale worker=2
|
||||
```
|
||||
|
||||
Troubleshooting `SIGKILL` when running manually
|
||||
- A `SIGKILL` on `db_worker` is usually the kernel OOM killer (out-of-memory), not a Django exception.
|
||||
- Check kernel OOM events:
|
||||
|
||||
```sh
|
||||
dmesg -T | grep -i -E "killed process|out of memory|oom"
|
||||
```
|
||||
|
||||
- If OOM is confirmed, prefer running worker inside compose (managed restart + container limits), reduce concurrency/parallel services, or add memory/swap.
|
||||
|
||||
Manual worker startup (non-Docker production)
|
||||
- If you are not yet running production in Docker, start the worker using the restart wrapper script:
|
||||
|
||||
```sh
|
||||
source .venv/bin/activate
|
||||
DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh
|
||||
```
|
||||
|
||||
- This script restarts `db_worker` if it exits unexpectedly.
|
||||
|
||||
- To run it in the background and keep logs:
|
||||
|
||||
```sh
|
||||
nohup DJANGO_SETTINGS_MODULE=rad.settings ./scripts/run-db-worker.sh > logs/db_worker.log 2>&1 &
|
||||
```
|
||||
|
||||
- To verify it is still running:
|
||||
|
||||
```sh
|
||||
ps aux | grep "manage.py db_worker" | grep -v grep
|
||||
tail -f logs/db_worker.log
|
||||
```
|
||||
|
||||
- By default the development nginx ports are set in `.env.dev` to avoid colliding with a host nginx. The defaults now are:
|
||||
- `NGINX_HTTP_PORT=8080`
|
||||
- `NGINX_HTTPS_PORT=8444`
|
||||
|
||||
Reference in New Issue
Block a user