statdx

This folder contains two example Python scripts showing common ways to programmatically log in and scrape pages from a site that requires authentication:

scrapers/statdx_requests.py — a "requests"-based approach that works when authentication is a plain form POST (non-JS or CSRF-protected with a token present in the HTML). It tries to discover the login form and submit it. You will likely need to update the login action URL or the form field names for STATdx.
scrapers/statdx_playwright.py — a Playwright-based, browser-automation approach which runs a real Chromium instance, so it works with JS-heavy sites and single-sign-on flows.

Files:

requirements.txt — Python deps
.env.example — environment variable names to copy into .env
scrapers/statdx_requests.py — example script
scrapers/statdx_playwright.py — playwright script

Security / Legal

Only run these scripts for accounts/data you own or have explicit permission to access.
Do not hard-code credentials into scripts. Use environment variables or a secrets manager.

Quick start (fish shell):

Create a virtualenv and install deps:

python -m venv .venv
. .venv/bin/activate.fish
pip install -r requirements.txt
# When using Playwright, install browsers once:
python -m playwright install

Copy .env.example to .env and edit with your username/password:

cp .env.example .env
# edit .env with your editor

Run the requests script (may need field names):

python scrapers/statdx_requests.py

Run the Playwright script (works for JS logins):

python scrapers/statdx_playwright.py

If the login flow fails, open your browser devtools (Network tab) while logging in manually to find the exact POST URL and field names and paste them into the script configuration.

If you want, I can adapt the scripts further after you either:

share the network POST URL + field names, or
allow me to run the script here with credentials (not possible without your consent)."