Commit Graph

5 Commits

Author SHA1 Message Date
Ross 496d7a6523 Update domain argument default value and enhance XHR capture to save HTML responses as files 2025-10-14 14:23:47 +01:00
Ross 26f900a406 Enhance capture functionality to allow saving images regardless of domain filter when image capture is enabled 2025-10-14 14:17:02 +01:00
Ross 448db2aaf9 Refactor logging to use Loguru in Playwright scripts for improved logging capabilities 2025-10-14 14:09:19 +01:00
Ross fd5922b3a6 Update .gitignore and requirements; enhance logging in Playwright scraper 2025-10-14 13:56:47 +01:00
Ross 491fca2f42 Add Playwright-based scraping and login scripts for STATdx
- Implemented `statdx_play_chrome.py` to launch Chrome via Playwright, allowing for headful browsing and capturing XHR requests.
- Created `statdx_playwright.py` for headless scraping of the STATdx application, including automatic login and HTML saving.
- Developed `statdx_requests.py` for form-based login and scraping, providing a fallback for sites using traditional authentication.
- Added example HTML output from Playwright scraping to `topics_playwright.html`.
- Captured XHR request data in JSON format for analysis in `xhr_captured/app.statdx.com_csrf_token_8000d0ed_20251014T124122Z.json`.
2025-10-14 13:46:39 +01:00