Commit Graph

6 Commits

Author SHA1 Message Date
Ross e3d081cbff Add scripts for capturing network responses and extracting document content
- Implement capture_with_cdp.py to capture all network response bodies using Chrome DevTools Protocol.
- Create extract_sections.py to extract sections from STATdx snapshot HTML files into JSON Lines.
- Add unpack_document_content.py to extract `documentHtml` from captured JSON bodies and save as HTML files.
- Update .gitignore to include playwright_profile.
2025-10-14 17:44:20 +01:00
Ross 496d7a6523 Update domain argument default value and enhance XHR capture to save HTML responses as files 2025-10-14 14:23:47 +01:00
Ross 26f900a406 Enhance capture functionality to allow saving images regardless of domain filter when image capture is enabled 2025-10-14 14:17:02 +01:00
Ross 448db2aaf9 Refactor logging to use Loguru in Playwright scripts for improved logging capabilities 2025-10-14 14:09:19 +01:00
Ross fd5922b3a6 Update .gitignore and requirements; enhance logging in Playwright scraper 2025-10-14 13:56:47 +01:00
Ross 491fca2f42 Add Playwright-based scraping and login scripts for STATdx
- Implemented `statdx_play_chrome.py` to launch Chrome via Playwright, allowing for headful browsing and capturing XHR requests.
- Created `statdx_playwright.py` for headless scraping of the STATdx application, including automatic login and HTML saving.
- Developed `statdx_requests.py` for form-based login and scraping, providing a fallback for sites using traditional authentication.
- Added example HTML output from Playwright scraping to `topics_playwright.html`.
- Captured XHR request data in JSON format for analysis in `xhr_captured/app.statdx.com_csrf_token_8000d0ed_20251014T124122Z.json`.
2025-10-14 13:46:39 +01:00