Commit Graph

9 Commits

Author SHA1 Message Date
Ross abbd4e24f2 feat: Enhance response capture logic to always include images and known document endpoints 2025-10-14 19:52:25 +01:00
Ross adc21f5795 feat: Implement passive XHR/fetch capture using Playwright
- Added `capture_passive_playwright.py` for synchronous passive capture of XHR/fetch responses.
- Added `capture_passive_playwright_async.py` for asynchronous passive capture using Playwright's async API.
- Introduced `save_page_snapshots.py` to save full page HTML snapshots with deduplication.
- Removed obsolete CSRF token capture JSON file.
2025-10-14 19:48:01 +01:00
Ross 683dd06643 Add automatic login functionality with optional credentials and selectors 2025-10-14 17:46:03 +01:00
Ross e3d081cbff Add scripts for capturing network responses and extracting document content
- Implement capture_with_cdp.py to capture all network response bodies using Chrome DevTools Protocol.
- Create extract_sections.py to extract sections from STATdx snapshot HTML files into JSON Lines.
- Add unpack_document_content.py to extract `documentHtml` from captured JSON bodies and save as HTML files.
- Update .gitignore to include playwright_profile.
2025-10-14 17:44:20 +01:00
Ross 496d7a6523 Update domain argument default value and enhance XHR capture to save HTML responses as files 2025-10-14 14:23:47 +01:00
Ross 26f900a406 Enhance capture functionality to allow saving images regardless of domain filter when image capture is enabled 2025-10-14 14:17:02 +01:00
Ross 448db2aaf9 Refactor logging to use Loguru in Playwright scripts for improved logging capabilities 2025-10-14 14:09:19 +01:00
Ross fd5922b3a6 Update .gitignore and requirements; enhance logging in Playwright scraper 2025-10-14 13:56:47 +01:00
Ross 491fca2f42 Add Playwright-based scraping and login scripts for STATdx
- Implemented `statdx_play_chrome.py` to launch Chrome via Playwright, allowing for headful browsing and capturing XHR requests.
- Created `statdx_playwright.py` for headless scraping of the STATdx application, including automatic login and HTML saving.
- Developed `statdx_requests.py` for form-based login and scraping, providing a fallback for sites using traditional authentication.
- Added example HTML output from Playwright scraping to `topics_playwright.html`.
- Captured XHR request data in JSON format for analysis in `xhr_captured/app.statdx.com_csrf_token_8000d0ed_20251014T124122Z.json`.
2025-10-14 13:46:39 +01:00