Add Playwright-based scraping and login scripts for STATdx
- Implemented `statdx_play_chrome.py` to launch Chrome via Playwright, allowing for headful browsing and capturing XHR requests. - Created `statdx_playwright.py` for headless scraping of the STATdx application, including automatic login and HTML saving. - Developed `statdx_requests.py` for form-based login and scraping, providing a fallback for sites using traditional authentication. - Added example HTML output from Playwright scraping to `topics_playwright.html`. - Captured XHR request data in JSON format for analysis in `xhr_captured/app.statdx.com_csrf_token_8000d0ed_20251014T124122Z.json`.
This commit is contained in:
+70
@@ -0,0 +1,70 @@
|
||||
# Python cache and bytecode
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
|
||||
# Virtual environments
|
||||
env/
|
||||
venv/
|
||||
.venv/
|
||||
ENV/
|
||||
env.bak/
|
||||
venv.bak/
|
||||
|
||||
# Distribution / packaging
|
||||
build/
|
||||
dist/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
pip-wheel-metadata/
|
||||
|
||||
# Installer logs
|
||||
pip-log.txt
|
||||
pip-delete-this-directory.txt
|
||||
|
||||
# Test, coverage, caches
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
.coverage.*
|
||||
.cache/
|
||||
nosetests.xml
|
||||
coverage.xml
|
||||
.hypothesis/
|
||||
|
||||
# IDEs and editors
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Logs and databases
|
||||
*.log
|
||||
*.sqlite3
|
||||
*.db
|
||||
|
||||
# Environment/secrets
|
||||
.env
|
||||
.env.*
|
||||
secrets.json
|
||||
|
||||
# Compiled extensions
|
||||
*.so
|
||||
*.dylib
|
||||
*.dll
|
||||
|
||||
# Jupyter
|
||||
.ipynb_checkpoints/
|
||||
|
||||
# Misc
|
||||
*.egg
|
||||
*.egg-info
|
||||
*.dist-info
|
||||
|
||||
.chrome_profile
|
||||
.playwright_profile
|
||||
|
||||
captured/
|
||||
Reference in New Issue
Block a user