Ross
491fca2f42
- Implemented `statdx_play_chrome.py` to launch Chrome via Playwright, allowing for headful browsing and capturing XHR requests. - Created `statdx_playwright.py` for headless scraping of the STATdx application, including automatic login and HTML saving. - Developed `statdx_requests.py` for form-based login and scraping, providing a fallback for sites using traditional authentication. - Added example HTML output from Playwright scraping to `topics_playwright.html`. - Captured XHR request data in JSON format for analysis in `xhr_captured/app.statdx.com_csrf_token_8000d0ed_20251014T124122Z.json`.
70 lines
707 B
Plaintext
70 lines
707 B
Plaintext
# Python cache and bytecode
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
|
|
# Virtual environments
|
|
env/
|
|
venv/
|
|
.venv/
|
|
ENV/
|
|
env.bak/
|
|
venv.bak/
|
|
|
|
# Distribution / packaging
|
|
build/
|
|
dist/
|
|
*.egg-info/
|
|
.eggs/
|
|
pip-wheel-metadata/
|
|
|
|
# Installer logs
|
|
pip-log.txt
|
|
pip-delete-this-directory.txt
|
|
|
|
# Test, coverage, caches
|
|
.pytest_cache/
|
|
.coverage
|
|
.coverage.*
|
|
.cache/
|
|
nosetests.xml
|
|
coverage.xml
|
|
.hypothesis/
|
|
|
|
# IDEs and editors
|
|
.vscode/
|
|
.idea/
|
|
*.swp
|
|
*.swo
|
|
|
|
# OS files
|
|
.DS_Store
|
|
Thumbs.db
|
|
|
|
# Logs and databases
|
|
*.log
|
|
*.sqlite3
|
|
*.db
|
|
|
|
# Environment/secrets
|
|
.env
|
|
.env.*
|
|
secrets.json
|
|
|
|
# Compiled extensions
|
|
*.so
|
|
*.dylib
|
|
*.dll
|
|
|
|
# Jupyter
|
|
.ipynb_checkpoints/
|
|
|
|
# Misc
|
|
*.egg
|
|
*.egg-info
|
|
*.dist-info
|
|
|
|
.chrome_profile
|
|
.playwright_profile
|
|
|
|
captured/ |