- Updated pediatric multiple sclerosis article reference ID.
- Improved document_to_markdown.py to handle differential diagnoses, tables, anatomy, and cases more effectively.
- Added logging for cached entries of DDX, Tables, Anatomy, and Cases.
- Adjusted file pattern matching to process all JSON files by default.
- Fixed document ID extraction logic for various data types.
- Created detailed articles for Adrenal Adenoma, Adrenal Cyst, Adrenal Myelolipoma, and general Adrenal anatomy.
- Included key facts, imaging findings, differential diagnoses, pathology, clinical issues, and diagnostic checklists for each condition.
- Enhanced understanding of adrenal tumors and their characteristics through structured documentation.
- Created a detailed article for Vascular Dementia covering key facts, terminology, imaging findings, differential diagnoses, pathology, clinical issues, and diagnostic checklist.
- Developed an extensive article on Wallerian Degeneration including key facts, terminology, imaging features, differential diagnoses, pathology, clinical issues, and diagnostic checklist.
- Created markdown files for CDC Diagnostic Criteria for Creutzfeldt-Jakob Disease and multiple media documents.
- Added a script to convert JSON documents containing HTML into markdown format.
- Implemented functions to extract titles and HTML content from JSON files.
- Enhanced markdown generation with support for various HTML elements including tables, lists, and images.
- Included error handling and logging for file processing.
- Added `capture_passive_playwright.py` for synchronous passive capture of XHR/fetch responses.
- Added `capture_passive_playwright_async.py` for asynchronous passive capture using Playwright's async API.
- Introduced `save_page_snapshots.py` to save full page HTML snapshots with deduplication.
- Removed obsolete CSRF token capture JSON file.
- Implement capture_with_cdp.py to capture all network response bodies using Chrome DevTools Protocol.
- Create extract_sections.py to extract sections from STATdx snapshot HTML files into JSON Lines.
- Add unpack_document_content.py to extract `documentHtml` from captured JSON bodies and save as HTML files.
- Update .gitignore to include playwright_profile.
- Implemented `statdx_play_chrome.py` to launch Chrome via Playwright, allowing for headful browsing and capturing XHR requests.
- Created `statdx_playwright.py` for headless scraping of the STATdx application, including automatic login and HTML saving.
- Developed `statdx_requests.py` for form-based login and scraping, providing a fallback for sites using traditional authentication.
- Added example HTML output from Playwright scraping to `topics_playwright.html`.
- Captured XHR request data in JSON format for analysis in `xhr_captured/app.statdx.com_csrf_token_8000d0ed_20251014T124122Z.json`.
Ross
Ross