Enhance document processing in scrapers

- Updated pediatric multiple sclerosis article reference ID.
- Improved document_to_markdown.py to handle differential diagnoses, tables, anatomy, and cases more effectively.
- Added logging for cached entries of DDX, Tables, Anatomy, and Cases.
- Adjusted file pattern matching to process all JSON files by default.
- Fixed document ID extraction logic for various data types.
This commit is contained in:
Ross
2025-10-18 15:41:33 +01:00
parent 5983ca3252
commit 66512aa439
71 changed files with 1176 additions and 15 deletions
@@ -180,7 +180,7 @@ breadcrumbs:
**Aging Brain, Normal**
*Axial NECT shows typical basal ganglia (BG) Ca⁺⁺ in this 75-year-old man who presented after minor trauma. Note the location within the globus pallidus (GP) <img src='/img/arrows/CS.png'/>, typical for normal aging brain. Physiologic Ca⁺⁺ is typically seen in adults over 30 years.*
![Axial NECT shows typical basal ganglia (BG) Ca⁺⁺ in this 75-year-old man who presented after minor trauma. Note the location within the globus pallidus (GP) <img src='/img/arrows/CS.png'/>, typical for normal aging brain. Physiologic Ca⁺⁺ is typically seen in adults over 30 years.](images/app.statdx.com_image_thumbnail_04a1f1d8-6210-4030-84ff-4f60f3f7f7bf_size_174_quality_85_63332bdc_20251018T115114Z.jpg)
![Axial NECT shows typical basal ganglia (BG) Ca⁺⁺ in this 75-year-old man who presented after minor trauma. Note the location within the globus pallidus (GP) <img src='/img/arrows/CS.png'/>, typical for normal aging brain. Physiologic Ca⁺⁺ is typically seen in adults over 30 years.](images/app.statdx.com_image_thumbnail_04a1f1d8-6210-4030-84ff-4f60f3f7f7bf_size_174_quality_85_02fa8f20_20251018T142330Z.jpg)
**Aging Brain, Normal**
*Axial NECT shows typical basal ganglia (BG) Ca⁺⁺ in this 75-year-old man who presented after minor trauma. Note the location within the globus pallidus (GP) <img src='/img/arrows/CS.png'/>, typical for normal aging brain. Physiologic Ca⁺⁺ is typically seen in adults over 30 years.*