Enhance document processing in scrapers

- Updated pediatric multiple sclerosis article reference ID.
- Improved document_to_markdown.py to handle differential diagnoses, tables, anatomy, and cases more effectively.
- Added logging for cached entries of DDX, Tables, Anatomy, and Cases.
- Adjusted file pattern matching to process all JSON files by default.
- Fixed document ID extraction logic for various data types.
This commit is contained in:
Ross
2025-10-18 15:41:33 +01:00
parent 5983ca3252
commit 66512aa439
71 changed files with 1176 additions and 15 deletions
@@ -409,9 +409,6 @@ breadcrumbs:
![Axial FLAIR MR shows peripheral, confluent areas of hyperintensity predominantly involving the subcortical white matter (WM) in this child with ADEM. The bilateral but asymmetric pattern is typical of ADEM.](images/app.statdx.com_image_thumbnail_deba3310-8e09-466c-97d2-1e6bccb28edf_size_168_quality_85_3dfef960_20251018T064936Z.jpg)
*Axial FLAIR MR shows peripheral, confluent areas of hyperintensity predominantly involving the subcortical white matter (WM) in this child with ADEM. The bilateral but asymmetric pattern is typical of ADEM.*
![Axial FLAIR MR shows peripheral, confluent areas of hyperintensity predominantly involving the subcortical white matter (WM) in this child with ADEM. The bilateral but asymmetric pattern is typical of ADEM.](images/app.statdx.com_image_thumbnail_deba3310-8e09-466c-97d2-1e6bccb28edf_size_174_quality_85_6f9ba8b5_20251018T122441Z.jpg)
*Axial FLAIR MR shows peripheral, confluent areas of hyperintensity predominantly involving the subcortical white matter (WM) in this child with ADEM. The bilateral but asymmetric pattern is typical of ADEM.*
![Axial T1 C+ MR in the same patient shows marked, irregular enhancement of nearly all lesions. As ADEM is a monophasic illness, enhancement of the majority of lesions is typical; all lesions have a similar time course. Enhancement of multiple sclerosis (MS) lesions is more variable.](images/app.statdx.com_image_e18458d3-5f6e-43fd-ab6d-72c7a86dca92_0f3893d6_20251018T064949Z.jpg)
*Axial T1 C+ MR in the same patient shows marked, irregular enhancement of nearly all lesions. As ADEM is a monophasic illness, enhancement of the majority of lesions is typical; all lesions have a similar time course. Enhancement of multiple sclerosis (MS) lesions is more variable.*