Repository navigation
feat(displacement): seed IDMC internal displacement events from HDX (#9014) - #9444
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Strix Security ReviewAll previously reported security findings have been resolved. 1 resolved finding
Review summaryReviewed all 12 changed files: the new IDMC IDU HDX ingestion seeder, its health/bundle/registry wiring, the generated attribution manifest, documentation count updates, and the 17-test suite. The previously reported HDX resource-URL finding is resolved: Updated for Reviewed by Strix |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Description check |  |
The description provides detailed implementation, failure-handling, wiring, licensing, and verification information, but it does not follow the required template or complete its required sections and … | Add the required Summary, Type of change, Affected areas, Checklist, Documentation Alignment Checklist, and Screenshots sections. Mark applicable checklist items, provide the required documentation-alignment evidence for the changed health … |
| Docstring Coverage | |
Docstring coverage is 47.06% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 5 files. (7 skipped: … | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Title check The title clearly identifies the main change: adding an IDMC internal displacement event seeder from HDX. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Description check
-
Explanation
The description provides detailed implementation, failure-handling, wiring, licensing, and verification information, but it does not follow the required template or complete its required sections and checklists.
Resolution
Add the required Summary, Type of change, Affected areas, Checklist, Documentation Alignment Checklist, and Screenshots sections. Mark applicable checklist items, provide the required documentation-alignment evidence for the changed health and data claims, and mark non-applicable items explicitly.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 47.06% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 5 files. (7 skipped: 7 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
- Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
![strix-security[bot]](https://avatars.githubusercontent.com/in/2749804?s=60&v=4){kind=small}
 [Medium impact] The change is contained to displacement data publishing and its health monitoring. The PR appears safe to merge, with non-blocking fixes recommended for stored totals and download retries. FindingsSummaryAdds a daily IDMC displacement worker that reads country CSVs from HDX and stores events, country totals, and surge flags in Redis.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const events = buildEvents(rows, { nowMs }); | ||
| const countries = summarizeCountries(events, { nowMs }); |
There was a problem hiding this comment.
buildEvents drops displacements older than 180 days before summarizeCountries counts newlyReported7d. A recommended figure published this week for an older displacement contributes nothing, despite the stated requirement to count new reports regardless of displacement date.
Count newly published figures from all valid rows. Keep the 180-day limit for event history and surge totals, and add a test for an older displacement published this week.
There was a problem hiding this comment.
Fixed in 704ee4d. summarizeCountries now takes every normalized event; a figure published in the last 7 dates counts toward newlyReported7d even when the displacement predates the 180-day window, and stays out of the events list and the baseline. Test: counts a figure published this week for a displacement older than the window....
| const recentFrom = daysAgo(nowMs, RECENT_DAYS); | ||
| const reportedFrom = daysAgo(nowMs, NEWLY_REPORTED_DAYS); | ||
| const baselineMonths = (WINDOW_DAYS - RECENT_DAYS) / 30; |
There was a problem hiding this comment.
On October 20, recentFrom is September 20. The inclusive comparison makes displaced30d cover 31 calendar dates, while newlyReported7d covers eight. The baseline still represents 30-day months, so the wider recent window can also produce a false surge.
Use consistent calendar-day windows and test both sides of each cutoff.
There was a problem hiding this comment.
Fixed in 704ee4d. Windows now cover exactly 180 / 30 / 7 calendar dates including today (windowStart = daysAgo(days - 1)), so the 30 recent dates and 150 baseline dates split at 2026-09-21 for 2026-10-20. Tests pin both sides of each cutoff.
| const resp = await getWithTimeout(url); | ||
| const length = Number(resp.headers.get('content-length')); | ||
| if (Number.isFinite(length) && length > MAX_CSV_BYTES) throw new Error(`HDX ${url}: ${length} bytes exceeds ${MAX_CSV_BYTES}`); | ||
| const text = await resp.text(); |
There was a problem hiding this comment.
getWithTimeout returns before the response body is read, so withRetry never sees failures from resp.text() or the listing's .json(). A connection reset during a country download immediately drops that country. More than three failed files abort the run without using the configured retries.
Read each body inside the retry callback. Keep CSV schema checks outside it.
There was a problem hiding this comment.
Fixed in 704ee4d. The body read (resp.json() / resp.text()) now runs inside the withRetry callback; size-cap failures are non-retryable, schema checks stay outside. Test: retries a download whose body breaks mid-stream.
…9014) IDMC has not issued a Helix API client_id, so read the same Internal Displacement Updates from the per-country <iso3>-idmc-idu-events CSVs that IDMC publishes on HDX (116 datasets, no key). - Keep "Recommended figure" rows from the last 180 days; "Triangulation" rows cross-check the same displacement and are dropped. - Per country: 30-day displaced total (conflict/disaster split), monthly baseline over the prior 150 days, figures newly reported in the last 7 days, and a surge flag (>= 5,000 and >= 3x baseline). IDMC enters figures a median of 23 days after displacement (p90 65), so a same-week window would be empty. - Skip up to 3 unreadable country files and report them; more fails the run. - Daily in seed-bundle-health; standalone /api/health key idmcIduDisplacement behind an activation marker until the first publish. Claude-Session: https://claude.ai/code/session_01KH6ud2RjKZ5rTa5WqmcNPt
…t windows (#9014) Review fixes on the IDMC IDU seeder: - Resource URLs come from HDX metadata, so every hop is checked: https only, data.humdata.org or the hdx-production-filestore S3 path its download links redirect to. Redirects are followed by hand; anything else is refused. - Read response bodies inside the retry, so a connection dropped mid-download is retried instead of dropping the country. - newlyReported7d counts figures published this week even when the displacement predates the 180-day window; such figures stay out of the events list and the baseline. - Windows cover exactly 180 / 30 / 7 calendar dates including today (displaced30d spanned 31 dates, newlyReported7d 8). Claude-Session: https://claude.ai/code/session_01KH6ud2RjKZ5rTa5WqmcNPt
81644e5 to
704ee4d
Compare
This branch was successfully deployed
1 participant
Part of #9014 (Step 2).
IDMC has not issued a Helix API
client_idafter two emails to ch.datainfo@idmc.ch, so this reads the same Internal Displacement Updates (IDU) from the per-country<iso3>-idmc-idu-eventsCSVs IDMC publishes on HDX: 116 datasets, no key, Helix column schema.What it publishes (
displacement:idmc-idu:v1)surge(≥ 5,000 and ≥ 3× baseline).Why 30 days, not same-week
The issue asked for same-week surge detection. IDMC enters a figure a median of 23 days after the displacement (p75 41, p90 65; measured across the 2026-10-20 files). Only 12 events carry a displacement date in the last 7 days. So surges use a 30-day window, and
newlyReported7dtracks what IDMC published this week, whatever the displacement date.Failure handling
missingCountries; more fails the run, and the previous snapshot stays (3-day TTL).data.humdata.orgor thehdx-production-filestore/S3 path its downloads redirect to. Redirects are followed manually; anything else is refused before a request is made.validaterequires ≥ 30 countries. Health requires ≥ 500 events.Wiring
seed-bundle-health(300 s timeout; the live run took ~28 s)./api/healthkeyidmcIduDisplacement, behind an activation marker until the first publish (same bridge as DTM). Probed-key total in the docs goes 327 → 328.created_at), max 14 days.Licence
HDX lists these datasets as CC BY-IGO, but IDMC's API page says CC BY-NC-SA 3.0 IGO. This is being confirmed with IDMC by phone. Nothing user-facing reads this key yet, so the question gates the follow-up consumer, not this seeder.
Verification
tests/idmc-idu-displacement-seed.test.mjs: 26 tests covering CSV quoting and BOM, role filter, both sides of each window cutoff, surge threshold (including the rounding edge), late reports, URL allowlist and redirect refusal, S3 redirect, mid-stream retry, truncation, and partial/total download failure. Each review fix was mutation-checked: reverting it turns at least one test red.npm run test:data: all green after regenerating the health totals; the 6 initial failures were the count and manifest gates, rerun and passing (197/197).npm run test:railway-registry: 153/153.No UI changes.
https://claude.ai/code/session_01KH6ud2RjKZ5rTa5WqmcNPt
Summary by CodeRabbit
New Features
Documentation
Tests