feat: auto migration resume with FETCH_ALL

[pp0rtal]

Fixes #30

Problem to solve

When a migration is stopped then re-run, MBDM will fetch using the same query:

This is not a problem if we know how to differentiate fields before and after (for example, a field is added)

new MongoBulkDataMigration<Score>({ ...
    query: { total: { $exists: false } },
    update: () => ({ total: Math.random() }),
});

However, it's a problem if we don't know how to differentiate a doc that has been migrated (for example, an updated field)

new MongoBulkDataMigration<Score>({ ...
    projection: { total: 1 },
    query: {},
    update: () => ({ total: total + 1 }),
});

In this case, the script is not resumable, it will start from the beginning.
In this case document already processed won't backed up again, but updated again. This is not really a case intended and covered.

Warning

It can also be dangerous ; if the migration is not idempotent, it will modify the document again 💥 (for instance, an increment ⚠️ )

Solution

We can solve this by preventing an empty query.
This PR does this with Type safety and hint like this:

This PR adds the new special query FETCH_ALL that wil do the same as {} empty query.

import { MongoBulkDataMigration, FETCH_ALL } from "@360-l/mongo-bulk-data-migration";
...
new MongoBulkDataMigration<Score>({
    db,
    id: "update_all_totals",
    collectionName: "scores",
    projection: { total:1 },
    // Automatic resume, iso {} 
    query: FETCH_ALL,
    update: () => ({ total: total + 1 }),
});

What it does

Instead of fetching all docs:

• We fetch the last updated doc
• We start the migration according the last id

To support big migrations, we fetch only one doc. So this features relies on the fact document _id is reliable and ascendant. (it won't work if we insert a doc with a lower _id in between of the migration).

[orca-security-eu]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	SAST	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca

[pp0rtal]

@hugop95 We did this PR with Elisa using Claude (our new best friend), but I think I would have exactly implemented this like that! Does the solution look good to you?

@elisap360 I added type safety to prevent query:{} I'm assigning you as a reviewer :)

[hugop95]

@pp0rtal

Thank you for this work! I've pushed 2 commits related to the comments I've written. Let me know what you think 👍

---

Also, you can run prettier to autofix some issues 😁

[pp0rtal]

@hugop95 Thanks a lot for the rework, I was honestly a bit hurried when I pushed and didn't take the necessary time to clean what Claude did (first time and lesson for later, won't happen again and sorry you had to waste time fixing it).
I'll wait your PR that add the pipeline checks

[hugop95]

@pp0rtal

Thank you!