Use other mongo driver (a.k.a mongo4cats) to read from secondary instance

[lenguyenthanh]

Similar to what We did in lila-search's ingestor

[cakiki]

Do we still want to switch to mongo4cats? (Assuming this is the lila-search change you're referring to: lichess-org/lila-search@33caf42)

[lenguyenthanh]

It was just my idea, I don't know if we really want to switch to mongo4cats here.

Since then lila-search-ingestor has been improved quite a lot, and currently it can watch game collection from mongodb and write to clickhouse and elasticsearch at the same time: lichess-org/lila-search#723. I also have some experiment with exporting other format like csv: https://github.com/lichess-org/lila-search/tree/master/modules/lila-game-export/src.

Now, I imagine, to switch the pgn export part to lila-search-ingestor. So we don't have to read the database twice and we have the new pgn file for each month automatically and immediately (after 00:00:00).

I also looked at parque formet, lila-search-ingestor can handle that too, so another time saving potential.

[cakiki]

Parquet would be super useful for the hugging face and the kaggle data exports.