We just dded MemTable and WAL in lance: https://lance.org/format/table/mem_wal/
I am wondering if it could be beneficial for lance context to use that instead of writing to a Lance table directly, because context is mostly for a single agent which means a single writer (correct me if I am wrong), and doing many small writes is not good for performance.
With MemTable & WAL, you don't need to do small writes and then run compaction. Data is accumulated in memory, and then flushed to disk/S3 when in reasonable size. It can also build indexes while writing, so you don't have to asynchronously build index.
You lose a bit of the time travel capability, but if your record already has a timestamp per row, it should be able to achieve similar effect.
Any thoughts? @beinan @dcfocus
We just dded MemTable and WAL in lance: https://lance.org/format/table/mem_wal/
I am wondering if it could be beneficial for lance context to use that instead of writing to a Lance table directly, because context is mostly for a single agent which means a single writer (correct me if I am wrong), and doing many small writes is not good for performance.
With MemTable & WAL, you don't need to do small writes and then run compaction. Data is accumulated in memory, and then flushed to disk/S3 when in reasonable size. It can also build indexes while writing, so you don't have to asynchronously build index.
You lose a bit of the time travel capability, but if your record already has a timestamp per row, it should be able to achieve similar effect.
Any thoughts? @beinan @dcfocus