perf: direct CSR-to-LP writer for frozen constraints#631
perf: direct CSR-to-LP writer for frozen constraints#631FBumann wants to merge 1 commit intoPyPSA:perf/matrix-accessor-rewritefrom
Conversation
Override Constraint.to_polars() to expand CSR data directly into a polars DataFrame, bypassing the expensive mutable() → xarray Dataset reconstruction. Also override iterate_slices() to yield CSR row-batches instead of relying on xarray's isel(). Move eliminate_zeros() to freeze time (from_mutable) so the cleanup happens once rather than on every to_polars() call. LP write is now 20-40% faster than master across all benchmark models. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
thank you |
|
this is already nice! I made some LP writing experiment in https://github.com/FabianHofmann/crs2lp which showed quite some improvements in performance. It directly writes the LP file from the CRS objects using custom rust module. That said, I am happy to pull this one in to keep changes small but in the long term it is probably worth to move the implementation in the repo. |
|
@coroa this is great, one big question that came up when looking at the code is: why renaming |
this should be in the other pr, moving it there (sorry) |
Summary
Constraint.to_polars()to expand CSR data directly into a polars DataFrame, bypassing the expensivemutable()→ xarray Dataset reconstructionConstraint.iterate_slices()to yield CSR row-batches instead of relying on xarray'sisel()(which doesn't exist on frozen constraints — this was crashing)eliminate_zeros()to freeze time (from_mutable) so cleanup happens onceBenchmark results (vs master)
Test plan
pytest test/test_io.py— all 25 tests passpytest test/test_constraint.py— all 72 tests passtest_to_file_lp_frozen_vs_mutableverifies byte-identical LP output between frozen and mutable pathsbenchmarks/test_lp_write.py— all benchmarks pass including previously crashingpypsa_scigrid🤖 Generated with Claude Code