Description
Currently, when we add new columns to the include list for a table, the system triggers a full reload ( via reload-requested) for the entire table. This can be costly in terms of processing time and storage.
Feature Proposal:
Enable support for partial reloading when only new columns are added.
Possible solution
Based on discussion with George, the idea is to:
-> identify which columns are newly added compared to the previous schema
-> perform a reload that fetches only the primary key (PK) and the newly added columns
-> Use this data to update the target table in place instead of replacing the entire dataset.
Alternatives
No response
Context
No response
Description
Currently, when we add new columns to the
includelist for a table, the system triggers a full reload ( viareload-requested) for the entire table. This can be costly in terms of processing time and storage.Feature Proposal:
Enable support for partial reloading when only new columns are added.
Possible solution
Based on discussion with George, the idea is to:
-> identify which columns are newly added compared to the previous schema
-> perform a reload that fetches only the primary key (PK) and the newly added columns
-> Use this data to update the target table in place instead of replacing the entire dataset.
Alternatives
No response
Context
No response