Description
Recently, we encountered a problem where the SQL streaming source was executing schema update queries on the target SQL table at a frequency equivalent to the change capture interval. Ideally, the stream should only initiate these updates when a schema change is detected. It's important to note, however, that we currently have no evidence linking this issue directly to the change capture interval. Therefore, to gain a better understanding of the issue, we need to improve the observability of our streaming sources.
Possible solution
Add additional metrics describing the following events:
- A schema change detection event happened.
- A change capture event happened in the stream
Alternatives
Add logging for those events.
Context
Required for all stream sources in the framework that supports schema change detection (SQL and CDM for now).
Description
Recently, we encountered a problem where the SQL streaming source was executing schema update queries on the target SQL table at a frequency equivalent to the change capture interval. Ideally, the stream should only initiate these updates when a schema change is detected. It's important to note, however, that we currently have no evidence linking this issue directly to the change capture interval. Therefore, to gain a better understanding of the issue, we need to improve the observability of our streaming sources.
Possible solution
Add additional metrics describing the following events:
Alternatives
Add logging for those events.
Context
Required for all stream sources in the framework that supports schema change detection (SQL and CDM for now).