Skip to content

Commit 12dfc74

Browse files
committed
fix(polling): prevent data loss on partial row failures and harden idempotency key
- Sheets: only advance lastKnownRowCount by processedCount when there are failures, so failed rows are retried on the next poll cycle (idempotency deduplicates already-processed rows on re-fetch) - Drive: add fallback for change.time in idempotency key to prevent key collisions if the field is ever absent from the API response
1 parent bbfbb98 commit 12dfc74

File tree

2 files changed

+11
-8
lines changed

2 files changed

+11
-8
lines changed

apps/sim/lib/webhooks/polling/google-drive.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -357,7 +357,7 @@ async function processChanges(
357357
}
358358

359359
try {
360-
const idempotencyKey = `${webhookData.id}:${change.fileId}:${change.time}`
360+
const idempotencyKey = `${webhookData.id}:${change.fileId}:${change.time || change.fileId}`
361361

362362
await pollingIdempotency.executeWithIdempotency('google-drive', idempotencyKey, async () => {
363363
const payload: GoogleDriveWebhookPayload = {

apps/sim/lib/webhooks/polling/google-sheets.ts

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -182,17 +182,20 @@ export const googleSheetsPollingHandler: PollingProviderHandler = {
182182
logger
183183
)
184184

185-
// Update state: advance row count by the number we fetched (not total new rows)
186-
// so remaining rows are picked up in the next poll.
187-
// When batching (more rows than maxRowsPerPoll), keep the old lastModifiedTime
188-
// so the Drive pre-check doesn't skip remaining rows on the next poll.
189-
const newLastKnownRowCount = config.lastKnownRowCount + rowsToFetch
190-
const hasRemainingRows = rowsToFetch < newRowCount
185+
// Advance row count only by successfully processed rows so failed rows
186+
// can be retried on the next poll cycle. Idempotency deduplicates the
187+
// already-processed rows when they are re-fetched.
188+
const rowsAdvanced = failedCount > 0 ? processedCount : rowsToFetch
189+
const newLastKnownRowCount = config.lastKnownRowCount + rowsAdvanced
190+
// When batching (more rows than maxRowsPerPoll) or retrying failed rows,
191+
// keep the old lastModifiedTime so the Drive pre-check doesn't skip
192+
// remaining/retried rows on the next poll.
193+
const hasRemainingOrFailed = rowsAdvanced < newRowCount
191194
await updateWebhookProviderConfig(
192195
webhookId,
193196
{
194197
lastKnownRowCount: newLastKnownRowCount,
195-
lastModifiedTime: hasRemainingRows ? config.lastModifiedTime : currentModifiedTime,
198+
lastModifiedTime: hasRemainingOrFailed ? config.lastModifiedTime : currentModifiedTime,
196199
lastCheckedTimestamp: now.toISOString(),
197200
},
198201
logger

0 commit comments

Comments
 (0)