Nightly sync misses new data
I have some strange edge cases where there may be old values on a Shopify model. But the Shopify updated at is the latest value (updated).
So the nightly sync only seem to check for newer ones so in one cases had for example the old ”title” that was updated one year ago.. 🤦🏻♂️
I can see the logic that its not a bug on the sync but its not doing what we think and hence in our case have a huge negative effect. We might need a different sync 🤔
In the other hand.. its hard to debug why it only saved the update at but not the ”title”.
Do Gadget save that before action update?
Maybe it crashed somehow.. and think its updated 🤔 dont have a timestamp in the near past sadly.
4 Replies
This has happened more than once so trying to understand why it got in that state.. but not sure its happening in present or bad code in the past by gadget/me 😅
Can you send the record ID and when you think it was updated erroneously (timestamp, or better yet the trace ID of the event that cuased the fault update where the time changed but not the title)
Im gonna keep talking in sync vs reconcile cause i really hate that our team made the two terms so interchangeable.
Webhooks are reconciled once a day by us running our own Sync API but with additional date filters. Reconciliations only correct missed events over the last 25 hours. But this strategy should not leave any records behind so my gut says something bugged out
But yes, reconciliation syncs are and have always only been on the lsat 25 hours of data, not everything since the start.
Hi i finally pinpointed to a newer trace id and could figure it out.
Nothing wrong with your syncs and or reconcilation, sorry 🙂
I save all historic values in a custom model, when a timeout or sandbox error occurs it ignores all other variants:update because i have my save-logic in the onSuccess and treats it (as it should) as completed.
The thing is if i put it in the onRun, it will more or less always timeout since the 5s timeout was put in place. So we cant have a satisfiyng record here..
Do you need some help figuring out a better structure for this?
Did you mean the 15s timeout of the action or the 5s timeout of the run function?