Syncing of data from Shopify to Gadget
If I remember correctly, there's a once a day syncing of data from shopify to gadget right? If I use includeFields in the update webhook of a model, will it only update the fields that I put in the includeFields? Thanks!
58 Replies
Yes, that's correct. And we willy sync for webhooks that match your filter
Hmmm is there other way to do it without having to put all the fields? I just needed to check if there's a change in one field but I still want the other fields to be synced.
You can simply remove the fields from your model
The field is a json format so there's a lot of field there. And the field might be used in the future if ever
Is checking the record from webhook then gadget will not suffice?
In that case, there's nothing you can do. If the field itself is JSON and you want a key on that field, filters and include fields won't affect that
Oh so even the include fields ain't gonna work
Just wanted to ask, how does the computing of resources work? What if I have a lot of updates in the company locations, is the time to finish it matters or the count per location?
Request time usage is the amount of time that any of your code runs. The webhook handling itself doesn't cost anything. Its only when the action code runs (run/onSuccess) that you'll be charged. If that code takes 5 seconds to run on average and you have 1 million actions running, you'll have incurred 5 million seconds of request time
Is that what you were asking about?
Yes. Thank you. I was finding a way to fix the 380+ hours in just 1-2 days.
What could be the issue with this high request time? ALthough I don't think my webhook process that much
Does mismatch api version can cause this?
That's the only thing i'm seeing in logs
No, mismatched API versions wont affect request time. Have you looked at what stores are making the webhooks come in? Also note that you should probably add webhook filtering so that you only get the changes you care about
This is just for one store. But have development and prod environment not sure if those will sum up.
But can I filter the json column?
Does the request time in development included in the total usage?
What kind of filter? A webhook filter or a filter on an api call?
Yes
Webhook filter. The one you recommended
What will happen if the same webhook is used in two different models?
Cause I just observed, the shopifyCompanyLocation model does not update the record of what i'm updating so it was not triggered at all. From what I see in logs it is creating/updating records in the shopifyCompanyAddress
I'm not sure if webhook filters can be done on a JSON field but you should try it out
Does timeout and retrying consumes a lot of request time?
Also if it's not enqueue do it consume a lot too?
Yes, again, any time your code is running you're charged for it
If you have something waiting to run (or are getting rate limited by Shopify) it will cause your actions to sit
The process is fast like it updates literally like less than a second per location but it consumes 5hours of request time in less than 5 minutes??
so 5hours of request time is not equivalent in real life time?
If your actions need to rerun, they take a long time, there are a lot of requests, etc (things of that sort) your request time will continue to go up. Its real calculated time (from the start to the end of the action runtime) for any instance of the code running
I'm wondering why is it adding up in the webhook trigger though
Whats the name of your application?
I'll just give you a breakdown
b20f3b-60.myshopify.com
Your average billing duration for that action over the last 1.5 days was 4 seconds
company_locations/update - Webhooks: 2,114,898,395.376ms
I don't understand why it's that big
From the graphs, it doesn't look like an issue on Gadget's end
Let me take a look at your code
this is the initial call
So that could definitely cause that
then this where the actions that updates it
You're making requests to the Shopify API without enqueuing them
I would also not recommend doing a for loop for requests
You should use something like p-map if you're trying to run many requests at once
Was planning to do this earlier
Will that lower the request time?
I'll try to enqueue it
This is where i'm going most likely
p-map should lower your request time as you can then do concurrent requests to enqueue actions
enqueuing will lower your request time because you're no longer being rate limited by shopify and waiting to make more calls
Hi what is the recommended concurrency for p-map and for maxConcurrency for enqueue? Does the two conflict each other? Thank you
Concurrency for p-map is the concurrency of the enqueues. The maxConcurrency of the enqueued actions is what affects the amount of calls that you make to Shopify at a given time. The p-map concurrency can be way higher than the maxConcurrency of the queue
Is this okay? maxConcurrency: 100, concurrency (p-map): 250
I really don't know if it's too much or not
100 is way too high for shopify rate limiting
I would say 25 for p-map and 10 for the queue
Okay will do this
You have to keep in mind the memory usage of your action cause 250 concurrent requests will take up some space
You can also always tweak it
Okay okay. Thanks. I'm monitoring my request time in the usage and it's having like 1hr per run in just development
I believe that's too big
Even with the changes? That must be something else then
How many requests are you making to the Shopify API in the enqueued action?
Per action
I'm updating all the locations so around 11k locations
In 1 action runtime, how many times do you call the Shopify API?
In the enqueued action, i'm skipping the mutations if no change in value
Including the fetch?
You fetch from Shopify?
Yes
Why not get the data from your database?
And yes, including the fetch
Its a call to the shopify api
Total of 3. if no change made, just 1
Hmmm well first the company locations model in shopify is not updating when updated from mutation not sure why but there is a webhook placed in the create,update, and delete
Ok, so you have to think of each action as using about 1/2 to 1.5 of the a lotted requests per second that you can make to the Shopify API. You might have to lower your queue's concurrency
Btw this is not the enqueued.
If the enqueue, it's 2 mutations that needed to change in shopify
Each action is probably sitting there waiting for the Shopify API to send back or accept your request
One is a fetch for checking the current value from shopify. I'm calling a metafield's value
This is some background information for you to take action on. Shopify will literally let your request sit in limbo while you make too many calls to it. That causes your action to sit running + your request time to go up. Sometimes slow is better in this case
You want to make sure that you do things at a rate that won't cause your actions to sit waiting for promises to resolve
Okay okay. I think the 25:10 is good. It added like less than .20hr
Will run again just to be sure
Noted on this. I thought the higher the better since I'm using enqueue and p-map
You can go higher for p-map concurrency to make your first action finish faster but when it comes to making calls to the Shopify API, slow is preferable
I would also recommend that you try to determine a dynamic maxConcurrency because different shop plans have different rate limits and Plus shops might allow you to get through data much faster
Dynamic maxConcurrency like set it depending in shop plans rate?
But that's a little more advanced so it would be a good idea to just start with a static concurrency
Okay okay. I will keep that in mind. Thanks again!!
No problem!