Is it possible to reduce shopifyCollect resource usage?
For a client's project we are seeing 85% of request time going to shopifyCollect sync. Last month that came to 85 hours of request time, which brought us over our 100 hour plan limit. Our app needs to know at the product level which custom collections it belongs to. We were using the shopifyCollect API to find records, which I've refactored to use shopifyProduct. However, shopifyProduct has a relationship on shopifyCollect on the customCollections field, so it seems our app is still reliant on shopifyCollect.
My question is is there a way to reduce the amount of request time for shopifyCollect or does our client just need to pay for more usage?
9 Replies
Hello,
When it comes to collections and collects, things get a little tricky. Shopify doesn't send webhooks for the collect model so we're forced to run a sync nightly to make sure that we have the updated data (no charge comes to you if there's no update to make).
Does the client make a lot of changes to collections?
Do you only need specific collections?
What kind of work are you trying to do with that collect data?
It is for this app
ilovespeelgoed-collections-tree.gadget.app. On the production environment I see a lot of new creates.. It looks like this has something to do with the large amount of products (65.000) and the way how they are organized.
I think it might be possible that it simply takes quite a while before the system has filled up the local cache.. I'm not sure if the customer changes this so often (the app is not used yet, so changes will have to come from other processes them)
I think we have to check with the client how often they change this.. but again, it might also be possible that it simply takes a long time before the cache is filled upOnce the collect data is in the database, you shouldn't really see too much usage on that model unless there are a ton of changes to the collections. I gather this was just an initial jump in usage from syncing data or is that not what you're seeing/describing?
We had one store on a public app that used some other app which was revising collections all the time using "AI". We used to spend several hundred hours a month in Gadget Compute on just on one client's Plus store's shopifyCollect... So for our purpose we threw away the shopifyCollect and instead ran a manual method that gathered the products for a collection direclt via shopify gql call; for specific products on certain conditions.... Of course depends on the exact use case you have, but for our case it was possible to skip the model with some custom implementation.
@[Gadget] Antoine looking at historical data for as far back as it goes, we see roughly the same request times for collect month after month. As of right now the store has 3, 594 custom collections.
Hey folks, I'm going to keep looking into the issue we've been discussing in this thread but I would like to point out a few different things that could cause issues in other parts of your application.
You have highly outdated packages:
-
@gadgetinc/react-shopify-app-bridge (can be upgraded via the command palette)
- shopify-api-node (recommend ^3.15.0)
- vite (I'd recommend ^6.3.5)
^ I'd also recommend moving to React 19.
You should not have the @gadgetinc/api-client-core package in your application as this is handled by the Gadget platform and can cause conflicts in yarn.
PS. Please don't ping (@) Gadget employees unless its an emergency or we haven't responded within 24 hours, during our 9-5 Mon-Fri EST work week
I would like to also propose that you delete the globalShopifySync and scheduledShopifySync actions from the actions folder. They aren't needed at all (scheduledShopifySync is only needed if you want to modify the nightly reconciliation sync logic)
Alright, I have an explanation.
Essentially, this usage is expected. shopifyCollect is a sync only model and doesn't have a corresponding resource in Shopify (anymore). The Shopify API has a limitation in this way; it leaves the need for us to fetch all collect records any time a collection (I also think product) has a change so that we can verify if we have the correct collects in your table.
We do want to add some sort of filterability and sortability on our end but it wouldn't change the shortcoming from the Shopify API.
The issue is especially present in your application because of the large number of collections in your database: 3602Yep, that's exactly the behavior we saw. Every few minutes this "other app" made some change to a manual collection, which made gadget start a resync of the whole collect. So we were essentially resyncing everything all the time š
I'm going to mark this thread as resolved. Please feel free to reopen it by writing a new message
Do you like this answer?
ā