ambitious-aqua

How can I automatically get the dataset id from instagram post scraper into Merge, Dedup & Transform

As above, so im using the instagram post scraper once a week, and then i used to use code to get only the fields i needed. i only need 3 fields, then i push this in make (integromat) to do some automated tasks. But now the instagram post scraper has stopped allowing the code to remove the 100 values i dont need. So ive been advised to use Merge, Dedup & Transform Datasets to get the data into the needed format, but i cant work out how to get this all to run automatically? So the flow would be, run instgram post scraper. somehow get the dataset id from the sucessful run into Merge, Dedup & Transform Datasets use that to remove all dataset item except 3 then fire the webhook into make thanks!

8 Replies

Pepa J•2y ago

Hello @Jacpat , Sorry for a late answer, as I understand it you wanna setup a webhook for sucessfull run on instagram scraper to call the MD&T Datasets actor. 1. I suggest you to go to the MD&T Datasets actor, and click the API button and select API endpoints -> then copy the Run Actor url with the copy button so it would copy it with your token and not with the asteriks symbols. 2. Go to Instagram scraper Actor -> Integrations -> and setup a new HTTP Webhook on run sucess 3. Copy the run actor utl to the URL field and setup payload by your needs:

{
    "datasetIds": [
        {{resource.defaultDatasetId}}
    ],
    "mode": "dedup-after-load",
    "postDedupTransformFunction": "async (items, { Apify }) => { return items.map((item) => ({ field1: item.field1, field2: item.field3, field3: item.field3 })); }"
}

{
    "datasetIds": [
        {{resource.defaultDatasetId}}
    ],
    "mode": "dedup-after-load",
    "postDedupTransformFunction": "async (items, { Apify }) => { return items.map((item) => ({ field1: item.field1, field2: item.field3, field3: item.field3 })); }"
}

Not sure which fields you wanna use but, feel free to change their names based on your needs.

ambitious-aquaOP•2y ago

thanks so much Pepa J! Im getting an error on the dataset id variable though? im new to all of this but couldnt find an answer in the docs? thanks

ambitious-aquaOP•2y ago

also do you know anyway to set the memory usage option in the payload? thanks so much, pulling my hair out with this

MEE6•2y ago

@Jacpat just advanced to level 1! Thanks for your contributions! 🎉

ambitious-aquaOP•2y ago

@Lukas Krivka is it possible to set the memory usage of this actor via a payload? thanks

Pepa J•2y ago

@Jacpat You may setup the memory with parameter in url, check https://docs.apify.com/academy/api/run-actor-and-retrieve-data-via-api#additional-settings Have you been successful with the webhook otherwise?

Run Actor and retrieve data via API | Apify Documentation

Learn how to run an Actor/task via the Apify API, wait for the job to finish, and retrieve its output data. Your key to integrating Actors with your projects.

ambitious-aquaOP•2y ago

Thanks Pepa, that solves one issue. No sadly the above {{defaultDatasetId}} as shown in the screenshot throws and error and will not save, and i cant figure this out. Obviously the dataset id is an integral part sadly. any ideas? thanks so much

Pepa J•2y ago

Hi @Jacpat, I am sorry now I see the value should be like:

    "datasetIds": [
        {{resource.defaultDatasetId}}
    ]

    "datasetIds": [
        {{resource.defaultDatasetId}}
    ]

Gaming

Programming

How can I automatically get the dataset id from instagram post scraper into Merge, Dedup & Transform

Did you find this page helpful?