Crawlee & Apify•4w ago

Dynamically change dataset id based on root_domain

Hey folks. I've attached an example of my code as a snippet Is it possible to dynamically change the dataset id so that each link has it's own dataset?

message.txt

3 Replies

Hall•4w ago

Someone will reply to you shortly. In the meantime, this might help:

automatic-azure•4w ago

Hey @Rykari Note the Dataset class - https://crawlee.dev/python/api/class/Dataset You can open different Datasets in handlers and write data to them

Dataset | API | Crawlee for Python · Fast, reliable Python web cra...

Crawlee helps you build and maintain your Python crawlers. It's open source and modern, with type hints for Python to help you catch bugs early.

sunny-greenOP•4w ago

Well, that was easy. Ty Changed

            await context.push_data(data)

            await context.push_data(data)

            hostname = urlparse(context.request.url).hostname
            dataset = await Dataset.open(name=hostname)
            await dataset.push_data(data)

            hostname = urlparse(context.request.url).hostname
            dataset = await Dataset.open(name=hostname)
            await dataset.push_data(data)

Gaming

Programming

Dynamically change dataset id based on root_domain

Did you find this page helpful?