CA
Crawlee & Apify•3y ago
plain-purple

Dataset.pushData()

I am trying to push 2 separate datasets into (2 seperate folders); first is just ids, the second is the whole object with all the data. After running, it creates a _ crawlee_temporary_0_ folder with only the second set inside and I also get an error saying
ENOENT: no such file or directory, storage/dataset/default/00000022.json but i think it doesn't get that path because it created the _ crawlee_temporary_0_ folder I am inside a forEach() with await Dataset.pushData({ id: it.id }); await Dataset.pushData(it); Why is it not working properly? what am i missing/doing wrong?
1 Reply
fair-rose
fair-rose•3y ago
The first issue is that you're trying to run async code within a forEach. It will never be awaited - I recommend using a for...of loop instead. See more about this here: https://stackoverflow.com/questions/37576685/using-async-await-with-a-foreach-loop The Dataset.pushData static method will always push to the default dataset, so if you're trying to push to different datasets, you should use the Dataset.open() method, then run operations on the Dataset instance returned by that. Example: 
// let's say this is the data you want to push
const array = [{ id: 123, name: 'foo' }, { id: 3, name: 'bar' }];

const namedDataset = await Dataset.open('some-name');

// for every item in the array of data
for (const item of array) {
    // push object with just the ID into the default dataset
    // and push the full item into the named dataset
    await Promise.all([Dataset.pushData({ id: item.id }), namedDataset.pushData(item)])
}
// let's say this is the data you want to push
const array = [{ id: 123, name: 'foo' }, { id: 3, name: 'bar' }];

const namedDataset = await Dataset.open('some-name');

// for every item in the array of data
for (const item of array) {
    // push object with just the ID into the default dataset
    // and push the full item into the named dataset
    await Promise.all([Dataset.pushData({ id: item.id }), namedDataset.pushData(item)])
}
Stack Overflow
Using async/await with a forEach loop
Are there any issues with using async/await in a forEach loop? I'm trying to loop through an array of files and await on the contents of each file. import fs from 'fs-promise' async function print...

Did you find this page helpful?