Crawlee & Apify•2y ago

ForceCloud

I find the docs around the forceCloud behavior confusing. Is this feature still available? None of these statements from the SDK docs seem to hold true currently, with the SDK:

Note that you can force usage of the cloud storage also by passing the forceCloud option to Dataset.open function, even if the APIFY_LOCAL_STORAGE_DIR variable is set.

Dataset stores its data either on local disk or in the Apify cloud, depending on whether the APIFY_LOCAL_STORAGE_DIR or APIFY_TOKEN environment variables are set.

Also, what is StorageManagerOptions as referenced in the docs? https://docs.apify.com/sdk/js/reference/class/Dataset#open

Dataset | API | SDK for JavaScript | Apify Documentation

The Dataset class represents a store for structured data where each object stored has the same attributes, such as online store products or real estate offers. You can imagine it as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - you can only add new records to it but you cannot modify or...

6 Replies

continuing-cyan•2y ago

Hello @Tobey, thank you for reporting this I will pass the information on to the team. The forceCloud option is still available if using the Actor.openDataset function.

rare-sapphireOP•17mo ago

Hello @vojtechmaslan thanks for the follow-up. Curiously, I did not find the method you mentioned through the docs search and the documentation (leftovers) were quite confusing. I can confirm that the cloud option on this method does work.

MEE6•17mo ago

@Tobey just advanced to level 3! Thanks for your contributions! 🎉

rare-sapphireOP•17mo ago

Actually, no. It does not work either. It's still pushing data to local dataset instead of the remote. Looks like none of the force cloud options actually work. @vojtechmaslan

Pepa J•17mo ago

Hi @Tobey , I just tried minimal working example and it works without issues for me, here is my code:

import { Actor } from 'apify';

await Actor.init();

const myDataset = await Actor.openDataset("my-happy-dataset", { forceCloud: true });
await myDataset.pushData({
    my: 'data'
});

await Actor.exit();

import { Actor } from 'apify';

await Actor.init();

const myDataset = await Actor.openDataset("my-happy-dataset", { forceCloud: true });
await myDataset.pushData({
    my: 'data'
});

await Actor.exit();

and run it via apify run I also see the newly created dataset under my user account:

rare-sapphireOP•17mo ago

Ahoj @Pepa J that's really strange. Thanks for checking it out though. My priorities shifted but I'll get back to this for sure and report back.

Gaming

Programming

ForceCloud

Did you find this page helpful?