How to disable storage directory creation?
Hi!
I tried to run multiple playwright crawlers in parallel and each crawler was conflicting with each other. I want to store all data (request queue, datasets, key value stores) in memory. How can I do it? I tried to set
persistStorage configuration option to false (it was mentioned in some discussion on GitHub), but it has no effect. Also I tried to set defaultKeyValueStoreId, defaultRequestQueueId and defaultDatasetId for each crawler. I thought that I will get separate directory for each crawler, but Crawlee creates storage/key_value_stores/default, storage/request_queues/default directories.
NodeJS version: 18.13.0
Crawlee version: 3.1.42 Replies
xenial-black•3y ago
you need to create multiple requests queues and add them to the constructors of each crawler
ambitious-aquaOP•3y ago
Thank you, it helped.