CA
Crawlee & Apify•12mo ago
memo23

File size upload to KeyValueStore

I created actor which downloads and uploads videos to KeyValueStore, some of videos are pretty big and getting this error: 2024-05-18T15:54:15.572Z Failed to upload video: RangeError [ERR_FS_FILE_TOO_LARGE]: File size (2183982088) is greater than 2 GiB so anyway to get around this and what are best practices on apify for downloading and uploading of big files
1 Reply
correct-apricot
correct-apricot•12mo ago
Hi, 2GB is the node's limit for reading files. You can avoid it by using streams: 
import got from 'got'
import stream from 'node:stream'

const headers = { 'Content-Type': contentType }
await streamPipeline(
    got.stream.get(DOWNLOAD_URL, { headers }),
    new stream.PassThrough(),
    got.stream.put(UPLOAD_URL, { headers }),
)
import got from 'got'
import stream from 'node:stream'

const headers = { 'Content-Type': contentType }
await streamPipeline(
    got.stream.get(DOWNLOAD_URL, { headers }),
    new stream.PassThrough(),
    got.stream.put(UPLOAD_URL, { headers }),
)

Did you find this page helpful?