S
Supabase•4d ago
nickynunchuck

Preferred way of inserting many files to Storage?

Hi, what's the preferred way of inserting large amount of files to storage? I have a small Python script using just 5 threads to insert into storage, i.e. 
supabase.storage.from_(bucket_name).upload(
            new_filename,
            file_data,
            file_options={"content-type": "image/jpeg", "upsert": "true"},
        )
supabase.storage.from_(bucket_name).upload(
            new_filename,
            file_data,
            file_options={"content-type": "image/jpeg", "upsert": "true"},
        )
And despite the relative low load on Supabase, I constantly get [Errno 35] Resource temporarily unavailable errors. I don't see a way to bulk insert. Is there some other suggested approach aside from slow and serial?
3 Replies
silentworks
silentworks•4d ago
How large are the files and how often are you trying to upload?
nickynunchuck
nickynunchuckOP•4d ago
1-5 MB per file and I am uploading concurrently with about 5 workers I am correct that a bulk upload is not available right?
silentworks
silentworks•4d ago
You should probably use resumable uploads for better uploading of many files, this is what Supabase uses in their Dashboard for their storage uploads.

Did you find this page helpful?