C
C#•4mo ago
Scald

How to mass upload a list of files to blob storage?

I have a large number of files I need to upload to a blob storage, and I'm currently using the BlobClient.UploadAsync(filePath, options); method to accomplish this. However, it's too slow, probably because I'm uploading files one by one. I've seen some recommend the Microsoft.Azure.Storage.DataMovement for this use case online, that being a bulk upload of files. However, my use case requires me to filter the files by matching with a CSV file first, which maps the current file names to the new names, and to avoid unnecessary files. Has anyone have had to deal with a similar use case before? Bulk uploading files, having to filter them first? I'd rather not have to copy all the files to a new directory if I could avoid it.
1 Reply
lycian
lycian•4mo ago
Without knowing too much about azure APIs, is there a reason you can't do multiple BlobClient.UploadAsync calls and wait for them? Batch into 10s, 20s, etc The other option I would think of is to use Microsoft.Azure.Storage.DataMovement to transfer a folder by first copying the files you want to transfer there, then uploading the whole folder
Want results from more Discord servers?
Add your server
More Posts
✅ How to solve "[MyType] cannot be cast to [MyType] due to different assembly versions.This is due to my lack of knowledge around how Visual Studio manages references, I think. I have a c✅ How to properly set a default value as UtcNow with EF Core and PostgreSQLHi there, I use EF Core with a PostgreSQL DB. I want that each time I add an entity with EF Core in✅ ASP Core Attribute Logic HelpI need assistance with structuring my controller attribute logic (they're authorization attributes) Moving local files onto docker containerHi I have files in folder c:/dev and I have a dockerfile in c:/dev/scripts/docker/dockerfile and ✅ Settting localizable in all formsIs there a way to do this without manually going into each designer ?✅ Using IProgress with await Task.WhenAll(tasks);I am trying to introduce a IProgress object into the code below. Can't for the life of me figure how