Topics

Supabase•2y ago

"Mime type not supported" error for jpeg files

I'm getting an odd error when trying to upload some files. I upload images to this bucket all the time, but for a couple of specific image files, I'm getting a Mime type not supported error. The mime type I'm uploading with is image/jpeg and I'm quite sure that the file is actually a jpeg (it starts with 0xFF 0xD8 as per https://en.wikipedia.org/wiki/JPEG_File_Interchange_Format) - Any ideas on how I can diagnose this?

80 Replies

Oli__OP•2y ago

Huh... searched the supabase GitHub org for that message looking at the source for validateMimeType - https://github.com/supabase/storage/blob/8fa98fab7716dccefb257b5fad8c3a9ba567df05/src/storage/uploader.ts#L227-L253 It really is just a string comparison. And my allowed MIME types are image/* so it seems like it must be the first part that's choking. Puzzling!

GitHub

storage/src/storage/uploader.ts at 8fa98fab7716dccefb257b5fad8c3a9b...

S3 compatible object storage service that stores metadata in Postgres - supabase/storage

Oli__OP•2y ago

Ok, new hypothesis: The file I'm uploading is actually being downloaded from another source, and that request may be failing.

Oli__OP•2y ago

No description

Oli__OP•2y ago

I was handling the case where fetch() throws an exception, but not the case where fetch successfuly retrieves an error status code from the server.

Oli__OP•2y ago

Yup that was it, lol. Thanks for the help everyone :RubberDucky:

No description

garyaustin•2y ago

Glad you found it... had no ideas and did not want to interrupt you train of thought.

Oli__OP•2y ago

Argh. Nope. That wasn't it. 😦

garyaustin•2y ago

Ahhh And the same file(s) always error?

Oli__OP•2y ago

Yeah, I added proper error handling and sentry notification to that missing error case, but it didn't get triggered. And in the sentry for the supabase upload failure, everything seems in order. Yeah, always the same file

Oli__OP•2y ago

My content type comes directly from that type field. (image/jpeg)

No description

Oli__OP•2y ago

And the link downloads fine, and is a valid jpeg image

garyaustin•2y ago

Can you try some mime analyzer like this? https://www.htmlstrip.com/mime-file-type-checker

Oli__OP•2y ago

The supabase storage log isn't really any help either Yup, image/jpeg according to that too

garyaustin•2y ago

Are you just doing an upload with OUT upsert?

Oli__OP•2y ago

await supabase.storage
            .from('item_photos')
            .upload(filename, blob, {
              upsert: true,
              contentType: photo.type,
            })
            .then(res => {
              if (res.error) {
                Sentry.captureException(res.error, { extra: { filename, photo } })
                return null;
              } else {
                return `@img/item_photos/${filename}`;
              }
            })

await supabase.storage
            .from('item_photos')
            .upload(filename, blob, {
              upsert: true,
              contentType: photo.type,
            })
            .then(res => {
              if (res.error) {
                Sentry.captureException(res.error, { extra: { filename, photo } })
                return null;
              } else {
                return `@img/item_photos/${filename}`;
              }
            })

garyaustin•2y ago

So any chance the file exists already?

Oli__OP•2y ago

That's the code that produces the sentry screenshot above Here's my put request from the supabase storage log for the record

[
  {
    "context": [
      {
        "host": "ip-10-109-64-83.ec2.internal",
        "pid": 1,
        "type": "event"
      }
    ],
    "err": [],
    "error": [],
    "event": "ObjectCreated:Put",
    "job": null,
    "jodId": "cc61f2c5-7e7a-419c-b505-0afbdfd08b2e",
    "level": "info",
    "objectPath": "ykkicgvsyjhmhrrupzpb/item_photos/attkbY0k1nozfwRU9.jpeg",
    "owner": null,
    "payload": "{\"name\":\"attkbY0k1nozfwRU9.jpeg\",\"reqId\":\"8341538bd17d1058-ORD\",\"tenant\":{\"ref\":\"ykkicgvsyjhmhrrupzpb\",\"host\":\"ykkicgvsyjhmhrrupzpb.supabase.co\"},\"bucketId\":\"item_photos\",\"metadata\":{\"eTag\":\"\\\"4654417098eb32b0f380f4ebf21c8360\\\"\",\"size\":2349667,\"mimetype\":\"image/jpeg\",\"cacheControl\":\"max-age=3600\",\"lastModified\":\"2023-12-11T22:57:25.000Z\",\"contentLength\":2349667,\"httpStatusCode\":200}}",
    "project": "ykkicgvsyjhmhrrupzpb",
    "rawError": [],
    "req": [],
    "reqId": "8341538bd17d1058-ORD",
    "res": [],
    "responseTime": null,
    "tenantId": "ykkicgvsyjhmhrrupzpb"
  }
]

[
  {
    "context": [
      {
        "host": "ip-10-109-64-83.ec2.internal",
        "pid": 1,
        "type": "event"
      }
    ],
    "err": [],
    "error": [],
    "event": "ObjectCreated:Put",
    "job": null,
    "jodId": "cc61f2c5-7e7a-419c-b505-0afbdfd08b2e",
    "level": "info",
    "objectPath": "ykkicgvsyjhmhrrupzpb/item_photos/attkbY0k1nozfwRU9.jpeg",
    "owner": null,
    "payload": "{\"name\":\"attkbY0k1nozfwRU9.jpeg\",\"reqId\":\"8341538bd17d1058-ORD\",\"tenant\":{\"ref\":\"ykkicgvsyjhmhrrupzpb\",\"host\":\"ykkicgvsyjhmhrrupzpb.supabase.co\"},\"bucketId\":\"item_photos\",\"metadata\":{\"eTag\":\"\\\"4654417098eb32b0f380f4ebf21c8360\\\"\",\"size\":2349667,\"mimetype\":\"image/jpeg\",\"cacheControl\":\"max-age=3600\",\"lastModified\":\"2023-12-11T22:57:25.000Z\",\"contentLength\":2349667,\"httpStatusCode\":200}}",
    "project": "ykkicgvsyjhmhrrupzpb",
    "rawError": [],
    "req": [],
    "reqId": "8341538bd17d1058-ORD",
    "res": [],
    "responseTime": null,
    "tenantId": "ykkicgvsyjhmhrrupzpb"
  }
]

It even says "mimetype":"image/jpeg" in there

garyaustin•2y ago

Going back to the upsert part. Is there a file already with that path/name?

Oli__OP•2y ago

no

garyaustin•2y ago

If you look in the table UI at storage schema and buckets table is the mimetypes column as you would expect?

Oli__OP•2y ago

For the files that are working, yeah

No description

garyaustin•2y ago

Not the objects table. The buckets table.

Oli__OP•2y ago

oh

garyaustin•2y ago

Just to make sure your bucket got set correctly.

Oli__OP•2y ago

Yup, that too

No description

garyaustin•2y ago

And right now other files upload? versus maybe they upload before you added the mimetype with /* as that is a newer feature.

Oli__OP•2y ago

Yeah, that's the weird part, this code path is working for thousands of files already

garyaustin•2y ago

since you did image/*?

Oli__OP•2y ago

Yeah that's been the mime filter the whole time

garyaustin•2y ago

Just one file or a couple?

Oli__OP•2y ago

select count(*) from storage.objects

select count(*) from storage.objects

3116

garyaustin•2y ago

No I mean bad ones.

Oli__OP•2y ago

Oh A couple Not many Maybe a dozen?

garyaustin•2y ago

Anything common about them that jumps out?

Oli__OP•2y ago

Nah they just seem like jpegs. Here's a problem file

No description

garyaustin•2y ago

Can you upload the file from the storage UI?

Oli__OP•2y ago

oh good idea Huh, yes I can!

garyaustin•2y ago

Ok sort of rules that the back end then.

Oli__OP•2y ago

Yeah

garyaustin•2y ago

Check the files mime type in storage.objects.

Oli__OP•2y ago

No description

Oli__OP•2y ago

I keep coming back to this... this is the exact message I'm hitting, and it's the only place it appears anywhere in the supabase GitHub org, so it's gotta be the code path that's being hit https://github.com/supabase/storage/blob/8fa98fab7716dccefb257b5fad8c3a9ba567df05/src/storage/uploader.ts#L252

GitHub

storage/src/storage/uploader.ts at 8fa98fab7716dccefb257b5fad8c3a9b...

S3 compatible object storage service that stores metadata in Postgres - supabase/storage

Oli__OP•2y ago

And the only way that gets hit is some string equality checks Checked for trailing whitespace and stuff The source of this data is the Airtable API, and they auto-detect the mime type too, so it's not like it's a potential user error thing.

garyaustin•2y ago

The fact that it uploaded thru the storage UI.....

Oli__OP•2y ago

I know, everything in my experience tells me I'm making a dumb mistake somewhere but I can't find it 😭

garyaustin•2y ago

The code from storage you show seems pretty strait forward if mimetype is correct. Any idea how that was obtained? I did not follow the trail.

Oli__OP•2y ago

It's so weird, I don't see how the mime type gets mangled here.

No description

No description

Oli__OP•2y ago

Not sure what you mean, where what was obtained? I wrote the code

garyaustin•2y ago

No you showed Supabase storage code. it gets mimeType from somewhere, just not sure if it is convoluted or trusts what you fed in.

Oli__OP•2y ago

Ohh, that part, yeah 1 sec

Oli__OP•2y ago

Yeah I don't have a good way to demonstrate it in code, but basically, the payload I'm processing is a webhook that I receive from Airtable. When someone updates a record, Airtable posts me a webhook, and I reflec the edit in Supabase. (I mirror the database so that it's fast for the website to query) So this photo variable comes directly from Airtable. I can't link directly to the API docs, because they're behind auth and customized for my database, but here's a screenshot of what it says about the photo field

No description

Oli__OP•2y ago

So that photo.type variable comes from there

garyaustin•2y ago

Hey. I'm saying I did not follow the storage-api code to see how they get mimeType in that call.

Oli__OP•2y ago

Oh, lol, sorry

garyaustin•2y ago

I for some reason thought blobs had the filetype in them.

Oli__OP•2y ago

No, it seems like it's just a string comparison between whatever I say the mime type is and what's allowed on the bucket Regardless of what's in the blob

garyaustin•2y ago

K. Then it works!

garyaustin•2y ago

Sorry. So looking at this:

No description

garyaustin•2y ago

just seems odd line break.

Oli__OP•2y ago

There's not a line break there, that's just word wrapping deciding that there's a word boundary there. You can see the same thing above with eTag

garyaustin•2y ago

Yeah, I got that, but eTag continues on versus the , after mimetype Just throwing darts though.

Oli__OP•2y ago

Is that from this^ message? Try resizing the discord window

garyaustin•2y ago

yep good.

Oli__OP•2y ago

idk, I'm looking at the supabase docs, my content variable is actually a Blob type so maybe I don't need to specify the content type anyway

No description

garyaustin•2y ago

That is what I thought. But still that would imply Supabase storage uses the blob info instead of what you send.... which could be the case.

Oli__OP•2y ago

Yeah

garyaustin•2y ago

But the problem with all of this is when you used the UI it said the mimetype was image/jpeg, which I assume came from the blob.... sigh

Oli__OP•2y ago

Yeah, no such luck, even leaving the contentType field blank

garyaustin•2y ago

I'm out of time. Hard for you to file a bug since the UI works. You would have to narrow down the code to reproduce to likely get any traction in github issues for storage.

Oli__OP•2y ago

Thanks for all the help! I really should take a break for dinner anyway Oh fml, I figured it out. Because I'm passing in a Blob type, which includes its own content type, the Supabase SDK was actually ignoring the contentType I was passing in explicitly, and instead using the value that was in the blob. Which was. Drumroll. application/octet-stream For some reason, Airtable knows what the mime type is in the API record, but when I actually download the image with fetch() it returns an application/octet-stream Not for most photos, but for some reason, it does with these ones. These photos come from a variety of sources, so maybe there's something funky with the image data after all, and something somewhere in the guts of airtable is getting mixed up about it. But I just assumed that if I declared the content type explicitly, supabase storage would use that one. But the content type in the Blob variable actually takes precedence, and the one passed in explicitly is ignored. TIL! I think I'm gonna make a PR to the supabase/storage repo to include what mime type failed. I think of the message had said Mime type application/octet-stream not supported I would have hunted this down pretty quickly. But I couldn't find anywhere what mime type it was actually trying to evaluate. Heyooooooo it works! 🎉

          return await supabase.storage
            .from('item_photos')
            .upload(filename, blob.stream(), {
              upsert: true,
              contentType: photo.type,
              duplex: 'half'
            })

          return await supabase.storage
            .from('item_photos')
            .upload(filename, blob.stream(), {
              upsert: true,
              contentType: photo.type,
              duplex: 'half'
            })

1. Pass in blob.stream() instead of just blob 2. Specify duplex: half because I guess that's required by fetch, and I read that passing a fetch stream into another fetch stream should be done at half duplex.

garyaustin•2y ago

Wow. No way to tell airtable to return the actual filetype? Could it be the way you are fetching the file. And why just the few files? Would they not all be bad?

Oli__OP•2y ago

So many questions, so few answers 😭

garyaustin•2y ago

The fact that the same file you downloaded from airtable(?) can be uploaded from the UI seems to point to the download process in your code. But if you got it working.

Oli__OP•2y ago

Yeah, you're right, it basically was in my code, just in a really sneaky way. When I downloaded the file in the browser, it basically threw out the Content-type: application/octet-stream at that point, and then wrote a jpeg file to disk. Then when I went to upload it to the supabase UI, the browser looked a the jpeg file on disk and went "Oh yeah, that's content-type: image/jpeg" But in my code, fetch() returned a blob, which includes not just the file data, but also the content-type header returned from airtable. So I have this blob, which is basically {type: "application/octet-stream", body: ...} And then when I passed that into the supabase SDK, it went "Oh I see, this is an application/octet-stream " and that's where I got the error.

garyaustin•2y ago

But only for some of the image files does airtable return the octet-stream? Sorry, I should just leave you in peace with a solution.

Oli__OP•2y ago

Haha all good. Yeah that's right, airtable was only doing it for some files. I have no idea why, and I don't think I have any ability to find out, some weirdness in the internal machinery of airtable.

garyaustin•2y ago

https://community.airtable.com/t5/development-apis/gallery-api-returned-image-type-not-matching-http-content-header/td-p/162376

Gallery API - Returned Image Type not matching HTTP-Content header ...

Hello all, I'm developing integrations with other third party providers where I can supply a URL and they ingest that information. However, I was told there were errors with the image. On investigation, the "type" provided by the API (image/jpeg). However, the HTTP Content type is actually binary (a...

Oli__OP•2y ago

HA Yup!

garyaustin•2y ago

2 hits on same thing 6 months apart and no answers. I'll stop there.

Oli__OP•2y ago

https://github.com/supabase/storage/pull/403

GitHub

Improve invalid_mime_type error to report the attempted mime type b...

What kind of change does this PR introduce? Feature What is the current behavior? When uploading an object to supabase storage, the mime type is checked against the allowed mime types on the bucket...

40KMembers

View on Discord

Did you find this page helpful?