400 Errors with allenai-olmocr on Serverless SGLang - Need Payload Help!
I'm trying to deploy the allenai/olmOCR-7B-0225-preview model (fintuned Qwen/Qwen2-VL-7B model) RunPod using the Serverless SGLang endpoint template, but I'm consistently getting 400 Bad Request errors when sending requests. running on L40S.
I'm trying to send PDF documents for OCR, and I hope the issue is with the input payload. I've tried various common input formats based on the RunPod documentation and examples, but no luck so far. I've tried sending as a pdf file & page number as well as what I originally tried (pdf anchor text and image). in the code below, I am using the retrieved https://molmo.allenai.org/paper.pdf
I'm using the allenai-olmocr model (Hugging Face link: https://huggingface.co/allenai/olmOCR-7B-0225-preview), deployed as a Serverless SGLang endpoint on RunPod. I deployed it the lazy way, providing huggingface handle and mostly default settings, and am wondering if I need to set up a handler and deploy using docker to get to work?
I've checked the RunPod documentation for Serverless requests (https://docs.runpod.io/serverless/overview) and the olmOCR documentation and examples (https://github.com/allenai/olmocr), but I'm still struggling to get the input payload correct. I've mainly only tried sending base64, I experimented with s3 url but didn't seem to make a difference.
Could someone please help me understand the exact JSON payload format expected by RunPod Serverless SGLang for a multimodal model like allenai/olmocr? Specifically, I'm unsure about the correct structure for including both text (anchor text) and image data in the request, and can't find any clear answers. I was able to get this to work on replicate (using an existing setup https://github.com/lucataco/cog-olmocr), but it was very much point and click setup. Replicate was pretty expensive & slow.
I've checked the RunPod logs and they indicate a "Bad Request", pointing to an issue with the input data format.
Any guidance or example payloads would be greatly appreciated!
I'm trying to send PDF documents for OCR, and I hope the issue is with the input payload. I've tried various common input formats based on the RunPod documentation and examples, but no luck so far. I've tried sending as a pdf file & page number as well as what I originally tried (pdf anchor text and image). in the code below, I am using the retrieved https://molmo.allenai.org/paper.pdf
I'm using the allenai-olmocr model (Hugging Face link: https://huggingface.co/allenai/olmOCR-7B-0225-preview), deployed as a Serverless SGLang endpoint on RunPod. I deployed it the lazy way, providing huggingface handle and mostly default settings, and am wondering if I need to set up a handler and deploy using docker to get to work?
I've checked the RunPod documentation for Serverless requests (https://docs.runpod.io/serverless/overview) and the olmOCR documentation and examples (https://github.com/allenai/olmocr), but I'm still struggling to get the input payload correct. I've mainly only tried sending base64, I experimented with s3 url but didn't seem to make a difference.
Could someone please help me understand the exact JSON payload format expected by RunPod Serverless SGLang for a multimodal model like allenai/olmocr? Specifically, I'm unsure about the correct structure for including both text (anchor text) and image data in the request, and can't find any clear answers. I was able to get this to work on replicate (using an existing setup https://github.com/lucataco/cog-olmocr), but it was very much point and click setup. Replicate was pretty expensive & slow.
I've checked the RunPod logs and they indicate a "Bad Request", pointing to an issue with the input data format.
Any guidance or example payloads would be greatly appreciated!
