RunpodR
Runpod2y ago
3 replies
Mandelion

Is there any way to do dynamic batching?

Say I have a vision model deployed and I send 5 images within x time is there a way to actually stack the images, pass them through the model and return the 5 responses? I was able to find concurrent handlers etc. but nothing actual batching (other than sending them all in the same request of course)
Was this page helpful?