Runpod•13mo ago

I don't know my serverless balance goes down

Hi, I recently made some changes to my platform. It analyzes videos using 3 different computer vision models.

I have a serverless endpoint for each. I think that somewhere I am making some requests that I should not be making, or that some endpoints are active when they should not.

For example, this happened from 14:20CET to 15:15CET today. I had $85.443 and when I came back I had $83.934. I was doing other tests on my app during this time, but I wasn't calling any endpoint here.

I've lost quite a bit of money because of this now. Could someone help me find or debug this? I am not able to replicate this balance drop.

Jason•12/10/24, 3:22 PM

Hey, Maybe check on billings > serverless

JJason Hey, Maybe check on billings > serverless

galakurpismo3OP•12/10/24, 6:15 PM

This is the endpoint that has abnormally high usage costs in the last 2 days is: t94jq90tvw8v0u (image A)

These are its usage metrics, they seem normal (Image B)

These are the logs during the time where I got abnormally high costs, nothing shows (image C)

I noticed that a worker kept running for a long time after a test run that I recently did. It showed an error message somewhere, related to some daemon thing.

Maybe that's what makes these costs high, but I don't know since nothing else appears on these logs and I didn't get the error again.

What can I do? I enabled execution timeout but I don't know if it will solve it: Here are the endpoint settings (Image D)

Jason•12/10/24, 10:54 PM

Okay what error Is it can you copy paste that one and maybe if it's from your code search your code for relevant thing

Jason•12/10/24, 10:56 PM

Or maybe let's open a ticket so staffs can check

Ggalakurpismo3 This is the endpoint that has abnormally high usage costs in the last 2 days is:...

PoddyAPP•12/10/24, 10:56 PM

@galakurpismo3

Escalated To Zendesk

The thread has been escalated to Zendesk!

Jason•12/10/24, 10:56 PM

Send your endpoint ID(s)

JJason Send your endpoint ID(s)

galakurpismo3OP•12/12/24, 3:41 PM

Hi, I opened the issue on zendesk too and I think that we solved it. But I have another question.

I want to process a video using an AI model. In order to make this quicker, I separate the video in different clips and send each clip to a worker to process the clips in parallel

So, for example, if I separate the video in 10 clips and send them to 10 workers, is it more expensive than sending it to 5 workers?

Jason•12/12/24, 11:17 PM

More Workers (10 workers):

Advantage: Processes all clips in parallel, potentially faster if workers are ready.

Disadvantage: Higher costs due to model loading time charged for each worker.

Fewer Workers (5 workers):

Advantage: Workers process clips sequentially but reuse the loaded model, minimizing setup costs.

Disadvantage: Slightly slower because fewer workers handle the clips.

Constant Requests Scenario:

If workers remain active (model stays loaded), using more workers is faster but not significantly more expensive. This is efficient for high throughput.

Recommendation:

For occasional jobs: Use fewer workers to reduce setup costs.

For frequent jobs: Use more workers to maximize speed and maintain efficiency.
@galakurpismo3

chee•12/12/24, 11:19 PM

@galakurpismo3 how did u end up fixing the issue

chee•12/12/24, 11:19 PM

I’m getting essentially the same thing

Cchee I’m getting essentially the same thing

Jason•12/12/24, 11:21 PM

In zendesk, a support rep checked and helped them, is your condition the same?

Jason•12/12/24, 11:21 PM

Then it's best to open a support ticket too

Jason•12/12/24, 11:21 PM

Check your thread

Cchee @galakurpismo3 how did u end up fixing the issue

galakurpismo3OP•12/15/24, 10:58 AM

I recommend using a Zendesk ticket yeah it's really helpful

I lowered the maximum execution time to that endpoint so if a worker gets an error and stays running for too long, this shuts it down

I don't know my serverless balance goes down

Similar Threads

Similar Threads

Similar Threads