RunPod•5mo ago

Are webhooks fired from Digital Ocean?

I setup a WAF in AWS to block bots and I am getting a bunch of requests to my RunPod Serverless Webhook blocked by AWS#AWSManagedRulesBotControlRuleSet#SignalKnownBotDataCenter . The IP address in these requests seems to be a Digital Ocean Data Center. I have disabled the WAF for my ALB for my RunPod webhooks temporarily, but hoping that someone can confirm whether these are legitimate requests or not, because I was under the impression that RunPod uses AWS and not Digital Ocean.

23 Replies

nerdylive•5mo ago

Yeah they can use multiple clouds @flash-singh can you confirm this

digigoblin•5mo ago

Obviously, but the question is DO they? 🙂

nerdylive•5mo ago

Wait ya, maybe support staffs can confirm im not sure

flash-singh•5mo ago

yes we use DO, AWS and Cloudflare

digigoblin•5mo ago

Thanks for confirming, do you just have 2 IPs in DO or multiple?

nerdylive•5mo ago

Maybe you can give the IP range for whitelisting

digigoblin•5mo ago

I just removed my webhook load balancer from the WAF and added an auth token to the webhook receiver

flash-singh•4mo ago

many different ips, we don't recommend whitelisting them since they can and will change over time we can introduce more security into the http request if it can help, rather avoid using ip filtering as a security measure since those change

digigoblin•4mo ago

Yeah it will be nice to add an auth token to the headers rather than in the query string as I am currently doing

flash-singh•4mo ago

why is that? something you can use to filter at load balancer level?

digigoblin•4mo ago

Just feels more secure putting the auth token in the header rather than in the query string. Do you use a specific user agent string for the requests or just the default of the library you use? Because then I can filter for user agent string at WAF level before it reaches the load balancer.

flash-singh•4mo ago

we can define specific one, right now its whatever is default in golang, can also define auth header

digigoblin•4mo ago

Thanks, this is useful indo, and auth header will be amazing 🫶

Arjun•4mo ago

@flash-singh @digigoblin Funny I was just implementing webhooks on our project today and the exact subject came up for us too. Putting auth tokens in a query param can be insecure and make the token highly visible if there are logs being generated within RunPod that log the url. Adding as a header: Authorization: Bearer <token> pattern would be standard and more secure. This could either be set on the endpoint config in the console, or how it currently is in the initial run post:

{
  input: {...},
  webhook: {url: 'https...', auth_token: 'your auth token'}
}

{
  input: {...},
  webhook: {url: 'https...', auth_token: 'your auth token'}
}

~~Even better if we could add additional body params to the webhook:~~ Scrap this idea

{
  input: {...},
  webhook: {
    url: 'https...',
    auth_token: 'your auth token'
    params: {
      custom_identifier_1: '',
      custom_identifier_2: '',
      ...
    }
  }
}

{
  input: {...},
  webhook: {
    url: 'https...',
    auth_token: 'your auth token'
    params: {
      custom_identifier_1: '',
      custom_identifier_2: '',
      ...
    }
  }
}

Given the current limitations, another way we were thinking we could do is just encrypt a single state param that contains a hash of a few things (incl. auth token).

nerdylive•4mo ago

Or maybe just headers field that we can customize, with some examples to use.. Like for Auth headers, and req. bodies are kind of useless here right?

Arjun•4mo ago

@nerdylive True, a headers field would be the most versatile and simplest. Anything else might get muddied with the webhook post body.

flash-singh•4mo ago

thanks for the input, will dive into this more next week and how we can improve it

ibrahim•4mo ago

Hello, I'm receiving completed Webhook more than one time. Why it could be? I I observed as it triggers webhook url when the process is completed. However it triggers 2rd or 3rd time after some time (like 30 secs) the same webhook for a single completed process.

nerdylive•4mo ago

You must return a successful response after receiving the web hook Or else it will retry sending until the third time Try reading the docs or maybe something like 200 http response

ibrahim•4mo ago

I'm already retuning http 200, however it takes about 40seconds. Maybe it's about timeout on Runpod side. Thank you. @nerdylive

nerdylive•4mo ago

https://docs.runpod.io/serverless/endpoints/send-requests#webhooks

Send a request | RunPod Documentation

Learn how to construct a JSON request body to send to your custom endpoint, including optional inputs for webhooks, execution policies, and S3-compatible storage, to optimize job execution and resource management.

flash-singh•4mo ago

@ibrahim if this is happening over and over, pm me details and ill take a look

Gaming

Programming

Are webhooks fired from Digital Ocean?