Runpod•2y ago

Are webhooks fired from Digital Ocean?

I setup a WAF in AWS to block bots and I am getting a bunch of requests to my RunPod Serverless Webhook blocked by AWS#AWSManagedRulesBotControlRuleSet#SignalKnownBotDataCenter . The IP address in these requests seems to be a Digital Ocean Data Center. I have disabled the WAF for my ALB for my RunPod webhooks temporarily, but hoping that someone can confirm whether these are legitimate requests or not, because I was under the impression that RunPod uses AWS and not Digital Ocean.

22 Replies

Unknown User•2y ago

Message Not Public

digigoblinOP•2y ago

Obviously, but the question is DO they? 🙂

Unknown User•2y ago

Message Not Public

flash-singh•2y ago

yes we use DO, AWS and Cloudflare

digigoblinOP•2y ago

Thanks for confirming, do you just have 2 IPs in DO or multiple?

Unknown User•2y ago

Message Not Public

digigoblinOP•2y ago

I just removed my webhook load balancer from the WAF and added an auth token to the webhook receiver

flash-singh•17mo ago

many different ips, we don't recommend whitelisting them since they can and will change over time we can introduce more security into the http request if it can help, rather avoid using ip filtering as a security measure since those change

digigoblinOP•17mo ago

Yeah it will be nice to add an auth token to the headers rather than in the query string as I am currently doing

flash-singh•17mo ago

why is that? something you can use to filter at load balancer level?

digigoblinOP•17mo ago

Just feels more secure putting the auth token in the header rather than in the query string. Do you use a specific user agent string for the requests or just the default of the library you use? Because then I can filter for user agent string at WAF level before it reaches the load balancer.

flash-singh•17mo ago

we can define specific one, right now its whatever is default in golang, can also define auth header

digigoblinOP•17mo ago

Thanks, this is useful indo, and auth header will be amazing 🫶

Arjun•17mo ago

@flash-singh @digigoblin Funny I was just implementing webhooks on our project today and the exact subject came up for us too. Putting auth tokens in a query param can be insecure and make the token highly visible if there are logs being generated within RunPod that log the url. Adding as a header: Authorization: Bearer <token> pattern would be standard and more secure. This could either be set on the endpoint config in the console, or how it currently is in the initial run post:

{
  input: {...},
  webhook: {url: 'https...', auth_token: 'your auth token'}
}

{
  input: {...},
  webhook: {url: 'https...', auth_token: 'your auth token'}
}

~~Even better if we could add additional body params to the webhook:~~ Scrap this idea

{
  input: {...},
  webhook: {
    url: 'https...',
    auth_token: 'your auth token'
    params: {
      custom_identifier_1: '',
      custom_identifier_2: '',
      ...
    }
  }
}

{
  input: {...},
  webhook: {
    url: 'https...',
    auth_token: 'your auth token'
    params: {
      custom_identifier_1: '',
      custom_identifier_2: '',
      ...
    }
  }
}

Given the current limitations, another way we were thinking we could do is just encrypt a single state param that contains a hash of a few things (incl. auth token).

Unknown User•17mo ago

Message Not Public

Arjun•17mo ago

@nerdylive True, a headers field would be the most versatile and simplest. Anything else might get muddied with the webhook post body.

flash-singh•17mo ago

thanks for the input, will dive into this more next week and how we can improve it

ibrahim•16mo ago

Hello, I'm receiving completed Webhook more than one time. Why it could be? I I observed as it triggers webhook url when the process is completed. However it triggers 2rd or 3rd time after some time (like 30 secs) the same webhook for a single completed process.

Unknown User•16mo ago

Message Not Public

ibrahim•16mo ago

I'm already retuning http 200, however it takes about 40seconds. Maybe it's about timeout on Runpod side. Thank you. @nerdylive

Unknown User•16mo ago

Message Not Public

flash-singh•16mo ago

@ibrahim if this is happening over and over, pm me details and ill take a look

Gaming

Programming

Are webhooks fired from Digital Ocean?

Did you find this page helpful?