Are webhooks fired from Digital Ocean?

I setup a WAF in AWS to block bots and I am getting a bunch of requests to my RunPod Serverless Webhook blocked by AWS#AWSManagedRulesBotControlRuleSet#SignalKnownBotDataCenter . The IP address in these requests seems to be a Digital Ocean Data Center. I have disabled the WAF for my ALB for my RunPod webhooks temporarily, but hoping that someone can confirm whether these are legitimate requests or not, because I was under the impression that RunPod uses AWS and not Digital Ocean.
17 Replies
nerdylive
nerdylive4w ago
Yeah they can use multiple clouds @flash-singh can you confirm this
digigoblin
digigoblin4w ago
Obviously, but the question is DO they? 🙂
nerdylive
nerdylive4w ago
Wait ya, maybe support staffs can confirm im not sure
flash-singh
flash-singh4w ago
yes we use DO, AWS and Cloudflare
digigoblin
digigoblin4w ago
Thanks for confirming, do you just have 2 IPs in DO or multiple?
nerdylive
nerdylive4w ago
Maybe you can give the IP range for whitelisting
digigoblin
digigoblin4w ago
I just removed my webhook load balancer from the WAF and added an auth token to the webhook receiver
flash-singh
flash-singh2w ago
many different ips, we don't recommend whitelisting them since they can and will change over time we can introduce more security into the http request if it can help, rather avoid using ip filtering as a security measure since those change
digigoblin
digigoblin2w ago
Yeah it will be nice to add an auth token to the headers rather than in the query string as I am currently doing
flash-singh
flash-singh2w ago
why is that? something you can use to filter at load balancer level?
digigoblin
digigoblin2w ago
Just feels more secure putting the auth token in the header rather than in the query string. Do you use a specific user agent string for the requests or just the default of the library you use? Because then I can filter for user agent string at WAF level before it reaches the load balancer.
flash-singh
flash-singh2w ago
we can define specific one, right now its whatever is default in golang, can also define auth header
digigoblin
digigoblin2w ago
Thanks, this is useful indo, and auth header will be amazing 🫶
Arjun
Arjun2w ago
@flash-singh @digigoblin Funny I was just implementing webhooks on our project today and the exact subject came up for us too. Putting auth tokens in a query param can be insecure and make the token highly visible if there are logs being generated within RunPod that log the url. Adding as a header: Authorization: Bearer <token> pattern would be standard and more secure. This could either be set on the endpoint config in the console, or how it currently is in the initial run post:
{
input: {...},
webhook: {url: 'https...', auth_token: 'your auth token'}
}
{
input: {...},
webhook: {url: 'https...', auth_token: 'your auth token'}
}
Even better if we could add additional body params to the webhook: Scrap this idea
{
input: {...},
webhook: {
url: 'https...',
auth_token: 'your auth token'
params: {
custom_identifier_1: '',
custom_identifier_2: '',
...
}
}
}
{
input: {...},
webhook: {
url: 'https...',
auth_token: 'your auth token'
params: {
custom_identifier_1: '',
custom_identifier_2: '',
...
}
}
}
Given the current limitations, another way we were thinking we could do is just encrypt a single state param that contains a hash of a few things (incl. auth token).
nerdylive
nerdylive2w ago
Or maybe just headers field that we can customize, with some examples to use.. Like for Auth headers, and req. bodies are kind of useless here right?
Arjun
Arjun2w ago
@nerdylive True, a headers field would be the most versatile and simplest. Anything else might get muddied with the webhook post body.
flash-singh
flash-singh2w ago
thanks for the input, will dive into this more next week and how we can improve it