what's the best way to use fetch with a proxy from a Worker?

sathoroOP•11/12/23, 6:27 AM

can't find any proxy providers that support a dedicated IP with an API endpoint

bkyerv•11/12/23, 1:20 PM

is there any way to create folders in the dashboard where all workers and pages are listed so to group them by projects?

Bbkyerv is there any way to create folders in the dashboard where all workers and pages ...

Cyb3r-Jak3•11/12/23, 3:27 PM

Unfortunately not

bkyerv•11/12/23, 3:56 PM

is this something in the roadmap? I thought this is something that most people would want to use. or am I being biased? now when I heavily use cloudflare for many small projects I spent a lot of time finding my workers and pages.

Theli•11/12/23, 4:31 PM

You are one pages project and few API calls away from it

Ssathoro what's the best way to use fetch with a proxy from a Worker?

dave•11/12/23, 7:34 PM

you mean like a SOCKS-like proxy service that works with Workers?

Ddave you mean like a SOCKS-like proxy service that works with Workers?

sathoroOP•11/12/23, 7:34 PM

yeah through HTTPS endpoint that has dedicated IP support

sathoroOP•11/12/23, 7:34 PM

"that has dedicated IP support" this is the part I can't find

sathoroOP•11/12/23, 7:34 PM

I already have rotating proxies with HTTPS endpoints

Ssathoro "that has dedicated IP support" this is the part I can't find

dave•11/12/23, 7:34 PM

what's the reasoning behind needing a dedicated IP from a third party? Why not use your own?

Ddave what's the reasoning behind needing a dedicated IP from a third party? Why not u...

sathoroOP•11/12/23, 7:35 PM

my own IP, from where?

Ssathoro I already have rotating proxies with HTTPS endpoints

dave•11/12/23, 7:35 PM

those work with Workers?

sathoroOP•11/12/23, 7:35 PM

there is a host that has completely blocked all incoming workers

sathoroOP•11/12/23, 7:35 PM

and we have lots of users that use that host for an integration

sathoroOP•11/12/23, 7:35 PM

and they won't work with us

Ddave those work with Workers?

sathoroOP•11/12/23, 7:35 PM

of course because its a normal HTTPS call

sathoroOP•11/12/23, 7:35 PM

no proxy config

Ssathoro of course because its a normal HTTPS call

dave•11/12/23, 7:36 PM

huh interesting, could you show an example fetch?

Ddave huh interesting, could you show an example fetch?

sathoroOP•11/12/23, 7:37 PM

const params = {
  url,
};

const queryString = new URLSearchParams(params).toString();

fetch(`https://app.scrapingbee.com/api/v1/?${queryString}`);

const params = {
  url,
};

const queryString = new URLSearchParams(params).toString();

fetch(`https://app.scrapingbee.com/api/v1/?${queryString}`);

Ssathoro ```js const params = { url, }; const queryString = new URLSearchParams(params...

dave•11/12/23, 7:37 PM

oh, that ,makes more sense lol. I was thinking it was a straight up standard HTTP proxy

Ssathoro my own IP, from where?

dave•11/12/23, 7:38 PM

why not spin up an instance on DigitalOcean or something?

dave•11/12/23, 7:38 PM

or is it a scaling problem too?

Ddave why not spin up an instance on DigitalOcean or something? 🙂

sathoroOP•11/12/23, 7:39 PM

I have a feeling that will be blocked already but yeah that is the next move if this doesn't work. this was supposed to be the easiest option but nobody offers it..

sathoroOP•11/12/23, 7:39 PM

might just use rotating residential proxy should work

dave•11/12/23, 7:40 PM

in theory you could use outbound TCP sockets and connect to a "normal" HTTP or SOCKS proxy.

liam | helius.dev•11/13/23, 3:39 AM

any idea why returning a 401 from a gRPC service would cause CF LB to return a 520? It doesn't return 520 if I made a http1 call to the same server w same response

Lliam | helius.dev any idea why returning a 401 from a gRPC service would cause CF LB to return a 5...

Unsmart•11/13/23, 3:43 AM

https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/#error-520-web-server-returns-an-unknown-error

Troubleshooting Cloudflare 5XX errors · Cloudflare Support docs

When troubleshooting most 5XX errors, the correct course of action is to first contact your hosting provider or site administrator to troubleshoot and …

liam | helius.dev•11/13/23, 3:44 AM

None of these are relevant, its something up with gRPC handling

ndy•11/13/23, 2:12 PM

Does @cf/meta/llama-2-7b-chat-int8 in Workers AI have a token restriction that generates incomplete results

Nndy Does @cf/meta/llama-2-7b-chat-int8 in Workers AI have a token restriction that g...

Erisa•11/13/23, 2:40 PM

A better channel is #workers-ai-beta
But yes, there are limits here: https://developers.cloudflare.com/workers-ai/platform/limits/

Erisa•11/13/23, 2:42 PM

Also @ndy please try to avoid sending your question to multiple channels as this can cause confusion and duplicate efforts

Lliam | helius.dev None of these are relevant, its something up with gRPC handling

CB•11/13/23, 5:10 PM

GRPC doesn't have 401s...

what GRPC code are you returning?

CCB GRPC doesn't have 401s... 🙂 what GRPC code are you returning?

CB•11/13/23, 5:11 PM

https://grpc.github.io/grpc/core/md_doc_statuscodes.html

CCB GRPC doesn't have 401s... 🙂 what GRPC code are you returning?

liam | helius.dev•11/13/23, 5:11 PM

set grpc-status header to PERMISSION_DENIED but didn't change anything

Lliam | helius.dev set grpc-status header to PERMISSION_DENIED but didn't change anything

CB•11/13/23, 5:13 PM

are you using some sort of GRPC gateway?

CCB are you using some sort of GRPC gateway?

liam | helius.dev•11/13/23, 5:14 PM

Using envoy

liam | helius.dev•11/13/23, 5:14 PM

Have a wasm plugin that for eg. will return 401 if no api key is present

liam | helius.dev•11/13/23, 5:14 PM

for http1 this works, for grpc the LB returns 520

Lliam | helius.dev Have a wasm plugin that for eg. will return 401 if no api key is present

CB•11/13/23, 5:15 PM

What that sounds like to me is the gateway you are using is translating the GRPC -> to http1 error codes incorrectly

CB•11/13/23, 5:15 PM

when the grpc request is authenticated does it work correctly?

CCB when the grpc request is authenticated does it work correctly?

liam | helius.dev•11/13/23, 5:16 PM

Yep it does

CB•11/13/23, 5:16 PM

yeah, so cloudflare part is working

CCB yeah, so cloudflare part is working

liam | helius.dev•11/13/23, 5:16 PM

If the code just responds with a 401, grpc-status header set, what else could be missing?

Lliam | helius.dev If the code just responds with a 401, grpc-status header set, what else could be...

CB•11/13/23, 5:17 PM

envoy config could be wrong

Panda•11/13/23, 7:18 PM

So to use Pages advanced mode I have to use this https://developers.cloudflare.com/pages/platform/functions/advanced-mode/

I did follow the docs, but like im struggling with putting the _worker.js in the actual output dir??

Like as you can see in the first photo, running the build command generates a _worker.js file, but it is not the one that I wrote, how do I get MY worker to be in the build directory? Keep in mind I'm using svelte

Panda•11/13/23, 7:20 PM

This is my wrangler.toml file, it is worth noting that my worker is located in /worker/index.js

Erisa•11/13/23, 8:04 PM

If you switch your Worker to the new Standard pricing (or even the old Unbound pricing), the subrequest limit will be 1000 instead of 50

what's the best way to use fetch with a proxy from a Worker?

Similar Threads

Similar Threads

Similar Threads