Hi, i want to save headers and other stuff from each visitor through workers, what would you recomme

Hi, i want to save headers and other stuff from each visitor through workers, what would you recommend for it? I made a worker for saving in D1 but i noticed the data cant be downloaded and i would like to download that data locally. I appreciate your help

Eehrenmann Hi, i want to save headers and other stuff from each visitor through workers, wh...

Cyb3r-Jak3•6/26/24, 9:44 PM

I personally use AE but you can't download from that. You can export the data from D1: https://developers.cloudflare.com/workers/wrangler/commands/#export

ehrenmannOP•6/26/24, 9:58 PM

would maybe hyperdrive be an option?

Unsmart•6/26/24, 10:00 PM

What exactly is your goal? Why are you downloading data locally?

ehrenmannOP•6/26/24, 10:04 PM

i would like to download it and collect it and then do some study about my traffic

ehrenmannOP•6/26/24, 10:04 PM

thorugh months

ehrenmannOP•6/26/24, 10:04 PM

through*

Unsmart•6/26/24, 10:06 PM

Why not just feed it to an analytics tool and use queries in that instead of downloading it? AE is good if you only need 3 months

ehrenmannOP•6/26/24, 10:08 PM

sorry but im new to this and dont understand completely what you mean by AE

Unsmart•6/26/24, 10:08 PM

#analytics-engine

ehrenmannOP•6/26/24, 10:15 PM

and is it possible to read that data externally or so? because i would like to create my own overview with all that information.

Unsmart•6/26/24, 10:15 PM

Yes there is an API you can post queries to

ehrenmannOP•6/26/24, 10:23 PM

do i need a page on my domain for that? because i use my domain mainly for redirecting

Greekdemon•6/27/24, 8:31 AM

still here from the discord block incident

Sstevefrench Is there a limit to the number of concurrent promises running at a time?

Hello, I’m Allie!•6/27/24, 9:54 AM

No, though there is a max number of outgoing connections(6)

Silent•6/27/24, 10:07 AM

Since this is backed by WAF does using it to limit an endpoint stop the requests from counting towards the daily limit?
https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/

Cloudflare Docs

Rate Limiting · Cloudflare Workers docs

Define rate limits and interact with them directly from your Cloudflare Worker

safah222•6/27/24, 1:12 PM

is there a way to decrease number of requests for workers , as each visit to my website is hitting the worker hundreds of time , for each path , and for each request it count 1 request

safah222•6/27/24, 1:12 PM

I just want 1 worker request for each ip address nothing more , can i do that

XXiao 🥑Hi I'm to create a simple api with itty-router, but ran into this error, anyone ...

Kevin•6/27/24, 1:24 PM

I’m traveling so I’m a bit late in seeing this, but switch router.handle —> router.fetch should fix your issue. “handle” was finally deprecated fully in v5

When you access router.handle now, it’s trying to register a route on the HANDLE verb instead of processing a request… so it fails to respond.

KKevin I’m traveling so I’m a bit late in seeing this, but switch router.handle —> rout...

Kevin•6/27/24, 1:25 PM

For more help I def recommend jumping on the itty discord - folks there will beat me to answers all the time!

Ssafah222 is there a way to decrease number of requests for workers , as each visit to my ...

Hard@Work•6/27/24, 2:01 PM

You would need to reduce the requests your Website makes. This can be aided with stuff like SSR, though you would probably still have 2/3 requests per page view

Murder Chicken•6/27/24, 2:22 PM

Is it possible to add specific query string parameters to be redacted from Cloudflare's request logs for GET requests?

Tthanhle when I use the Unpdf package to extract the text from the pdf. maybe the CPU tim...

anthony•6/27/24, 3:07 PM

Did you find a way to make this work for (larger) files in the cf ecosystem?

Jjohtso okay, so my PDF parsing code takes >4 seconds to run, are there any tricks to sp...

anthony•6/27/24, 3:07 PM

Did you find a desirable alternative using cf building blocks?

Aanthony Did you find a desirable alternative using cf building blocks?

johtso•6/27/24, 3:10 PM

Nope, it's just slow

johtso•6/27/24, 3:14 PM

@anthony the limits and allowances are pretty generous on the paid plan though, especially if it's a cron trigger or queue consumer (15 minutes)

johtso•6/27/24, 3:15 PM

@anthony are you bumping up against some limits?

Paul•6/27/24, 3:32 PM

I naively thought I could share the logic from my nextjs app with my cloudflare workers but the environment variables is causing quite a bit of issues. Even when polyfilling with

import * as process from 'node:process';

import * as process from 'node:process';

, whenever I use process.env, the nextjs app doesn't understand "node:process". Is it common to simply duplicate the logic for the cloudflare worker? Even my database calls requires a separate env and it's a bit too integrated. Is it common for folks to duplicate their code logic for CF workers or are people commonly able to reuse logic from their app?

Paul•6/27/24, 3:43 PM

Is it common to declare something like globalThis.process = process (imported from node:process) when using cloudflare workers for nextjs and cloudflare worker compatability?

lupin•6/27/24, 4:02 PM

Hi everyone ! Is there a way to send a fetch request from the Cloudflare worker without waiting the response ? The API that I try to contact is just there to receive data and then will do a lot of operations that take some time and I want to close my connection without waiting for the reponse :)

Cheers !

Llupin Hi everyone ! Is there a way to send a fetch request from the Cloudflare worker ...

Chaika•6/27/24, 4:06 PM

ctx.waitUntil is your friend, hand it a promise and it'll let the response return but not kill the execution
ctx.waitUntil(fetch(..))

Chaika•6/27/24, 4:07 PM

https://developers.cloudflare.com/workers/runtime-apis/context/#waituntil

Cloudflare Docs

Context (ctx) · Cloudflare Workers docs

The Context API in Cloudflare Workers, including waitUntil and passThroughOnException.

lupin•6/27/24, 4:11 PM

We've been exploring waitUntil recently. I'll let you know if this is the magicial solution we needed @Chaika :)

Wai #NeyiSevsekKapatılıyor•6/27/24, 5:31 PM

umm what did i do wrong

Code_Megaraptor_Witty_Spanish_2024_June_Thursday_20_30_39.png

Jjohtso Nope, it's just slow 🤷

anthony•6/27/24, 8:23 PM

Pretty similar as to what i've read here. Ingesting PDF's via fetch in context of http request. Want to parse it for the text using unpdf, so i can use it in Workers AI/LLM. Works fine code-wise but explodes in prod because of timeout/memory issues.

Works for small pdf's but reality is i can have some larger ones as well which does not seem possible. Streaming is not an option afaik forthese operations...

johtso•6/27/24, 8:39 PM

Yeah, pdf is a pig of a format. My only recommendations are, if you're hitting timeouts, make the process asynchronous by storing the pdf in R2 and putting a message on a queue. Then you'll have 15 minutes to process it.

If you're hitting memory limits, one possibility could be too find a way to split the pdf into pages without loading the whole thing into memory (I don't know if this is actually possible given the format).. maybe using WASM

johtso•6/27/24, 8:39 PM

@anthony this looks interesting.. https://dev.to/wcchoi/browser-side-pdf-processing-with-go-and-webassembly-13hn

Sstevefrench do you know if there's a way to tell if your worker is exceeding this limit?

Hello, I’m Allie!•6/27/24, 8:45 PM

If a call doesn’t immediately start? It doesn’t throw an error, it just pauses it until a slot frees up

johtso•6/27/24, 8:54 PM

@anthony ooo.. looks like mupdf can handle streams.. https://github.com/ArtifexSoftware/mupdf.js/blob/master/examples/tools/split-pages.js

johtso•6/27/24, 9:02 PM

https://mupdfjs.readthedocs.io/en/latest/how-to-guide/node/document/index.html#extracting-document-text