Cloudflare D1 inserting data outside of workers
Hi, I have a python scraper running on a private server and I'd like it to insert data into D1. I used D1 inside of workers but how can I use it outside workers in a regular script? Is it possible?
10 Replies
It's not just inserting data but interacting with DB in general. I want to query and modify some rows as well.
?d1-remote-connections
Accessing D1 from outside workers:
D1 is based on SQLite, which does not natively support remote connections and isn't MySQL/PostgreSQL wire compatible, nor does D1 itself offer that functionality. As such, your options are:
1 -> The Cloudflare API Query Endpoint (https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database), with the upside of needing less setup, all requests have to go back to Cloudflare's core, so request latency is far from optimal from some locations. You would also be subject to the API Rate limit of 1200 requests per 5 minutes.
2 -> Create a worker to proxy requests to your D1 Database. Either tightly bound to your application/schema or loose and just accepting any query. This would allow your worker to execute potentially closer to your D1 Database and avoid the api rate limit.
So worker proxy is the only way? It's a bit over-engineered :(. I was hoping for quick data insertion. Maybe I will take a look at turso/neon with hyperdrive
the API is a way, the first option there
performance might be bad though? if I query it from EU and it's in US for example
yes, it's def not good for performance. If it's just a script to insert data though and you'd query the data/serve to users via workers binding, then can work
but will it work if there are 50000 potential daily requests
Requests from the scraper to update the database? You'd likely hit the 1200 requests per 5 minutes api rate limit unless they were spaced out enough
yeah... so in my case worker proxy is the only way
i have over 50000 listings that i monitor with a scraper and need to update them accordingly in my own db