R2 Caching for parqurt
So I have a BI app hosted on cloudflare Pages that used DuckDB Wasm. It uses the duckdb wasm https extension to make range requests to the parquet file to query data, with the idea being that duckdb is smart enough to read parquet metadata so it doesn’t read entire data off disk to execute query.
I basically want to figure out the best way to design the cache for this.
I need to design the cache so that cloudflare does no compression on the parquet file, because this messes up the range requests and makes duckdb read the whole file again.
However, I also need to design it so if the underlying file is overwritten, the cache gets updated, so duckdb doesn’t read stale data.
I think there’s ways to do it with tags and stuff, but I’m pretty novice with this.
Is there a programmatic way to set special rules for this? I’d prefer to be able to do this with auditable code as well instead of clickops if possible
Thanks you!
I basically want to figure out the best way to design the cache for this.
I need to design the cache so that cloudflare does no compression on the parquet file, because this messes up the range requests and makes duckdb read the whole file again.
However, I also need to design it so if the underlying file is overwritten, the cache gets updated, so duckdb doesn’t read stale data.
I think there’s ways to do it with tags and stuff, but I’m pretty novice with this.
Is there a programmatic way to set special rules for this? I’d prefer to be able to do this with auditable code as well instead of clickops if possible
Thanks you!
