hiya, is there a way to use a local directory as the source for simulating an r2 bucket?

CCode hiya, is there a way to use a local directory as the source for simulating an r2...

H

Sort of? You can tell wrangler to dump all of the bindings into a directory, but using it requires reading a SQLite file

Original message was deleted

H

Hello, I’m Allie!•5/1/24, 11:57 AM

I don’t think Miniflare has a directory import function

H

Hello, I’m Allie!•5/1/24, 11:57 AM

Though wrangler might, come to think?

C

CodeOP•5/1/24, 11:58 AM

i've tried using miniflare and wrangler but they both appear to use a sqlite database

CCode i've tried using miniflare and wrangler but they both appear to use a sqlite dat...

H

Hello, I’m Allie!•5/1/24, 11:58 AM

Yeah, that’s what I mean. There isn’t a way to just mount a directory, but you should be able to import a directory with wrangler/miniflare

C

CodeOP•5/1/24, 11:59 AM

i've looked into it and i think it's no longer being worked on

C

CodeOP•5/1/24, 11:59 AM

https://github.com/cloudflare/workers-sdk/issues/3687

GitHub

🚀 Feature Request: Ability to emulate a public R2 bucket when devel...

Describe the solution Under Wrangler 2, when running locally any files you'd upload to a "local" R2 bucket would be saved in a directory with the exact filename you gave it at upload ...

C

CodeOP•5/1/24, 11:59 AM

hmm

C

CodeOP•5/1/24, 12:02 PM

do you mind pointing me towards an example of either being used to like serve local files?

C

CodeOP•5/1/24, 12:03 PM

the examples i'm finding are simply referencing copying or accessing local files like within rclone/minio

B

bun•5/1/24, 12:04 PM

i have an example that i use locally:

import { Database } from "bun:sqlite";
import p from "path";
import fs from "node:fs";
import { drizzle } from "drizzle-orm/bun-sqlite";
import { eq } from "drizzle-orm";
import * as schema from "./schema";

const blobsPath = p.join(
  p.dirname(p.dirname(import.meta.dir)),
  ".wrangler/state/v3/r2/rn-updates/blobs"
);

const basePath = p.join(
  p.dirname(p.dirname(import.meta.dir)),
  ".wrangler/state/v3/r2/miniflare-R2BucketObject"
);

const sqlitePath = fs.readdirSync(basePath).find((f) => f.endsWith(".sqlite"));

if (!sqlitePath) {
  throw new Error("SQLite file not found");
}

const file = p.join(basePath, sqlitePath);

const client = new Database(file);

console.log("Using SQLite file", file);

const db = drizzle(client, { schema });

Bun.serve({
  port: 8788,
  async fetch(request) {
    const filename = new URL(request.url).pathname.slice(1);

    console.log("Requesting", filename);

    const object = await db.query.objects
      .findMany({
        where: eq(schema.objects.key, filename),
        limit: 1,
      })
      .then((r) => r[0]);

    if (!object || !object.blobId) {
      return new Response("Not found", { status: 404 });
    }

    const file = Bun.file(p.join(blobsPath, object.blobId));

    return new Response(file, {
      headers: {
        "content-disposition": `attachment; filename="${filename}"`,
      },
    });
  },
});

import { Database } from "bun:sqlite";
import p from "path";
import fs from "node:fs";
import { drizzle } from "drizzle-orm/bun-sqlite";
import { eq } from "drizzle-orm";
import * as schema from "./schema";

const blobsPath = p.join(
  p.dirname(p.dirname(import.meta.dir)),
  ".wrangler/state/v3/r2/rn-updates/blobs"
);

const basePath = p.join(
  p.dirname(p.dirname(import.meta.dir)),
  ".wrangler/state/v3/r2/miniflare-R2BucketObject"
);

const sqlitePath = fs.readdirSync(basePath).find((f) => f.endsWith(".sqlite"));

if (!sqlitePath) {
  throw new Error("SQLite file not found");
}

const file = p.join(basePath, sqlitePath);

const client = new Database(file);

console.log("Using SQLite file", file);

const db = drizzle(client, { schema });

Bun.serve({
  port: 8788,
  async fetch(request) {
    const filename = new URL(request.url).pathname.slice(1);

    console.log("Requesting", filename);

    const object = await db.query.objects
      .findMany({
        where: eq(schema.objects.key, filename),
        limit: 1,
      })
      .then((r) => r[0]);

    if (!object || !object.blobId) {
      return new Response("Not found", { status: 404 });
    }

    const file = Bun.file(p.join(blobsPath, object.blobId));

    return new Response(file, {
      headers: {
        "content-disposition": `attachment; filename="${filename}"`,
      },
    });
  },
});

import { Database } from "bun:sqlite";
import p from "path";
import fs from "node:fs";
import { drizzle } from "drizzle-orm/bun-sqlite";
import { eq } from "drizzle-orm";
import * as schema from "./schema";

const blobsPath = p.join(
  p.dirname(p.dirname(import.meta.dir)),
  ".wrangler/state/v3/r2/rn-updates/blobs"
);

const basePath = p.join(
  p.dirname(p.dirname(import.meta.dir)),
  ".wrangler/state/v3/r2/miniflare-R2BucketObject"
);

const sqlitePath = fs.readdirSync(basePath).find((f) => f.endsWith(".sqlite"));

if (!sqlitePath) {
  throw new Error("SQLite file not found");
}

const file = p.join(basePath, sqlitePath);

const client = new Database(file);

console.log("Using SQLite file", file);

const db = drizzle(client, { schema });

Bun.serve({
  port: 8788,
  async fetch(request) {
    const filename = new URL(request.url).pathname.slice(1);

    console.log("Requesting", filename);

    const object = await db.query.objects
      .findMany({
        where: eq(schema.objects.key, filename),
        limit: 1,
      })
      .then((r) => r[0]);

    if (!object || !object.blobId) {
      return new Response("Not found", { status: 404 });
    }

    const file = Bun.file(p.join(blobsPath, object.blobId));

    return new Response(file, {
      headers: {
        "content-disposition": `attachment; filename="${filename}"`,
      },
    });
  },
});

import { Database } from "bun:sqlite";
import p from "path";
import fs from "node:fs";
import { drizzle } from "drizzle-orm/bun-sqlite";
import { eq } from "drizzle-orm";
import * as schema from "./schema";

const blobsPath = p.join(
  p.dirname(p.dirname(import.meta.dir)),
  ".wrangler/state/v3/r2/rn-updates/blobs"
);

const basePath = p.join(
  p.dirname(p.dirname(import.meta.dir)),
  ".wrangler/state/v3/r2/miniflare-R2BucketObject"
);

const sqlitePath = fs.readdirSync(basePath).find((f) => f.endsWith(".sqlite"));

if (!sqlitePath) {
  throw new Error("SQLite file not found");
}

const file = p.join(basePath, sqlitePath);

const client = new Database(file);

console.log("Using SQLite file", file);

const db = drizzle(client, { schema });

Bun.serve({
  port: 8788,
  async fetch(request) {
    const filename = new URL(request.url).pathname.slice(1);

    console.log("Requesting", filename);

    const object = await db.query.objects
      .findMany({
        where: eq(schema.objects.key, filename),
        limit: 1,
      })
      .then((r) => r[0]);

    if (!object || !object.blobId) {
      return new Response("Not found", { status: 404 });
    }

    const file = Bun.file(p.join(blobsPath, object.blobId));

    return new Response(file, {
      headers: {
        "content-disposition": `attachment; filename="${filename}"`,
      },
    });
  },
});

Bbun i have an example that i use locally: ```ts import { Database } from "bun:sqlit...

C

CodeOP•5/1/24, 12:05 PM

ah, that's cool :o

C

CodeOP•5/1/24, 12:05 PM

oh

C

CodeOP•5/1/24, 12:05 PM

wait that's

C

CodeOP•5/1/24, 12:05 PM

way easier than i thought

C

CodeOP•5/1/24, 12:05 PM

interesting

C

CodeOP•5/1/24, 12:05 PM

i missed rclone serve then

C

CodeOP•5/1/24, 12:05 PM

thank you!

Original message was deleted

N

nikiv.dev•5/1/24, 2:03 PM

https://github.com/maccman/cloudflare-r2-edge so i found this

GitHub

GitHub - maccman/cloudflare-r2-edge

Contribute to maccman/cloudflare-r2-edge development by creating an account on GitHub.

N

nikiv.dev•5/1/24, 2:03 PM

but why does it ask me to get s3 creds?

N

nikiv.dev•5/1/24, 2:04 PM

can I not just work with cloudflare to store my blobs?

N

nikiv.dev•5/1/24, 2:04 PM

i.e. get cloudflare keys etc.

N

nikiv.dev•5/1/24, 2:04 PM

i really don't want to open aws dashboard, its pure pain

Nnikiv.dev i really don't want to open aws dashboard, its pure pain

C

Chaika•5/1/24, 2:04 PM

you don't have to do that lol

Nnikiv.dev but why does it ask me to get s3 creds?

C

Chaika•5/1/24, 2:05 PM

r2's primary api is their S3 Compatible API

C

Chaika•5/1/24, 2:05 PM

so it uses all the same terminology and supports most of the normal s3 api calls

N

nikiv.dev•5/1/24, 2:05 PM

N

nikiv.dev•5/1/24, 2:06 PM

only see s3 api one

N

nikiv.dev•5/1/24, 2:06 PM

i just want a hash table of images

N

nikiv.dev•5/1/24, 2:06 PM

mm cant find how to

C

Chaika•5/1/24, 2:06 PM

under R2 -> Manage R2 API Tokens you get the credentials for the s3 api

N

nikiv.dev•5/1/24, 2:08 PM

ok so this lib is not good https://github.com/maccman/cloudflare-r2-edge

GitHub

GitHub - maccman/cloudflare-r2-edge

Contribute to maccman/cloudflare-r2-edge development by creating an account on GitHub.

N

nikiv.dev•5/1/24, 2:08 PM

mm i wonder if there is something like it that just lets me auth with the token

N

nikiv.dev•5/1/24, 2:08 PM

and point to bucket

N

nikiv.dev•5/1/24, 2:08 PM

or i guess i can just go through cloudflare http api

N

nikiv.dev•5/1/24, 2:09 PM

https://developers.cloudflare.com/r2/get-started/ wants me to make wrangler

Cloudflare Docs

Get started guide · Cloudflare R2 docs

Cloudflare R2 Storage allows developers to store large amounts of unstructured data without the costly egress bandwidth fees associated with typical …

C

Chaika•5/1/24, 2:09 PM

the normal cf rest api doesn't have documented endpoints for uploading or managing objects

C

Chaika•5/1/24, 2:09 PM

have to use the s3 compatible one

N

nikiv.dev•5/1/24, 2:09 PM

ok so here

N

nikiv.dev•5/1/24, 2:09 PM

i now have my r2 token

N

nikiv.dev•5/1/24, 2:09 PM

which is

accessKeyId

accessKeyId

accessKeyId

accessKeyId

N

nikiv.dev•5/1/24, 2:09 PM

i think

N

nikiv.dev•5/1/24, 2:09 PM

oh wait

N

nikiv.dev•5/1/24, 2:10 PM

N

nikiv.dev•5/1/24, 2:10 PM

I think i got it, thanks

CChaika the normal cf rest api doesn't have documented endpoints for uploading or managi...

N

nikiv.dev•5/1/24, 2:10 PM

kinda weird tbh

N

nikiv.dev•5/1/24, 2:11 PM

like cloudflare focuses on dx, I can see many ways to make this flow easier

Nnikiv.dev kinda weird tbh

C

Chaika•5/1/24, 2:11 PM

well the normal cf api has rate limits (1200 requests/5 minutes) and strict upload limits (iirc 500 mb)

C

Chaika•5/1/24, 2:11 PM

it's shared among all teams, and they're working on more granular rate limiting but nothing yet

C

Chaika•5/1/24, 2:12 PM

probably at least part of the reason they want to push you to using the s3 api which has no rate limits and just the r2 max upload max of 5gb before having to use multipart

hiya, is there a way to use a local directory as the source for simulating an r2 bucket?

Similar Threads

Similar Threads

Similar Threads