Is there any way to query for all record IDs stored within a Vectorize index? I'd like to know to bu

Is there any way to query for all record IDs stored within a Vectorize index? I'd like to know to build a way of viewing records (metadata mainly) like you can with D1, and to build a script to easily purge an index without fully recreating it.

RRihan Is there any way to query for all record IDs stored within a Vectorize index? I'...

zwily•6/28/24, 10:06 AM

No. It’s a tough omission.

KorigamiK•6/28/24, 12:52 PM

Can you add me to the vectorize beta?

Yyevgen We are focusing to start onboarding first customers on new version of Vectorize ...

cosbgn•6/29/24, 11:03 AM

Hey @yevgen I'm exploring vectorize, could you tell me briefly what will be the limits/new features coming soon on v2?

cosbgn•6/29/24, 11:06 AM

One of my biggest concerns is 1K namespaces, if I want to add a namespace for each "workspace" my users might have I can only have a max of 100 users (if each user has 10 workspaces) which makes it really restrictive.

Alternatively I could do one namespace per users and use metadata to filter the correct workspace but seems inefficient

Yyevgen We are focusing to start onboarding first customers on new version of Vectorize ...

hydroxide•7/1/24, 4:54 AM

I'd like to trial this

Vinodh•7/1/24, 5:01 AM

Vectorize is not available for local development ?

VVinodh Vectorize is not available for local development ?

Vero•7/1/24, 8:37 AM

It's not supported yet

Ccosbgn One of my biggest concerns is 1K namespaces, if I want to add a namespace for ea...

ulysses•7/2/24, 1:48 AM

I have this question as well–what can we expect the limits to be?

thipperz•7/3/24, 5:29 PM

Please Vectorize V2 come soon

thipperz•7/3/24, 5:30 PM

my main concern is the topK

Yyevgen We are focusing to start onboarding first customers on new version of Vectorize ...

Derpenstein69•7/4/24, 7:58 PM

Please add me to the list, thank you

Yyevgen We are focusing to start onboarding first customers on new version of Vectorize ...

kingmesal•7/6/24, 6:52 PM

I'd like in.

Bierman•7/9/24, 2:02 PM

Is there a rough ETA on being able to run vectorize locally?

For context: I have a worker using a lot of cf services and it seems I have to use the api for vectorize as the —remote flag points all services to remote dev when really my goal is to run as many services locally as possible, then use remote for the unsupported ones. Is there a better way to handle this?

Ppdwittig Hey all 👋 - New product launch - Vectorize V2 In the next week or two, we'll s...

Fokke•7/9/24, 4:36 PM

Are you also increasing the number of vectors in an index or just the number of indexes? I am working on a use case where we need something close to 20-40MM vectors. We could store those all over multiple indexes, but a single index would be much preferred.

Or do you perhaps mean the number of vectors per index is increased to 2MM and the number of indexes stays the same?

Ppdwittig Hey all 👋 - New product launch - Vectorize V2 In the next week or two, we'll s...

yigit•7/9/24, 6:28 PM

hey @pdwittig we have a unique case with 500m docs but each of them are 256 dimensions. do you think that we can try vectorize v2 for our usecase?

leesus•7/12/24, 12:46 PM

Is there any page/documentation around Vectorize error codes?

leesus•7/12/24, 12:47 PM

Am seeing a handful of upsert/delete/query errors in sentry with codes 2006 and 2008 and "internal server error" messaging but struggling to find any indication of what the actual issue is

Maltmax•7/14/24, 3:49 PM

Hi! I don't suppose there is a way to get all vectors stored in a vectorize database without knowing the vector IDs, right?

akazwz•7/16/24, 3:09 PM

@pdwittig hello team. my account id is a275d70ec37a6046e6778c77d65dd38b. and i have problems in dashboard vectorize. it is empty now.

Idle•7/17/24, 10:56 AM

?pings

Flare•7/17/24, 10:56 AM

Please do not ping community members for non-moderation reasons. Doing so will not solve your issue faster and will make people less likely to want to help you.

SteffTek•7/18/24, 7:48 AM

vectorize ui not loading :pain:

SteffTek•7/18/24, 7:48 AM

akazwz•7/19/24, 3:50 AM

me too

yigit•7/20/24, 12:25 AM

Is there anyone with access to vectorize v2? I'm eagerly waiting.

Zeeshan•7/24/24, 4:22 PM

Hi folks, just to confirm here once again, if i want to convert data in my d1 db to vectors, i can reach out to vectorise right? and it is paid right?

zensin•7/26/24, 2:08 AM

I keep putting off part of a project hoping this will come out of beta. hows she lookin?

Zzensin I keep putting off part of a project hoping this will come out of beta. hows she...

DaniFoldi•7/26/24, 11:43 AM

:soontm: is the most accurate answer you'll probably get

yevgen•7/31/24, 5:58 PM

I'm happy to announce that all customers who requested access to Vectorize V2 https://forms.gle/jtSJAqz478DsULN86 have been onboarded.

Google Docs

Vectorize V2 - Waitlist

In the next week or two, we'll start rolling out the next generation of Vectorize, which has been completely re-architected from the ground up. This new version supports 2M+ vector indexes, delivers significantly improved query latency, and much more.

Over the coming weeks, we'll be gradually inviting new teams to onboard. If you're interested ...

LordQ•8/1/24, 2:05 PM

Can we expect a visualisation window for Cloudflare Vectorize to manage or monitor our embedding data? Like what we can use in D1 Database

Yyevgen I'm happy to announce that all customers who requested access to Vectorize V2 ht...

zwily•8/3/24, 1:38 PM

I’m 90% sure i filled out the form when you posted it, and never got an email. I just filled it out again.

Zzwily I’m 90% sure i filled out the form when you posted it, and never got an email. I...

Victor•8/3/24, 3:15 PM

I didn't get an email other, but if you hit the /V2 API endpoint either they'll work or you'll get a 403 operation forbidden

VVictor I didn't get an email other, but if you hit the /V2 API endpoint either they'll ...

zwily•8/3/24, 3:16 PM

Where is the documentation?

Zzwily Where is the documentation?

Victor•8/3/24, 3:17 PM

https://developers.cloudflare.com/api/operations/vectorize-list-vectorize-indexes

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

thipperz•8/5/24, 11:34 PM

Hi there,

I filled out the form but I'm getting vectorize.operation_forbidden [code: 1011]vectorize.operation_forbidden [code: 1011] when trying to create a V2 index. I've installed the required Wrangler prerelease version.

The full message also includes

✘ [ERROR] A request to the Cloudflare API (/accounts/***/vectorize/v2/indexes) failed.✘ [ERROR] A request to the Cloudflare API (/accounts/***/vectorize/v2/indexes) failed.

BlockTxn•8/10/24, 2:15 PM

I'm trying to delete an index, i can confirm that the token and account id are 100% correct, followed the doc yet

  curl --request DELETE \
  --url https://api.cloudflare.com/client/v4/accounts/c9XXXXXXXXXX/vectorize/v2/indexes/vector-index \
  --header 'Authorization: Bearer h1IXXXXXXXXX' \
  --header 'Content-Type: application/json'

  curl --request DELETE \
  --url https://api.cloudflare.com/client/v4/accounts/c9XXXXXXXXXX/vectorize/v2/indexes/vector-index \
  --header 'Authorization: Bearer h1IXXXXXXXXX' \
  --header 'Content-Type: application/json'

got

{"success":false,"errors":[{"code":10000,"message":"Authentication error"}]}

{"success":false,"errors":[{"code":10000,"message":"Authentication error"}]}

any ideas ? thanks

BBlockTxn I'm trying to delete an index, i can confirm that the token and account id are 1...

Isaac McFadyen•8/10/24, 2:17 PM

You're using tokens correct? Not global API key?

IIsaac McFadyen You're using tokens correct? Not global API key?

BlockTxn•8/10/24, 2:19 PM

Not sure, I went to https://dash.cloudflare.com/profile/api-tokens
Created api key
Tested it with

curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
-H "Authorization: Bearer h1XXX" \
-H "Content-Type:application/json"

then followed on with the tutorials no problems but cannot delete, maybe i'm wrong

Yyevgen I'm happy to announce that all customers who requested access to Vectorize V2 ht...

DISPLAY NAME•8/13/24, 6:21 AM

do we need to fill this form to test V2 or need to subscribe a paid plan?

yevgen•8/14/24, 11:43 AM

Vectorize V2 is in public beta. As long as you have access to Vectorize and use the latest version of Wrangler https://github.com/cloudflare/workers-sdk/releases/tag/wrangler%403.71.0 you should have access to Vectorize V2.

GitHub

Release wrangler@3.71.0 · cloudflare/workers-sdk

Minor Changes

#6464 da9106c Thanks @AnantharamanSI! - feat: rename --count to --limit in wrangler d1 insights
This PR renames wrangler d1 insight's --count flag to --limit to improve clarity ...

BBlockTxn Not sure, I went to https://dash.cloudflare.com/profile/api-tokens Created api k...

garvitg•8/14/24, 3:09 PM

Hi @BlockTxn. Could you please confirm that the API Token you are using includes the Vectorize:Edit permission for the account in which this index is present? That permission would be needed to create or delete Vectorize indexes.

steyblind•8/15/24, 1:19 PM

Question about metadata indexing behavior: does it apply to existing vectors or only to those created after the metadata index is created? otherwise, does it backfill the index async?

steyblind•8/15/24, 1:28 PM

FYI, submitted API site feedback, https://developers.cloudflare.com/api/operations/vectorize-list-metadata-indexes should be GET not POST; also fails in

wrangler 3.71.0 when running $ wrangler vectorize list-metadata-index <index-name>$ wrangler vectorize list-metadata-index <index-name>

thipperz•8/18/24, 4:01 AM

there is something bad going on with vectorize indexes, I tried three times to create a text embeddings index, I defined them with metric cosine and 1536 dimensions. It appears that after inserting a certain amout of vectors my index breaks and my queries starting returning absolutely nonsense "scores" - for example right now my queries to this cosine index are returning scores like 28.955 or something like that (and that is breaking search results)

Saatvik•8/19/24, 3:21 AM

I don't see my vectorize v2 index in cloudflare dashboard. I only see v1 vector index. listing it with wrangler cli works

Is there any way to query for all record IDs stored within a Vectorize index? I'd like to know to bu

Similar Threads

Similar Threads

Similar Threads