Can we also expect a way to CRUD a vector (row) directly by its id/uid?

ZZak @ insertchat.com Can we also expect a way to CRUD a vector (row) directly by its id/uid?

J

Jerome•10/5/23, 12:45 PM

there is

getByIds

getByIds

getByIds

getByIds and

deleteByIds

deleteByIds

deleteByIds

deleteByIds operations;

insert

insert

insert

insert and

upsert

upsert

upsert

upsert take the id you provide as well

Z

Zak @ insertchat.comOP•10/5/23, 12:46 PM

I'm an API user, don't have those (I think)

Z

Zak @ insertchat.comOP•10/5/23, 1:31 PM

Hopefully, API users will get some love from Cloudflare x)

P

prmptCoder•10/5/23, 3:14 PM

Is it possible to train or fine-tune the text-generation-model distilbert-sst-2-int8 and deploy@everyone

PprmptCoder Is it possible to train or fine-tune the text-generation-model distilbert-sst-2-...

O

Old Man Maker•10/5/23, 3:23 PM

AFAIK - model hosting with Hugging Face is on the road map to be exposed by cloudflare, so you should be able to at some point

A

Andrew Mayne•10/6/23, 12:07 AM

Hey folks! Two questions:

1. Is there some knack to saving meta data? Everything else saves in this example except the metadata:

const embedding = await getOpenAIEmbedding(description, env.OPENAI_KEY);
await env.VECTORIZE.upsert([{ id: promptId, values: embedding, metadata:{ title:"Updated title", description: "Description"} }]);

console.log("Item vector added:", { title: title, description: description })

2. Everytime I update from the CLI my vector binding are erased. Adding this to the TOML doesn't seem to help:

vectorize = [
{ binding = "VECTORIZE", index_name = "my-index" }
]

Or:

[[vectorize]]
binding = "VECTORIZE"
index_name = "my-index"

AAndrew Mayne Hey folks! Two questions: 1. Is there some knack to saving meta data? Everythin...

O

Old Man Maker•10/6/23, 2:48 AM

Keep in mind that metadata as a type may change in future if filtering by metadata is added.

At the moment, this is how I create it:

//I defined a new object type for the metadata - MetadataSchema 
let newMetaData:MetadataSchema ={
    d1Key: keyValue,
    prjRef: projectRef,
    accRef: accountRef,
    parentRef:parentRef
}

//Stuff the object in there
const metaRecord:Record<string,VectorizeVectorMetadata>= { 
    "data": JSON.stringify(newMetaData)
};

//Add to the vector
let tempVec:VectorizeVector ={
    id:id,
    values: data[0], //the embeddings
    namespace:<The Namespace>,
    metadata:metaRecord
}

//I defined a new object type for the metadata - MetadataSchema 
let newMetaData:MetadataSchema ={
    d1Key: keyValue,
    prjRef: projectRef,
    accRef: accountRef,
    parentRef:parentRef
}

//Stuff the object in there
const metaRecord:Record<string,VectorizeVectorMetadata>= { 
    "data": JSON.stringify(newMetaData)
};

//Add to the vector
let tempVec:VectorizeVector ={
    id:id,
    values: data[0], //the embeddings
    namespace:<The Namespace>,
    metadata:metaRecord
}

//I defined a new object type for the metadata - MetadataSchema 
let newMetaData:MetadataSchema ={
    d1Key: keyValue,
    prjRef: projectRef,
    accRef: accountRef,
    parentRef:parentRef
}

//Stuff the object in there
const metaRecord:Record<string,VectorizeVectorMetadata>= { 
    "data": JSON.stringify(newMetaData)
};

//Add to the vector
let tempVec:VectorizeVector ={
    id:id,
    values: data[0], //the embeddings
    namespace:<The Namespace>,
    metadata:metaRecord
}

//I defined a new object type for the metadata - MetadataSchema 
let newMetaData:MetadataSchema ={
    d1Key: keyValue,
    prjRef: projectRef,
    accRef: accountRef,
    parentRef:parentRef
}

//Stuff the object in there
const metaRecord:Record<string,VectorizeVectorMetadata>= { 
    "data": JSON.stringify(newMetaData)
};

//Add to the vector
let tempVec:VectorizeVector ={
    id:id,
    values: data[0], //the embeddings
    namespace:<The Namespace>,
    metadata:metaRecord
}

Ccfnathan Can you explain what behavior you're seeing a bit more? It sounds like the upser...

A

Andrew Mayne•10/6/23, 9:56 PM

I get a null value when I try to get the metadata. I can't tell if it's not added or not found:

I can retrieve the id and score, but not the meta.

Ccfnathan for (2): do you mean your Worker loses your Vectorize binding (e.g. `env.VECTORI...

A

Andrew Mayne•10/6/23, 10:00 PM

npm install wrangler@latest install 3.6.0, so that's what I've been using.

npm install wrangler@3.11.0 also installs 3.6.0.

npx wrangler deploy works though.

To be honest, I may be completely ignorant of how npm install works.

Ccfnathan we'll dig a bit more into (1) and get back to you. for (2), 3.11.0 should be ava...

A

Andrew Mayne•10/6/23, 11:31 PM

Thank you!

K

Keiagi•10/7/23, 7:45 AM

Hi, I just found this open-beta awesome project for a serverless vector database!
Is anyone developing python SDK? I couldn't find one!

KKeiagi Hi, I just found this open-beta awesome project for a serverless vector database...

M

Matt Silverlock•10/7/23, 12:09 PM

Vectorize has a REST API that you can call from Python.

KKeiagi Hi, I just found this open-beta awesome project for a serverless vector database...

M

Matt Silverlock•10/7/23, 12:10 PM

https://developers.cloudflare.com/vectorize/learning/insert-vectors/#http-api - example

Insert vectors · Vectorize

Vectorize indexes allow you to insert vectors at any point: Vectorize will optimize the index behind the scenes to ensure that vector search remains …

MMatt Silverlock Vectorize has a REST API that you can call from Python.

K

Keiagi•10/7/23, 2:11 PM

Great! Thanks for sharing, Matt!
I hope that python SDK will be available in the near future, but I will try REST API first in the meantime, because serverless vector db is awesome!

JJohnny Is anyone else running into the issue of `env.VECTORIZE_INDEX` not being loaded ...

C

craigsdennis•10/7/23, 3:37 PM

I'm thinking you should remove the comment on the binding line and try again.

S

Sobev•10/9/23, 9:01 AM

wrangler vectorize create vector-index --dimensions=768 --metric=cosine

X [ERROR] Unknown arguments: dimensions, metric, vectorize, create, vector-index
why i get an error

SSobev wrangler vectorize create vector-index --dimensions=768 --metric=cosine X [ERRO...

H

Hello, I’m Allie!•10/9/23, 9:11 AM

What's your

wrangler

wrangler

version?

HHello, I’m Allie!What's your `wrangler` version?

S

Sobev•10/9/23, 9:18 AM

3.8.0, ok i upgraded ,it worked

J

juand4.dev•10/10/23, 12:50 AM

I'm following this the get-starter https://developers.cloudflare.com/vectorize/get-started/embeddings/,

But when I do

npx wrangler vectorize create embeddings-index --dimensions=768 --metric=cosine

npx wrangler vectorize create embeddings-index --dimensions=768 --metric=cosine

npx wrangler vectorize create embeddings-index --dimensions=768 --metric=cosine

npx wrangler vectorize create embeddings-index --dimensions=768 --metric=cosine, I got

"

Creating index: 'tutorial-index'

✘ [ERROR] A request to the Cloudflare API (/accounts/..../vectorize/indexes) failed.

vectorize.not_entitled [code: 1005]"

I'm new, so I think I'm missing some configuration !!

K

kian•10/10/23, 12:50 AM

Do you have a paid Workers plan?

J

juand4.dev•10/10/23, 1:18 AM

Oh no, I just thought that had a "free layer" :c

-> I figure it out, it work just for paid worker planes, for now. And CF is planning to launch a free tier soon.

So, someone know when is It planed to come?

C

charl.dev•10/10/23, 8:47 AM

I need some info around the security due diligence etc around vectorize, like is the data encrypted, is there encryption at rest, GDPR, CCPA, HIPAA compliance.

Is there a doc or something we can give to our infosec guys?

K

kian•10/10/23, 8:50 AM

The information generic to Cloudflare is likely all that is available so far - https://www.cloudflare.com/en-gb/trust-hub/

Assuming that Vectorize is built on top of the existing primitives like Workers, Durable Objects and Workers KV then there will be information that already covers those.

Generally all of Cloudflare's offerings are covered under Standard Contractual Clauses with regards to GDPR.

C

charl.dev•10/10/23, 8:57 AM

thanks kian, just fyi there's a dead link on cloudflare.com/en-gb/ssl/

C

charl.dev•10/10/23, 8:58 AM

ill dm you not to spam here

C

charl.dev•10/10/23, 2:24 PM

whats the metadata size limit, im doing something similar as Alex but splitting into like 1000 chars with overlap and storing in metadata.

e.g. metadata: { pageContent: 'content here' }

Ccharl.dev whats the metadata size limit, im doing something similar as Alex but splitting ...

M

Matt Silverlock•10/10/23, 5:32 PM

All documented here BTW: https://developers.cloudflare.com/vectorize/platform/limits/

C

charl.dev•10/10/23, 5:49 PM

k cool so looking at around 5k - 10k chars, ill do some tests but awesome news

C

charl.dev•10/10/23, 5:52 PM

not sure if you guys have any guidance on optimal chunk size, think langchain textsplitter defaults to 1000 with 200 overlap. Some supabase pgvector articles kinda saying the same, 500 - 1000 chars..

have you done any testing in terms of this?

Ccharl.dev not sure if you guys have any guidance on optimal chunk size, think langchain te...

O

Old Man Maker•10/10/23, 6:36 PM

Just remember that your chunk size is going to be limited by the input token max on the embedding model if you're using CF hosted models for that - generally, it appears to be around 170 words based on 1-3 tokens per word:
https://developers.cloudflare.com/workers-ai/models/embedding/

Embedding · Cloudflare Workers AI docs

Feature extraction models transform raw data into numerical features that can be processed while preserving the information in the original dataset.

L

lolo_sup•10/11/23, 7:40 AM

how to fix this error? {'result': None, 'success': False, 'errors': [{'code': 1004, 'message': 'vectorize.unknown_content_type'}], 'messages': []}

L

lolo_sup•10/11/23, 7:40 AM

def create_vector(self,index_name,vec):
self.conn.request("POST", f"/client/v4/accounts/xxxxxxxx/vectorize/indexes/{index_name}/insert",vec, headers=self.headers)
res = self.conn.getresponse()
data = res.read()
return json.loads(data.decode("utf-8"))

if name == 'main':
vec = vectorize()
with open("a.ndjson","r") as f:
vecs = f.read()
print(vec.create_vector("test",vecs))

L

lolo_sup•10/11/23, 7:41 AM

this is my code

L

lolo_sup•10/11/23, 7:41 AM

anyone can help me ?

Llolo_sup how to fix this error? {'result': None, 'success': False, 'errors': [{'code': ...

N

Nick•10/11/23, 11:39 AM

Hi Lolo, can we see what headers you are sending? As it looks like you are sending ndjson, either no content-type at all (we'll assume ndjson) or

application/x-ndjson

application/x-ndjson

application/x-ndjson

application/x-ndjson should work

Can we also expect a way to CRUD a vector (row) directly by its id/uid?

Similar Threads

Similar Threads

Similar Threads