Hi, I just found this open-beta awesome project for a serverless vector database! Is anyone developi

Hi, I just found this open-beta awesome project for a serverless vector database!
Is anyone developing python SDK? I couldn't find one!

KKeiagi Hi, I just found this open-beta awesome project for a serverless vector database...

Matt Silverlock•10/7/23, 12:09 PM

Vectorize has a REST API that you can call from Python.

KKeiagi Hi, I just found this open-beta awesome project for a serverless vector database...

Matt Silverlock•10/7/23, 12:10 PM

https://developers.cloudflare.com/vectorize/learning/insert-vectors/#http-api - example

Insert vectors · Vectorize

Vectorize indexes allow you to insert vectors at any point: Vectorize will optimize the index behind the scenes to ensure that vector search remains …

MMatt Silverlock Vectorize has a REST API that you can call from Python.

KeiagiOP•10/7/23, 2:11 PM

Great! Thanks for sharing, Matt!
I hope that python SDK will be available in the near future, but I will try REST API first in the meantime, because serverless vector db is awesome!

JJohnny Is anyone else running into the issue of `env.VECTORIZE_INDEX` not being loaded ...

craigsdennis•10/7/23, 3:37 PM

I'm thinking you should remove the comment on the binding line and try again.

Sobev•10/9/23, 9:01 AM

wrangler vectorize create vector-index --dimensions=768 --metric=cosine

X [ERROR] Unknown arguments: dimensions, metric, vectorize, create, vector-index
why i get an error

SSobev wrangler vectorize create vector-index --dimensions=768 --metric=cosine X [ERRO...

Hello, I’m Allie!•10/9/23, 9:11 AM

What's your

wrangler

wrangler

version?

HHello, I’m Allie!What's your `wrangler` version?

Sobev•10/9/23, 9:18 AM

3.8.0, ok i upgraded ,it worked

juand4.dev•10/10/23, 12:50 AM

I'm following this the get-starter https://developers.cloudflare.com/vectorize/get-started/embeddings/,

But when I do

npx wrangler vectorize create embeddings-index --dimensions=768 --metric=cosine

npx wrangler vectorize create embeddings-index --dimensions=768 --metric=cosine

, I got

"

Creating index: 'tutorial-index'

✘ [ERROR] A request to the Cloudflare API (/accounts/..../vectorize/indexes) failed.

vectorize.not_entitled [code: 1005]"

I'm new, so I think I'm missing some configuration !!

kian•10/10/23, 12:50 AM

Do you have a paid Workers plan?

juand4.dev•10/10/23, 1:18 AM

Oh no, I just thought that had a "free layer" :c

-> I figure it out, it work just for paid worker planes, for now. And CF is planning to launch a free tier soon.

So, someone know when is It planed to come?

charl.dev•10/10/23, 8:47 AM

I need some info around the security due diligence etc around vectorize, like is the data encrypted, is there encryption at rest, GDPR, CCPA, HIPAA compliance.

Is there a doc or something we can give to our infosec guys?

kian•10/10/23, 8:50 AM

The information generic to Cloudflare is likely all that is available so far - https://www.cloudflare.com/en-gb/trust-hub/

Assuming that Vectorize is built on top of the existing primitives like Workers, Durable Objects and Workers KV then there will be information that already covers those.

Generally all of Cloudflare's offerings are covered under Standard Contractual Clauses with regards to GDPR.

charl.dev•10/10/23, 8:57 AM

thanks kian, just fyi there's a dead link on cloudflare.com/en-gb/ssl/

charl.dev•10/10/23, 8:58 AM

ill dm you not to spam here

charl.dev•10/10/23, 2:24 PM

whats the metadata size limit, im doing something similar as Alex but splitting into like 1000 chars with overlap and storing in metadata.

e.g. metadata: { pageContent: 'content here' }

Ccharl.dev whats the metadata size limit, im doing something similar as Alex but splitting ...

Matt Silverlock•10/10/23, 5:32 PM

All documented here BTW: https://developers.cloudflare.com/vectorize/platform/limits/

charl.dev•10/10/23, 5:49 PM

k cool so looking at around 5k - 10k chars, ill do some tests but awesome news

charl.dev•10/10/23, 5:52 PM

not sure if you guys have any guidance on optimal chunk size, think langchain textsplitter defaults to 1000 with 200 overlap. Some supabase pgvector articles kinda saying the same, 500 - 1000 chars..

have you done any testing in terms of this?

Ccharl.dev not sure if you guys have any guidance on optimal chunk size, think langchain te...

Old Man Maker•10/10/23, 6:36 PM

Just remember that your chunk size is going to be limited by the input token max on the embedding model if you're using CF hosted models for that - generally, it appears to be around 170 words based on 1-3 tokens per word:
https://developers.cloudflare.com/workers-ai/models/embedding/

Embedding · Cloudflare Workers AI docs

Feature extraction models transform raw data into numerical features that can be processed while preserving the information in the original dataset.

lolo_sup•10/11/23, 7:40 AM

how to fix this error? {'result': None, 'success': False, 'errors': [{'code': 1004, 'message': 'vectorize.unknown_content_type'}], 'messages': []}

lolo_sup•10/11/23, 7:40 AM

def create_vector(self,index_name,vec):
self.conn.request("POST", f"/client/v4/accounts/xxxxxxxx/vectorize/indexes/{index_name}/insert",vec, headers=self.headers)
res = self.conn.getresponse()
data = res.read()
return json.loads(data.decode("utf-8"))

if name == 'main':
vec = vectorize()
with open("a.ndjson","r") as f:
vecs = f.read()
print(vec.create_vector("test",vecs))

lolo_sup•10/11/23, 7:41 AM

this is my code

lolo_sup•10/11/23, 7:41 AM

anyone can help me ?

Llolo_sup how to fix this error? {'result': None, 'success': False, 'errors': [{'code': ...

Nick•10/11/23, 11:39 AM

Hi Lolo, can we see what headers you are sending? As it looks like you are sending ndjson, either no content-type at all (we'll assume ndjson) or application/x-ndjsonapplication/x-ndjson should work

lolo_sup•10/12/23, 12:22 PM

How to get the total number of vectors stored in vectorize？

madEngineer•10/13/23, 3:09 PM

Is vectorize currently supported for binding to Pages Functions or just regular Workers?

kian•10/13/23, 3:10 PM

Just regular Workers.

Aleksandr S.•10/13/23, 6:01 PM

Hi, first of all, thank you very much for your new AI direction!
I have a question about Vector database best practices.
I added some information about a company and tried to search for it using different questions. For example, one of the embeddings is "Store opening hours: Monday to Saturday: 9:00 AM to 9:00 PM, Sunday: 10:00 AM to 6:00 PM". If I ask different questions like "Are you open on Sundays", the similarity score is quite low, about 0.62 and it doesn't pass SIMILARITY_CUTOFF. Then the final answer is wrong. Do you have any advice on how should I prepare my texts before making embeddings? Should I create separate embeddings for every day of the week? But in that case the question "What are opening hours?" might not return the correct answer. Or do I need to add every day separately, plus my initial embedding with all days together? If I try to add every combination separately, then later I will have a mess in my database and I will lose control.
Please share any suggestions based on your experience. Thanks a lot.

Cc42alyst Is there any way we can query the vectorize data using sql?

Old Man Maker•10/15/23, 3:23 AM

Not sure what you are thinking here - the vectors are a numerical representation of the data, so you need to store the actual data in say a D1 instance (could be other data sources as well such as R2 or KV) - but the point of metadata is to allow you to connect those embedding back to a datasource on retrieval. i.e. get back the vector response, look at the metadata and then look up the data in D1 using SQL

AAleksandr S.Hi, first of all, thank you very much for your new AI direction! I have a quest...

Old Man Maker•10/15/23, 3:40 AM

Short answer is that you need to consider strategy and how to shape the data in your training phase. Similarity is exactly what the name suggests - mathematical similarity in terms of position, generally returned from cosine (or dot product) equations - giving you an approximation of similarity based on vector positions and or directions and or magnitudes of said vectors etc The values that make the calculation possible are from the embeddings model you use - which is where the real magic takes place.

The only way for similarity to happen accurately is for the initial embedding pass to reflect tokens with spatial efficiency in terms of relationships. So when you're asking for a similarity score return of 'What are the opening hours' - the similarity of that ENTIRE sentence to the ENTIRE embedding sentence is what is being computed - you're comparing 5 tokens in your query against 33 tokens in your embedding, so you must expect a much lower score.

The 'similarity' is not some magical logic that is reasoning over what you have asked, it is a pure math calculation.

charl.dev•10/15/23, 6:18 AM

If it’s a once off thing for a client, I’d put the opening hours in the system message. E.g. if the question relates to the opening hours, respond based on the following opening hours..

OOld Man Maker Short answer is that you need to consider strategy and how to shape the data in ...

Matt Silverlock•10/17/23, 12:50 PM

Great answer

cosbgn•10/20/23, 10:14 AM

Can I delete a vector from an index using the API?

Hi, I just found this open-beta awesome project for a serverless vector database! Is anyone developi

Similar Threads

Similar Threads

Similar Threads