not sure if you guys have any guidance on optimal chunk size, think langchain textsplitter defaults

not sure if you guys have any guidance on optimal chunk size, think langchain textsplitter defaults to 1000 with 200 overlap. Some supabase pgvector articles kinda saying the same, 500 - 1000 chars..

have you done any testing in terms of this?

Ccharl.dev not sure if you guys have any guidance on optimal chunk size, think langchain te...

Old Man Maker•10/10/23, 6:36 PM

Just remember that your chunk size is going to be limited by the input token max on the embedding model if you're using CF hosted models for that - generally, it appears to be around 170 words based on 1-3 tokens per word:
https://developers.cloudflare.com/workers-ai/models/embedding/

Embedding · Cloudflare Workers AI docs

Feature extraction models transform raw data into numerical features that can be processed while preserving the information in the original dataset.

lolo_sup•10/11/23, 7:40 AM

how to fix this error? {'result': None, 'success': False, 'errors': [{'code': 1004, 'message': 'vectorize.unknown_content_type'}], 'messages': []}

lolo_sup•10/11/23, 7:40 AM

def create_vector(self,index_name,vec):
self.conn.request("POST", f"/client/v4/accounts/xxxxxxxx/vectorize/indexes/{index_name}/insert",vec, headers=self.headers)
res = self.conn.getresponse()
data = res.read()
return json.loads(data.decode("utf-8"))

if name == 'main':
vec = vectorize()
with open("a.ndjson","r") as f:
vecs = f.read()
print(vec.create_vector("test",vecs))

lolo_sup•10/11/23, 7:41 AM

this is my code

lolo_sup•10/11/23, 7:41 AM

anyone can help me ?

Llolo_sup how to fix this error? {'result': None, 'success': False, 'errors': [{'code': ...

Nick•10/11/23, 11:39 AM

Hi Lolo, can we see what headers you are sending? As it looks like you are sending ndjson, either no content-type at all (we'll assume ndjson) or

application/x-ndjson

application/x-ndjson

should work

lolo_sup•10/12/23, 12:22 PM

How to get the total number of vectors stored in vectorize？

madEngineer•10/13/23, 3:09 PM

Is vectorize currently supported for binding to Pages Functions or just regular Workers?

kian•10/13/23, 3:10 PM

Just regular Workers.

Aleksandr S.•10/13/23, 6:01 PM

Hi, first of all, thank you very much for your new AI direction!
I have a question about Vector database best practices.
I added some information about a company and tried to search for it using different questions. For example, one of the embeddings is "Store opening hours: Monday to Saturday: 9:00 AM to 9:00 PM, Sunday: 10:00 AM to 6:00 PM". If I ask different questions like "Are you open on Sundays", the similarity score is quite low, about 0.62 and it doesn't pass SIMILARITY_CUTOFF. Then the final answer is wrong. Do you have any advice on how should I prepare my texts before making embeddings? Should I create separate embeddings for every day of the week? But in that case the question "What are opening hours?" might not return the correct answer. Or do I need to add every day separately, plus my initial embedding with all days together? If I try to add every combination separately, then later I will have a mess in my database and I will lose control.
Please share any suggestions based on your experience. Thanks a lot.

Cc42alyst Is there any way we can query the vectorize data using sql?

Old Man Maker•10/15/23, 3:23 AM

Not sure what you are thinking here - the vectors are a numerical representation of the data, so you need to store the actual data in say a D1 instance (could be other data sources as well such as R2 or KV) - but the point of metadata is to allow you to connect those embedding back to a datasource on retrieval. i.e. get back the vector response, look at the metadata and then look up the data in D1 using SQL

AAleksandr S.Hi, first of all, thank you very much for your new AI direction! I have a quest...

Old Man Maker•10/15/23, 3:40 AM

Short answer is that you need to consider strategy and how to shape the data in your training phase. Similarity is exactly what the name suggests - mathematical similarity in terms of position, generally returned from cosine (or dot product) equations - giving you an approximation of similarity based on vector positions and or directions and or magnitudes of said vectors etc The values that make the calculation possible are from the embeddings model you use - which is where the real magic takes place.

The only way for similarity to happen accurately is for the initial embedding pass to reflect tokens with spatial efficiency in terms of relationships. So when you're asking for a similarity score return of 'What are the opening hours' - the similarity of that ENTIRE sentence to the ENTIRE embedding sentence is what is being computed - you're comparing 5 tokens in your query against 33 tokens in your embedding, so you must expect a much lower score.

The 'similarity' is not some magical logic that is reasoning over what you have asked, it is a pure math calculation.

charl.devOP•10/15/23, 6:18 AM

If it’s a once off thing for a client, I’d put the opening hours in the system message. E.g. if the question relates to the opening hours, respond based on the following opening hours..

OOld Man Maker Short answer is that you need to consider strategy and how to shape the data in ...

Matt Silverlock•10/17/23, 12:50 PM

Great answer

cosbgn•10/20/23, 10:14 AM

Can I delete a vector from an index using the API?

cosbgn•10/20/23, 10:15 AM

It seems to be missing somehow

Jerome•10/20/23, 3:20 PM

@cosbgn vector deletion on the REST api is on the backlog. Upsert will ship soon as well.

cosbgn•10/20/23, 6:14 PM

That would be amazing, it's quite complicated now to deal with situation where a user for example deletes a file but I can't delete the vectors for it

cosbgn•10/20/23, 6:14 PM

Upseet is nice, but delete is really really needed.

SSushidata How come this name is failing?

Hello, I’m Allie!•10/21/23, 9:23 AM

I would guess so?

SSushidata but `"name": "bb70",` is fine? Is it because it starts with a number?

Old Man Maker•10/22/23, 11:43 AM

Correct - a valid Javascript variable name must start with a letter or underscore or $

wadefletch•10/23/23, 6:35 PM

If I was only planning to query data via vector similarity, is there a downside to using KV over D1 for the underlying data storage? Would the performance increase be significant enough to justify the lost flexibility?

wadefletch•10/23/23, 6:42 PM

Also, can anyone from CF (@Matt?) give any timeline estimates (can be rough: this month/quarter/year) on when metadata filtering will be available, even in Beta?

Wwadefletch Also, can anyone from CF (@Matt?) give any timeline estimates (can be rough: thi...

Matt Silverlock•10/23/23, 7:20 PM

This quarter, on the nearer side.

Wwadefletch If I was only planning to query data via vector similarity, is there a downside ...

Matt Silverlock•10/23/23, 7:20 PM

Ultimately up to your data access patterns & structure. Impossible to otherwise advise.

MMatt Silverlock This quarter, on the nearer side.

wadefletch•10/23/23, 7:43 PM

Thanks! I am building a multi-tenant RAG application, in which users can upload their own documents. Is the better way to isolate these, a) hybird queries on some namespace property or b) separate indexes for each user? I imagine it would depend on the size of the indicies (aka the # of embeddings per user)? Do you expect the index limit will increase sufficiently to even make this possible?

Victor•10/23/23, 9:11 PM

On the limits page, I don't see betabeta flag next to Maximum index name lengthMaximum index name length, so is this a infrastructure limitation or can this be increased later?

VVictor On the limits page, I don't see `beta` flag next to `Maximum index name length`,...

Matt Silverlock•10/23/23, 9:45 PM

What kind of length do you need and why? 63 characters per name is common across most Cloudflare (and cloud provider) naming - would love to understand the need for longer names

Matt Silverlock•10/23/23, 9:45 PM

(63 = max length of a DNS label)

MMatt Silverlock What kind of length do you need and why? 63 characters per name is common across...

Victor•10/23/23, 9:55 PM

I'm looking for a way to structure the organization of the data in vectorize. Like a SaaS platform would have the tenant id/data space id/source id/resource idtenant id/data space id/source id/resource id since a new vectorize index is created per resource (each vector inside = each line in the resource). If each id is a uuid, that's 16 (uuid length) * 4ids + 3 separators = 67 characters.

Victor•10/23/23, 9:56 PM

That way when queried through code (rest not workers bindings), we can jump to the right vectorize store directly

Victor•10/23/23, 9:57 PM

I mean an alternative is to have just a resource uuid and a lookup table that has the other info in it, but that's an extra pre-call before the vectorize call

VVictor I'm looking for a way to structure the organization of the data in vectorize. Li...

Matt Silverlock•10/23/23, 9:58 PM

Note that Vectorize supports namespaces per index and (soon) metadata filtering. How many indexes overall do you expect? If we bumped one limit (name length) I don’t want you to just run into another. Better to understand what you’re trying to architect here.

not sure if you guys have any guidance on optimal chunk size, think langchain textsplitter defaults

Similar Threads

Similar Threads

Similar Threads