Is there a way when querying vectorize to filter by ids as well as the vector themselves? You mentio

Is there a way when querying vectorize to filter by ids as well as the vector themselves? You mentioned metadata filtering, I guess that's pretty close to what I'm looking for. I formatted my IDs in such a way where they contain both an index and the ID of them in my D1 database.

MMatt Silverlock Soon! Ideally before the next holiday period. Prioritizing some stability/scalab...

divby0•11/30/23, 9:34 PM

Aweomse!

Cchand1012 Is there a way when querying vectorize to filter by ids as well as the vector th...

huw•12/1/23, 3:24 AM

Yeah, this would be supported by metadata filtering or even namespaces. If you only ever want to filter on one of your ‘indices’ at a time, you could totally use namespaces today; for anything more complex you should wait for metadata filtering

Ccfnathan Noted as a feature request! Just so we're on the same page: would metadata filte...

chand1012OP•12/1/23, 3:56 PM

Metadata filtering would be fine, do you have an ETA on when this feature will be ready?

chand1012OP•12/1/23, 4:16 PM

My use case for either feature is I'd like to have one vector index for all the items in a database table, when some of the content from the table may be from both user A and user B, and I only want to query for user A as they're the one making the request.

steyblind•12/1/23, 5:08 PM

Not sure if I missed something here:

1. metadata "included in results" was released
2. on an existing worker, I updated the compatibility date
3. on an existing vectorize db there was no change
4. I updated my worker to use the

includeMetadata

includeMetadata

param

and... no metadata returned?

Exception Thrown @ 12/1/2023, 12:06:34 PM
  (log) {
  count: 2,
  matches: [
    { vectorId: 'hzz2v', score: 0.804275747, vector: [Object] },
    { vectorId: 'aml6o', score: 0.726804477, vector: [Object] }
  ]
}

Exception Thrown @ 12/1/2023, 12:06:34 PM
  (log) {
  count: 2,
  matches: [
    { vectorId: 'hzz2v', score: 0.804275747, vector: [Object] },
    { vectorId: 'aml6o', score: 0.726804477, vector: [Object] }
  ]
}

Cchand1012 Metadata filtering would be fine, do you have an ETA on when this feature will b...

Vy•12/1/23, 5:08 PM

we're hoping it'll be out in December, so stay tuned!

Ssteyblind Not sure if I missed something here: 1. metadata "included in results" was rele...

Vy•12/1/23, 5:09 PM

metadata filtering is not available yet, we're hoping to make it available for new indexes in Dec, so stay tuend

steyblind•12/1/23, 5:09 PM

sorry, I mispoke

steyblind•12/1/23, 5:09 PM

I meant metadata in results

steyblind•12/1/23, 5:09 PM

per vectorize @vaiton13

steyblind•12/1/23, 5:15 PM

let contextResult = await env.DEMO_VECTORDB.query(queryVector, { topK: 5, returnMetadata: true })

let contextResult = await env.DEMO_VECTORDB.query(queryVector, { topK: 5, returnMetadata: true })

steyblind•12/1/23, 5:16 PM

(the exception is thrown when I try to access the metadata prop)

steyblind•12/1/23, 7:27 PM

thank you! good to know it's not me

steyblind•12/1/23, 7:27 PM

godspeed

steyblind•12/1/23, 7:27 PM

I'll go back to metadata in KV

dave•12/2/23, 8:44 PM

https://github.com/langchain-ai/langchain/issues/14179

GitHub

Cloudflare Vectorize Support · Issue #14179 · langchain-ai/langchain

Feature request Looks like it should be doable to have Cloudflare Vectorize support, as there's a HTTP API for Vectorize. https://developers.cloudflare.com/api/operations/vectorize-list-vectori...

dave•12/2/23, 8:44 PM

if anybody official wants to chime in on the ticket to confirm if my assumptions are wrong/right

Ddave https://github.com/langchain-ai/langchain/issues/14179

Matt Silverlock•12/4/23, 2:28 PM

See https://js.langchain.com/docs/integrations/text_embedding/cloudflare_ai

Cloudflare Workers AI | 🦜️🔗 Langchain

If you're deploying your project in a Cloudflare worker, you can use Cloudflare's built-in Workers AI embeddings with LangChain.js.

Matt Silverlock•12/4/23, 2:28 PM

import { CloudflareVectorizeStore } from "langchain/vectorstores/cloudflare_vectorize";

import { CloudflareVectorizeStore } from "langchain/vectorstores/cloudflare_vectorize";

Matt Silverlock•12/4/23, 2:28 PM

and https://js.langchain.com/docs/integrations/vectorstores/cloudflare_vectorize

Cloudflare Vectorize | 🦜️🔗 Langchain

If you're deploying your project in a Cloudflare worker, you can use Cloudflare Vectorize with LangChain.js.

MMatt Silverlock See https://js.langchain.com/docs/integrations/text_embedding/cloudflare_ai

dave•12/4/23, 2:28 PM

Oh I know, I just want it to use Vectorize in Python too!

Matt Silverlock•12/4/23, 2:29 PM

Ah, sorry, misread. You should contribute it

umair mehmood•12/5/23, 9:47 AM

I am getting this when creating vector index from API
Payload

{
  "config": {
    "preset": "@cf/baai/bge-small-en-v1.5",
    "dimensions": 768,
    "metric": "cosine"

  },
  "description": "This is my example index.",
  "name": "example-index"
}

{
  "config": {
    "preset": "@cf/baai/bge-small-en-v1.5",
    "dimensions": 768,
    "metric": "cosine"

  },
  "description": "This is my example index.",
  "name": "example-index"
}

response

{
    "result": null,
    "success": false,
    "errors": [
        {
            "code": 1005,
            "message": "vectorize.not_entitled"
        }
    ],
    "messages": []
}

{
    "result": null,
    "success": false,
    "errors": [
        {
            "code": 1005,
            "message": "vectorize.not_entitled"
        }
    ],
    "messages": []
}

kian•12/5/23, 9:50 AM

Does that account have a Workers Paid plan?

umair mehmood•12/5/23, 9:56 AM

umair mehmood•12/5/23, 9:57 AM

I see now its avilable on paid plan only.

umair mehmood•12/5/23, 9:58 AM

btw from where I can get the account_identifieraccount_identifier?

kian•12/5/23, 10:15 AM

https://developers.cloudflare.com/fundamentals/setup/find-account-and-zone-ids/

Find zone and account IDs · Cloudflare Fundamentals docs

Once you set up a new account and add your domain to Cloudflare, you may need access to your zone and account IDs for API operations.

kian•12/5/23, 10:16 AM

It'll be in the URL when you're on the dashboard, but otherwise you can find it on the sidebar as the docs say.

Ccfnathan There is currently a bug with this: fix is here - https://github.com/cloudflare/...

steyblind•12/5/23, 2:14 PM

this is fixed, but awaiting release, is that correct?

Vy•12/6/23, 5:53 PM

Hi folks, it's a fun day because metadata filtering is now available with Vectorize! Check out the docs here, https://developers.cloudflare.com/vectorize/learning/metadata-filtering/. Happy building and please share any feedback or issues!

Metadata filtering · Vectorize

In addition to providing an input vector to your query, you can also filter by vector metadata associated with every vector. Query results only …

VVy Hi folks, it's a fun day because metadata filtering is now available with Vector...

scotto•12/7/23, 12:12 AM

awesome! namespace filtering is before than metadata filtering?

Sscotto awesome! namespace filtering is before than metadata filtering?

Jerome•12/7/23, 9:33 AM

yes, but it shouldn't matter regarding the set of vectors in the response, as the intersection of both match sets is what's taken into consideration when results are evaluated

JJerome yes, but it shouldn't matter regarding the set of vectors in the response, as th...

scotto•12/7/23, 11:41 AM

perfect, re performance is the same for namespace filtering and metadata filtering ?

Sscotto perfect, re performance is the same for namespace filtering and metadata filteri...

Jerome•12/7/23, 12:46 PM

no, namespace filtering is generally faster than metadata filtering

Old Man Maker•12/7/23, 4:02 PM

refreshing my memory here - we're still looking at 1000 namespaces max per account?

OOld Man Maker refreshing my memory here - we're still looking at 1000 namespaces max per accou...

Jerome•12/7/23, 4:24 PM

yep; https://developers.cloudflare.com/vectorize/platform/limits/

OOld Man Maker refreshing my memory here - we're still looking at 1000 namespaces max per accou...

Matt Silverlock•12/7/23, 7:37 PM

Per index

Frank•12/7/23, 8:01 PM

whats the expected error when we exceed # of vec, namespace per index

Frank•12/7/23, 10:59 PM

hopfully the hard limit on namespace will be lifted post beta

채

채찬•12/8/23, 2:55 AM

After connecting to Vectorize, there is no results data when requesting a D1 query.
Previously, it worked fine.
In the dashboard, the number of query requests is increasing.
Probably a bug.

MMatt Silverlock Per _index_

Old Man Maker•12/8/23, 4:58 AM

Yep, the per index part is important

BBhanu Teja P Awesome. How to request an increase in limits. Right now, it shows 200k vectors ...

Vy•12/8/23, 3:29 PM

With open beta product today, we can't support that large of an index. However, it's something we're working on! Are you often (1) searching across all 5M vectors or (2) search space is smaller but used to operating a single index?

Is there a way when querying vectorize to filter by ids as well as the vector themselves? You mentio

Similar Threads

Similar Threads

Similar Threads