Vectorize is a beta product thus many of the limits are something that will eventually increase but

UUnsmart Vectorize is a beta product thus many of the limits are something that will even...

O

Old Man Maker•3/7/24, 7:55 AM

Correct - there was a breaking change for vector metadata filtering

A

Angel Luis•3/8/24, 3:19 PM

It would be nice to be able to see the vectors inside the Vectorize dashboard like other vector databases like Qdrant

A

Angel Luis•3/8/24, 3:20 PM

or at least from Wrangler, now I'm working blindly, I've to make my own script to get the vectors through the Worker

A

Angel Luis•3/8/24, 4:03 PM

to be honest I've been working 2 hours with Vectorize and it's a disaster. It's like to have a database without access to any row, the only way to have any track is creating your own fetcher with

getByIds()

getByIds()

getByIds()

getByIds(), track exactly what ids you have inserted, and at maximum fetch 20 ids.

A

Angel Luis•3/8/24, 4:04 PM

I've been working with Milvus and Qdrant, and Vectorize is still very far

A

Angel Luis•3/8/24, 4:28 PM

also I've passed by mistake literally 20 times

[ Object, Object ]

[ Object, Object ]

[ Object, Object ]

[ Object, Object ] as id to a vector index, so the only way I've to fix this is removing the entire vector

omg

A

Angel Luis•3/8/24, 4:45 PM

I will share my current script to help others, but in my opinion the service is too experimental to be public...

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

(and this is a tremendous waste of resources...)

A

Angel Luis•3/11/24, 2:36 PM

I've been thinking and in my opinion Vectorize should have its own numeric index as id, like any SQL database, and then with metadata we can skip the need of using a D1 database, like other vector databases systems. Then just adding a simple way to delete vectors via CLI based on filtering would do the trick. But at this point, the only purpose I'm using D1 is for tracking the ID and don't work blindly. With that I mean that for example, I've a Vector index called "image generation papers", and let's say I've the paper A, which occupies 30 vector entries. Then, if paper A needs and update, the only way I've to track it is via D1, fetching its ids based on the column I've provided in the moment of the vectorization

K

Korcan•3/11/24, 7:57 PM

Hello, is it possible to use Vectorize with Cloudflare Pages?

W

Wang•3/12/24, 6:44 AM

@Korcan Sure, you can use the REST API for invocation.

K

knoedelkopp•3/12/24, 2:51 PM

Hi!

I was wondering: are vector IDs unique per index or unique per namespace?

Reading https://developers.cloudflare.com/vectorize/reference/client-api/#vectors , I think it's unique per index?

Cloudflare Docs

Vectorize API · Vectorize

This page covers the Vectorize API available within Cloudflare Workers, including usage examples.

A

A-Social-Ninja-University•3/16/24, 9:14 AM

ninja-mann07.github.ioi

Ccarrot sticks how silly will it be for me to build my website around this vectorize functional...

D

dakom•3/18/24, 11:43 AM

any n00b tips to learn more about how to use vector search for that kinda thing? still getting my feet wet in this area coming from traditional dbs

Kknoedelkopp Hi! I was wondering: **are vector IDs unique per index or unique per namespace?...

J

Jerome•3/22/24, 1:43 PM

yep, unique per index

O

Old Man Maker•3/22/24, 3:39 PM

Hi - can someone check a vector index for me please. I am looking at some logs that show an insert call but the return is :

{
      "message": [
        "Response from Index =>",
        "{\"count\":0,\"ids\":[]}"
      ],
      "level": "log",
      "timestamp": 1711117766889
    },

{
      "message": [
        "Response from Index =>",
        "{\"count\":0,\"ids\":[]}"
      ],
      "level": "log",
      "timestamp": 1711117766889
    },

{
      "message": [
        "Response from Index =>",
        "{\"count\":0,\"ids\":[]}"
      ],
      "level": "log",
      "timestamp": 1711117766889
    },

{
      "message": [
        "Response from Index =>",
        "{\"count\":0,\"ids\":[]}"
      ],
      "level": "log",
      "timestamp": 1711117766889
    },

No other information why the supplied values are not being inserted
What information is needed from me to check this?

V

Victor•3/25/24, 8:30 AM

Hey there, I was wondering if there was any benefit to using namespaces with regards to billing?
Say you have a namespace with a large number of vectors, and a smaller one, it would still be preferable to create separate indexes to reduce the query cost, right?

KKorcan Hello, is it possible to use Vectorize with Cloudflare Pages?

S

schkovich•3/25/24, 11:15 PM

To overcome the lack of Vectorize in Pages, I use Service Bindings as a reverse proxy to the Vectorize database.

In the application code (functions), I have a class that acts as a facade, implementing all the methods from the Vectorize bindings that are, in the background, sending requests to the bound worker, which would proxy the request to the Vectorize database and the response back.

There are downsides to such an approach. The most notable one is that only remote development is supported.

P

pkpio•3/26/24, 2:05 PM

Is topK expected to bump soon? Realistically what value can we expect?

A

Anubhav Singh•3/31/24, 12:28 PM

I have tried almost everything trying to get the

upsert

upsert

upsert

upsert or

insert

insert

insert

insert method to work for me via the REST API or the

cloudflare

cloudflare

cloudflare

cloudflare npm package but keep getting

  errors: [ { code: 1005, message: 'vectorize.unknown_content_type' } ]

errors: [ { code: 1005, message: 'vectorize.unknown_content_type' } ]

  errors: [ { code: 1005, message: 'vectorize.unknown_content_type' } ]

errors: [ { code: 1005, message: 'vectorize.unknown_content_type' } ]

The same code when used via wrangler (and bindings) works flawlessly.

Can anyone please share a working sample of upsert/insert via REST API?

Ccfnathan Are you sure you're specifying the content type (`application/x-ndjson`) when se...

A

Anubhav Singh•4/1/24, 6:02 AM

All the samples on the API docs too want me to submit ndjson files, but I do not want to involve files in the process. If its possible to do this via wrangler's vector bindings, whose code I extracted and tried using without success, it should be possible to use via REST API too!

Ccfnathan We're in the midst of getting things in ship shape for a GA release of Vectorize...

P

pkpio•4/3/24, 9:51 AM

Hoping to see 200. Since LLMs context is expanding, for certain queries we can provide better answers by including more context. For example, tasks like summarize a video transcript. The current 25 docs limit is simply not usable to produce a good enough summary for lack of sufficient information in the context.

Ccfnathan We're in the midst of getting things in ship shape for a GA release of Vectorize...

A

AndyJessop•4/4/24, 8:27 AM

At the current time, 100 would work for my purposes. But with context windows going up exponentially, it's making VectorDB results less and less useful. At some point I'll be able to fit my entire repo into context and that would eclipse any benefit from Vectors. In that light, I think that topK needs to start high and go higher (maybe in proportion to context window size?).

S

sbentley•4/4/24, 3:16 PM

Has anyone come up with a good solution to filter metadata with a logical or? It doesn't look like the vectorize metadata filtering supports it so I'm trying to figure out a good work around.

N

nuno•4/4/24, 4:42 PM

Hey guys, I'm try to insert a vector via REST API, but I'm getting

{
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 4000,
      "message": "vectorize.vector.missing_part"
    }
  ],
  "messages": []
}

{
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 4000,
      "message": "vectorize.vector.missing_part"
    }
  ],
  "messages": []
}

{
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 4000,
      "message": "vectorize.vector.missing_part"
    }
  ],
  "messages": []
}

{
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 4000,
      "message": "vectorize.vector.missing_part"
    }
  ],
  "messages": []
}

I'm creating a blob to simulate as a file and send it via formdata

const blob = new Blob([JSON.stringify({ id, values }) + '\n'], { type: 'application/x-ndjson' });
console.log('blob :', blob);
const formData = new FormData();
formData.append('file', blob, 'data.ndjson');

const response = await fetch(`${c.env.CLOUDFLARE_API_URL}/accounts/${c.env.CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/${vectorize}/insert`, {
   method: 'POST',
   headers: {
     Authorization: `Bearer ${c.env.CLOUDFLARE_API_TOKEN}`,
   },
   body: formData,
 });

const blob = new Blob([JSON.stringify({ id, values }) + '\n'], { type: 'application/x-ndjson' });
console.log('blob :', blob);
const formData = new FormData();
formData.append('file', blob, 'data.ndjson');

const response = await fetch(`${c.env.CLOUDFLARE_API_URL}/accounts/${c.env.CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/${vectorize}/insert`, {
   method: 'POST',
   headers: {
     Authorization: `Bearer ${c.env.CLOUDFLARE_API_TOKEN}`,
   },
   body: formData,
 });

const blob = new Blob([JSON.stringify({ id, values }) + '\n'], { type: 'application/x-ndjson' });
console.log('blob :', blob);
const formData = new FormData();
formData.append('file', blob, 'data.ndjson');

const response = await fetch(`${c.env.CLOUDFLARE_API_URL}/accounts/${c.env.CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/${vectorize}/insert`, {
   method: 'POST',
   headers: {
     Authorization: `Bearer ${c.env.CLOUDFLARE_API_TOKEN}`,
   },
   body: formData,
 });

const blob = new Blob([JSON.stringify({ id, values }) + '\n'], { type: 'application/x-ndjson' });
console.log('blob :', blob);
const formData = new FormData();
formData.append('file', blob, 'data.ndjson');

const response = await fetch(`${c.env.CLOUDFLARE_API_URL}/accounts/${c.env.CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/${vectorize}/insert`, {
   method: 'POST',
   headers: {
     Authorization: `Bearer ${c.env.CLOUDFLARE_API_TOKEN}`,
   },
   body: formData,
 });

Am I doing something wrong? Any ideas?

S

schkovich•4/12/24, 4:13 PM

Bad Request / Internal Server Error

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

headers = {
    'Content-Type': "application/json",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("GET", f"/client/v4/accounts/{account_id}/vectorize/indexes", headers=headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

headers = {
    'Content-Type': "application/json",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("GET", f"/client/v4/accounts/{account_id}/vectorize/indexes", headers=headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

headers = {
    'Content-Type': "application/json",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("GET", f"/client/v4/accounts/{account_id}/vectorize/indexes", headers=headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

headers = {
    'Content-Type': "application/json",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("GET", f"/client/v4/accounts/{account_id}/vectorize/indexes", headers=headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

returns correctly list of indexes.

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

payload = "/home/ubuntu/jupy/notebooks/test_vector.ndjson"

headers = {
    'Content-Type': "application/x-ndjson",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("POST", f"client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

payload = "/home/ubuntu/jupy/notebooks/test_vector.ndjson"

headers = {
    'Content-Type': "application/x-ndjson",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("POST", f"client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

payload = "/home/ubuntu/jupy/notebooks/test_vector.ndjson"

headers = {
    'Content-Type': "application/x-ndjson",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("POST", f"client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

payload = "/home/ubuntu/jupy/notebooks/test_vector.ndjson"

headers = {
    'Content-Type': "application/x-ndjson",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("POST", f"client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

unfortunately upsert is returning 400 Bad Request

path to ndjson file is correct. there is only a single 768 vector.

!ls -lh test_vector.ndjson
-rw-r--r-- 1 ubuntu ubuntu 17K Apr 12 15:49 test_vector.ndjson

!ls -lh test_vector.ndjson
-rw-r--r-- 1 ubuntu ubuntu 17K Apr 12 15:49 test_vector.ndjson

!ls -lh test_vector.ndjson
-rw-r--r-- 1 ubuntu ubuntu 17K Apr 12 15:49 test_vector.ndjson

!ls -lh test_vector.ndjson
-rw-r--r-- 1 ubuntu ubuntu 17K Apr 12 15:49 test_vector.ndjson

I am getting the same results with curl. curl response differs slightly:

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

My best guess that something might be wrong with ndjson format

[0.032991889864206314, 0.0012868141056969762, ,,,]

[0.032991889864206314, 0.0012868141056969762, ,,,]

[0.032991889864206314, 0.0012868141056969762, ,,,]

[0.032991889864206314, 0.0012868141056969762, ,,,]

Ccfnathan I suspect something is broken with our raw body reader. We'll look into this - i...

S

schkovich•4/12/24, 7:58 PM

Same response.

import requests
from requests_toolbelt import MultipartEncoder

def upsert(url, file_name, content_type, authorization_token):
  """Sends a POST request.
  :param url: URL for the new :class:`Request` object.
  :param file_name: File to be sent/attached in the body of the request.
  :content_type: File content type
  :authorization_token: Cloudflare authorization token
  :return: :class:`Response <Response>` object.
  """
  payload = MultipartEncoder(
    fields = {'field1': ('filename', open(file_name, 'rb'), content_type)}
  )
  headers = {
    'Content-Type': payload.content_type,
    'Authorization': f"Bearer {authorization_token}"
    }
  return requests.post(url, data = payload, headers = headers)
 
upsert(
    f"https://api.cloudflare.com/client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert",
    "test_vector.ndjson",
    "application/x-ndjson",
    vectorize_token
)
<Response [400]>

import requests
from requests_toolbelt import MultipartEncoder

def upsert(url, file_name, content_type, authorization_token):
  """Sends a POST request.
  :param url: URL for the new :class:`Request` object.
  :param file_name: File to be sent/attached in the body of the request.
  :content_type: File content type
  :authorization_token: Cloudflare authorization token
  :return: :class:`Response <Response>` object.
  """
  payload = MultipartEncoder(
    fields = {'field1': ('filename', open(file_name, 'rb'), content_type)}
  )
  headers = {
    'Content-Type': payload.content_type,
    'Authorization': f"Bearer {authorization_token}"
    }
  return requests.post(url, data = payload, headers = headers)
 
upsert(
    f"https://api.cloudflare.com/client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert",
    "test_vector.ndjson",
    "application/x-ndjson",
    vectorize_token
)
<Response [400]>

import requests
from requests_toolbelt import MultipartEncoder

def upsert(url, file_name, content_type, authorization_token):
  """Sends a POST request.
  :param url: URL for the new :class:`Request` object.
  :param file_name: File to be sent/attached in the body of the request.
  :content_type: File content type
  :authorization_token: Cloudflare authorization token
  :return: :class:`Response <Response>` object.
  """
  payload = MultipartEncoder(
    fields = {'field1': ('filename', open(file_name, 'rb'), content_type)}
  )
  headers = {
    'Content-Type': payload.content_type,
    'Authorization': f"Bearer {authorization_token}"
    }
  return requests.post(url, data = payload, headers = headers)
 
upsert(
    f"https://api.cloudflare.com/client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert",
    "test_vector.ndjson",
    "application/x-ndjson",
    vectorize_token
)
<Response [400]>

import requests
from requests_toolbelt import MultipartEncoder

def upsert(url, file_name, content_type, authorization_token):
  """Sends a POST request.
  :param url: URL for the new :class:`Request` object.
  :param file_name: File to be sent/attached in the body of the request.
  :content_type: File content type
  :authorization_token: Cloudflare authorization token
  :return: :class:`Response <Response>` object.
  """
  payload = MultipartEncoder(
    fields = {'field1': ('filename', open(file_name, 'rb'), content_type)}
  )
  headers = {
    'Content-Type': payload.content_type,
    'Authorization': f"Bearer {authorization_token}"
    }
  return requests.post(url, data = payload, headers = headers)
 
upsert(
    f"https://api.cloudflare.com/client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert",
    "test_vector.ndjson",
    "application/x-ndjson",
    vectorize_token
)
<Response [400]>

Let me know if I can provide further details.

S

schkovich•4/12/24, 8:06 PM

same. let me try to insert that vector using wrangler. just to be sure that the file is in the correct format.

S

schkovich•4/12/24, 8:14 PM

@cfnathan Okay, it appears that file format is wrong.

$ npx wrangler vectorize insert hey-hi-index --file test_vector.ndjson 
✨ Uploading vector batch (1 vectors)

✘ [ERROR] A request to the Cloudflare API (/accounts/account_id/vectorize/indexes/hey-hi-index/insert) failed.

  vectorize.invalid_body_payload [code: 1006]

$ npx wrangler vectorize insert hey-hi-index --file test_vector.ndjson 
✨ Uploading vector batch (1 vectors)

✘ [ERROR] A request to the Cloudflare API (/accounts/account_id/vectorize/indexes/hey-hi-index/insert) failed.

  vectorize.invalid_body_payload [code: 1006]

$ npx wrangler vectorize insert hey-hi-index --file test_vector.ndjson 
✨ Uploading vector batch (1 vectors)

✘ [ERROR] A request to the Cloudflare API (/accounts/account_id/vectorize/indexes/hey-hi-index/insert) failed.

  vectorize.invalid_body_payload [code: 1006]

$ npx wrangler vectorize insert hey-hi-index --file test_vector.ndjson 
✨ Uploading vector batch (1 vectors)

✘ [ERROR] A request to the Cloudflare API (/accounts/account_id/vectorize/indexes/hey-hi-index/insert) failed.

  vectorize.invalid_body_payload [code: 1006]

How it should it be formatted?

[0.032991889864206314, 0.0012868141056969762, ,,,]

[0.032991889864206314, 0.0012868141056969762, ,,,]

[0.032991889864206314, 0.0012868141056969762, ,,,]

[0.032991889864206314, 0.0012868141056969762, ,,,]

Sschkovich @cfnathan Okay, it appears that file format is wrong. ```sh $ npx wrangler vecto...

S

schkovich•4/12/24, 8:28 PM

@cfnathan validator says OK

Ccfnathan There's an example here. https://developers.cloudflare.com/vectorize/configurati...

S

schkovich•4/12/24, 8:42 PM

perfect! that's it. wrangler works as well as multipart request. however, the first example is still returning 400. you might have a bug there. i will review the code in quesion once again.

Sschkovich perfect! that's it. wrangler works as well as multipart request. however, the fi...

S

schkovich•4/12/24, 8:47 PM

curl is giving the same error.

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

W

Wang•4/17/24, 5:57 AM

Hi, Cloudflare team

W

Wang•4/17/24, 5:58 AM

I encountered a 502 error when accessing the Vectorize Query API using a Singapore IP, but it works fine with a China IP. How can I resolve this?

W

Wang•4/17/24, 5:58 AM

This is the API being used.
const url =

https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_IDENTIFIER}/vectorize/indexes/${indexName}/query

https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_IDENTIFIER}/vectorize/indexes/${indexName}/query

https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_IDENTIFIER}/vectorize/indexes/${indexName}/query

https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_IDENTIFIER}/vectorize/indexes/${indexName}/query;

N

notorious_tom•4/18/24, 9:41 PM

Does anyone know where I can find OpenAPI spec files for vectorize db?

Nnotorious_tom Does anyone know where I can find OpenAPI spec files for vectorize db?

D

DaniFoldi•4/19/24, 6:30 AM

https://developers.cloudflare.com/api/operations/vectorize-list-vectorize-indexes
if you go back to https://developers.cloudflare.com/api/ you can export the whole cloudflare openapi spec, if you need that

Z

zensin•4/19/24, 9:24 AM

All the examples show populating a

wrangler.toml

wrangler.toml

with:

[[vectorize]]
binding = "VECTOR_INDEX"
index_name = "vector-index"

[[vectorize]]
binding = "VECTOR_INDEX"
index_name = "vector-index"

[[vectorize]]
binding = "VECTOR_INDEX"
index_name = "vector-index"

[[vectorize]]
binding = "VECTOR_INDEX"
index_name = "vector-index"

and then they show using it in a worker script.

Do I have to declare the index in the toml file? What I would instead like to do is dynamically create indexes and let the client specify which one it wants to use (or create) in the request. That way each user can have their own index of whatever.

I know I can create indexes using the REST API. However, idk how to then dynamically load them by name in a worker to use them for search.

Ccfnathan Currently this isn't possible as you're correct, you need to bind to individual ...

S

Saqib•4/23/24, 5:05 PM

@cfnathan is there an ETA on when the limits will be increased — namespaces per index and vectors per index?

S

scotto•4/24/24, 12:04 AM

What is the best way to manage for example if a product has many vectors, for example one per image and we want to obtain the top 10 products without getting the same product multiple times

Sscotto What is the best way to manage for example if a product has many vectors, for ex...

P

pkpio•4/24/24, 9:16 AM

Try MMR search - I think what you are looking for is diversity in results

Ppkpio Try MMR search - I think what you are looking for is diversity in results

S

scotto•4/24/24, 11:35 AM

cloudflare vectorize supports inserting multiples vectors to the same id?

Sscotto cloudflare vectorize supports inserting multiples vectors to the same id?

P

pkpio•4/24/24, 1:21 PM

Don't know

Vectorize is a beta product thus many of the limits are something that will eventually increase but

Similar Threads

Similar Threads

Similar Threads