SD is bad at text, Should be better when CF support FLUX i think 🤔

K

katopzOP•9/7/24, 2:06 PM

Text Embeddings also useless to me, only en no multi-language model

K

katopzOP•9/7/24, 2:10 PM

Other model also outdate, LoRA stuck at

llama-2

llama-2

, CF ai seem to be useless to me atm.

Kkatopz SD is bad at text, Should be better when CF support FLUX i think 🤔

I

Isaac McFadyen•9/7/24, 5:07 PM

This ^ plus SD uses CLIP for prompting which doesn't like natural language very much... comma seperated tags work better, for example:
Bad:

create me an image that is 512x512, with text that says "my text here" and a red background, with yellow text

create me an image that is 512x512, with text that says "my text here" and a red background, with yellow text

Good:

red background, text saying "my text here", arial font, yellow color

red background, text saying "my text here", arial font, yellow color

K

katopzOP•9/8/24, 4:16 AM

yeah, that too.

U

user•9/8/24, 10:14 AM

is the documentation for stable-diffusion-xl-base-1.0 incorrect?

Cloudflare Docs

stable-diffusion-xl-base-1.0 | Cloudflare Workers AI docs

Diffusion-based text-to-image generative model by Stability AI. Generates and modify images based on text prompts.

U

user•9/8/24, 10:14 AM

it says the response is a binary string but its an object

U

user•9/8/24, 10:15 AM

or is it a blob?

U

user•9/8/24, 10:23 AM

response, not input

U

user•9/8/24, 10:24 AM

https://user.you-sk.id/XMGRn78KexWR

uwu

owo

U

user•9/8/24, 10:25 AM

it claims its a binary string but the type is an object with zero keys

U

user•9/8/24, 10:26 AM

sending a request to the api directly returns an image correctly

U

user•9/8/24, 10:27 AM

but using env.AI does not

U

user•9/8/24, 10:31 AM

ah

U

user•9/8/24, 10:35 AM

Then how come doing

New blob([res])

New blob([res])

returns a blob with raw content of

[object Object]

[object Object]

?

U

user•9/8/24, 10:36 AM

logging JSON.stringify() on the response returns

{}

{}

U

user•9/8/24, 10:41 AM

    const input: TTIInput = {
        prompt: prompt,
        strength: strength,
    };

    const res = await env.AI.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", input);

    const blob = new Blob([res]);

    console.log(await blob.text());

    const input: TTIInput = {
        prompt: prompt,
        strength: strength,
    };

    const res = await env.AI.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", input);

    const blob = new Blob([res]);

    console.log(await blob.text());

    const input: TTIInput = {
        prompt: prompt,
        strength: strength,
    };

    const res = await env.AI.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", input);

    const blob = new Blob([res]);

    console.log(await blob.text());

    const input: TTIInput = {
        prompt: prompt,
        strength: strength,
    };

    const res = await env.AI.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", input);

    const blob = new Blob([res]);

    console.log(await blob.text());

U

user•9/8/24, 10:41 AM

this logs [object Object]

U

user•9/8/24, 10:52 AM

well, when I log JSON.stringify(res) it prints "{}"

U

user•9/8/24, 10:52 AM

so...

U

user•9/8/24, 10:57 AM

well i'd figure it would throw or return nil, or at the bare minimum return "[]"

U

user•9/8/24, 11:01 AM

? wdym

U

user•9/8/24, 11:03 AM

if i do

let a = new Uint8Array(1)
console.log(JSON.stringify(a))

let a = new Uint8Array(1)
console.log(JSON.stringify(a))

let a = new Uint8Array(1)
console.log(JSON.stringify(a))

let a = new Uint8Array(1)
console.log(JSON.stringify(a))

it logs

{"0":0}

{"0":0}

, but when i log the response from ai.run it's just

{}

{}

U

user•9/8/24, 11:06 AM

also, doing

console.log(res instanceof Uint8Array)

console.log(res instanceof Uint8Array)

logs "false", so

U

user•9/8/24, 11:07 AM

:UShrug:

U

user•9/8/24, 11:08 AM

im fairly certain it's an object

L

landon•9/8/24, 11:28 PM

with the run method

const resp = await this.ai.run(model, input);

const resp = await this.ai.run(model, input);

it is returning a

ReadableStream<Uint8Array>

ReadableStream<Uint8Array>

rather than a

Uint8Array

Uint8Array

like the types are saying. I'm having to coerce it into the correct type before being able to manipulate it

here is some working code:

      const resp = (await this.ai.run(model, input)) as unknown;

      let buffer: Uint8Array;

      if (resp instanceof ReadableStream) {
        const chunks: Uint8Array[] = [];
        // eslint-disable-next-line no-restricted-syntax
        for await (const chunk of resp as ReadableStream<Uint8Array>) {
          chunks.push(chunk);
        }
        buffer = new Uint8Array(chunks.reduce((acc, chunk) => acc + chunk.length, 0));
        let offset = 0;
        // eslint-disable-next-line no-restricted-syntax
        for (const chunk of chunks) {
          buffer.set(chunk, offset);
          offset += chunk.length;
        }
      }
      else if (resp instanceof Uint8Array) {
        buffer = resp;
      }
      else {
        throw new Error("Unknown return type for the ai run");
      }

      const base64Image = fromUint8Array(buffer);

      const resp = (await this.ai.run(model, input)) as unknown;

      let buffer: Uint8Array;

      if (resp instanceof ReadableStream) {
        const chunks: Uint8Array[] = [];
        // eslint-disable-next-line no-restricted-syntax
        for await (const chunk of resp as ReadableStream<Uint8Array>) {
          chunks.push(chunk);
        }
        buffer = new Uint8Array(chunks.reduce((acc, chunk) => acc + chunk.length, 0));
        let offset = 0;
        // eslint-disable-next-line no-restricted-syntax
        for (const chunk of chunks) {
          buffer.set(chunk, offset);
          offset += chunk.length;
        }
      }
      else if (resp instanceof Uint8Array) {
        buffer = resp;
      }
      else {
        throw new Error("Unknown return type for the ai run");
      }

      const base64Image = fromUint8Array(buffer);

      const resp = (await this.ai.run(model, input)) as unknown;

      let buffer: Uint8Array;

      if (resp instanceof ReadableStream) {
        const chunks: Uint8Array[] = [];
        // eslint-disable-next-line no-restricted-syntax
        for await (const chunk of resp as ReadableStream<Uint8Array>) {
          chunks.push(chunk);
        }
        buffer = new Uint8Array(chunks.reduce((acc, chunk) => acc + chunk.length, 0));
        let offset = 0;
        // eslint-disable-next-line no-restricted-syntax
        for (const chunk of chunks) {
          buffer.set(chunk, offset);
          offset += chunk.length;
        }
      }
      else if (resp instanceof Uint8Array) {
        buffer = resp;
      }
      else {
        throw new Error("Unknown return type for the ai run");
      }

      const base64Image = fromUint8Array(buffer);

      const resp = (await this.ai.run(model, input)) as unknown;

      let buffer: Uint8Array;

      if (resp instanceof ReadableStream) {
        const chunks: Uint8Array[] = [];
        // eslint-disable-next-line no-restricted-syntax
        for await (const chunk of resp as ReadableStream<Uint8Array>) {
          chunks.push(chunk);
        }
        buffer = new Uint8Array(chunks.reduce((acc, chunk) => acc + chunk.length, 0));
        let offset = 0;
        // eslint-disable-next-line no-restricted-syntax
        for (const chunk of chunks) {
          buffer.set(chunk, offset);
          offset += chunk.length;
        }
      }
      else if (resp instanceof Uint8Array) {
        buffer = resp;
      }
      else {
        throw new Error("Unknown return type for the ai run");
      }

      const base64Image = fromUint8Array(buffer);

L

landon•9/8/24, 11:34 PM

fromUint8Array(buffer)

fromUint8Array(buffer)

is from

js-base64

js-base64

L

landon•9/8/24, 11:38 PM

so you can use that to convert your response to an actual Uint8Array. hopefully they fix the types, or the implementation to return a Uint8Array soon. its gonna break my code when they do though...

L

landon•9/8/24, 11:43 PM

editing it to make it resilient against the inevitable bug fix

Llandon editing it to make it resilient against the inevitable bug fix

U

user•9/9/24, 2:27 AM

U

user•9/9/24, 2:27 AM

thank you much

U

user•9/9/24, 2:28 AM

should this be a bug report?

R

rob•9/9/24, 11:14 AM

Slow af

Llandon with the run method `const resp = await this.ai.run(model, input);` it is return...

R

Raylight•9/9/24, 2:16 PM

Cool! An alternative way to read the stream to completion is

const buffer = await new Response(resp).arrayBuffer();

const buffer = await new Response(resp).arrayBuffer();

.

IIsaac McFadyen This ^ plus SD uses CLIP for prompting which doesn't like natural language very ...

J

JustinNoel•9/10/24, 11:42 AM

Thanks for the replies @katopz and Isaac. I was so hopeful about "Good: red background, text saying "my text here", arial font, yellow color".

I

Isaac McFadyen•9/10/24, 11:52 AM

Yeaaah Stable Diffusion isn't great at text sadly :(

J

Jian•9/10/24, 9:14 PM

Does cloudflare worker ai support custom models? or just using the huggingface api?

JJian Does cloudflare worker ai support custom models? or just using the huggingface a...

I

Isaac McFadyen•9/10/24, 9:46 PM

Custom models are not currently supported. Only models that are added by the Cloudflare team (so not any HuggingFace model, just the ones that are on the list)

_

_querty123_•9/11/24, 12:23 AM

all listed models enter on paid plan? i mean, if i pay all moedls listed will be discount of my usage?

_

_querty123_•9/11/24, 12:23 AM

from my neurons?

_

_querty123_•9/11/24, 12:25 AM

im starting with ia workers, ive created one worker with await env.AI.run("@hf/meta-llama/meta-llama-3-8b-instruct" on it, i want to know: when i call endpoint, usage is discounted from where? here?

IIsaac McFadyen Custom models are not currently supported. Only models that are added by the Clo...

J

Jadugar_Jaggu•9/11/24, 4:57 AM

I think, using custom models will be a more competitive feature, and can summon the whole Ai industry to CF.

CF should look into this

L

lee•9/11/24, 7:31 AM

Can the AI models used by Worker AI only be those provided by Cloudflare officially, or can users upload their own private AI models for use?

V

vin•9/11/24, 9:22 AM

For now users can not upload custom models

V

vin•9/11/24, 9:24 AM

about a year ago, cloudflare introduced a project called Constellation, its APIs are much much lower level https://blog.cloudflare.com/introducing-constellation

The Cloudflare Blog

Introducing Constellation, bringing AI to the Cloudflare stack

Today, we're excited to welcome Constellation to the Cloudflare stack. Constellation allows you to run fast, low-latency inference tasks on pre-trained machine learning models natively on Workers.

V

vin•9/11/24, 9:26 AM

you can upload any small ONNX models to constellation, and do inference things. but it's hardly to use, you have to translate all inference preprocess/postprocess to workers code