Software Engineering Courses (SECourses)•3y ago

I do a test render using ddim sampling method, 30 steps.. It looks like this, which I am ok with. Bu

I do a test render using ddim sampling method, 30 steps.. It looks like this, which I am ok with. But when I select my animatediff, it turns to this dark red backdrop. I keep the seed the same so no idea.. Does this have something to do with controlnet?

AIGambinoOP•12/18/23, 12:52 AM

AAIGambino I do a test render using ddim sampling method, 30 steps.. It looks like this, wh...

Furkan Gözükara SECourses•12/18/23, 1:06 AM

i dont have experience with animateddiff yet

Stefanoid9343•12/18/23, 4:32 AM

Good night group any of you knows how to create those talking avatars with videos? I would like to create those but with training images

goalpwr•12/18/23, 4:56 AM

@Dr. Furkan Gözükara trying to apply this video: https://www.youtube.com/watch?v=343I11mhnXs however when running the script I'm getting the following error

File "N:\AI\Stable_Diffusion\ClassImages\SECourses\Deepface\venv\lib\site-packages\tensorflow\python\tf2.py", line 21, in <module>
from tensorflow.python.platform import _pywrap_tf2
ImportError: DLL load failed while importing _pywrap_tf2: A dynamic link library (DLL) initialization routine failed.

I've already tried installing the zlibwapi.dll into the scripts folder. I have an older computer with an NVIDA 3060 any assitance would be greatly appreciated and thanks for all your very detailed explanations!

YouTubeSECourses

How To Find Best Stable Diffusion Generated Images By Using DeepFac...

If you are also getting tired of trying to find good images among thousands of generated images you don't have to anymore. By using #DeepFace AI library, you can sort images by their similarity to your target images and quickly find the best Stable Diffusion #DreamBooth LoRA trained model generated images. I am explaining everything step by step...

Lorelai AI•12/18/23, 5:10 AM

The old comfy install I had isn't booting up, I tried a fresh install and had the same issue. It's probably a dumb item. I hadn't started it since we changed over Python.

PPflanzen die Tanzen Test from Epoch 09 at 0.85 weight. Pretty nice for a one shot with no negative p...

Hustle with AI•12/18/23, 6:22 AM

Nice work! How are you going about maintaining the consistency in the character?

HHustle with AI Nice work! How are you going about maintaining the consistency in the character?

Pflanzen die Tanzen•12/18/23, 7:19 AM

It’s already trained like this. I can now prompt Mike the boy with or without his dog Willy or just Willy the dog alone. The character stays consistent most of the time. Good enough to pick from a batch of 4 ATM. I will train at least one more version, but I’ll try dreambooth next and add a lot more images which I can now easily generate. The first 24 pages book (prototype) is already done.

PPflanzen die Tanzen It’s already trained like this. I can now prompt Mike the boy with or without hi...

Hustle with AI•12/18/23, 7:36 AM

Ok, I was doing something similar with Dreambooth but havent been able to do so. If you are following any tutorial or decide to make one or have any tips :D, pls do share.
I want to create a comic strip with few characters which can remain consistent across their poses, expressions etc. So looking for ideas around it!

HHustle with AI Ok, I was doing something similar with Dreambooth but havent been able to do so....

Pflanzen die Tanzen•12/18/23, 8:41 AM

Yeah. I might create a tutorial about the process at some point. Not sure if video or blog or even both. . Also my training data and Lora will be open-sourced first research (no commercial use)

Pflanzen die Tanzen•12/18/23, 9:38 AM

First prototype.

Mike_and_the_Castle_small.pdf3.68MB

Pflanzen die Tanzen•12/18/23, 10:03 AM

My process allows creating new character sheets (I trained character sheets, solo images and coupled images.
Looks like it was extremely effective

Hustle with AI•12/18/23, 10:05 AM

Yeah looks great. So you trained this model on Dreambooth? Or is it Lora?

Pflanzen die Tanzen•12/18/23, 10:10 AM

just a standard LoRA with my usual settings. It is v2.

version was trained on 14 images (including solo, coupled and character sheets) all on white background
version included 35 images with more poses (including solo, coupled and character sheets) all on white background

Next will be dreambooth, with ~ 60 - 100 images. Will have to test what works best.

All training images are 4k - 8k, and captioned in a very similar manner

Mike the boyMike the boy
Willy the dogWilly the dog
Mike the boy with his dog WillyMike the boy with his dog Willy
Mike the boy is runningMike the boy is running
Mike the boy with hands in pocket, upper bodyMike the boy with hands in pocket, upper body
Mike the boy with mouth open, screaming, close-upMike the boy with mouth open, screaming, close-up
.. and so on.

See the attached screenshot on repeats. (the caption is done via text files, so only the number is used from those folder names)

I trained 10 epochs of which 7 - 10 at 0.7 - 1.0 weight are actually usable.

Hustle with AI•12/18/23, 10:16 AM

got it, thanks for sharing

I am using img2img with Cheyenne model using Canny controlnet to convert my imgs to cartoon. It is looking good so far. But next step is to train models for more flexibility.

HHustle with AI got it, thanks for sharing 👍 I am using img2img with Cheyenne model using C...

Pflanzen die Tanzen•12/18/23, 10:44 AM

I only trained on whimsical watercolor images (as seen in the results). When I use a photography style on them I get these (1-shot) which is not perfect, but still quite impressive, considering I never trained any photos

SStefanoid9343 Good night group any of you knows how to create those talking avatars with video...

Furkan Gözükara SECourses•12/18/23, 10:50 AM

i worked on a repo but its resolution too low

Furkan Gözükara SECourses•12/18/23, 10:50 AM

https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model

GitHub

GitHub - yoyo-nb/Thin-Plate-Spline-Motion-Model: [CVPR 2022] Thin-P...

[CVPR 2022] Thin-Plate Spline Motion Model for Image Animation. - GitHub - yoyo-nb/Thin-Plate-Spline-Motion-Model: [CVPR 2022] Thin-Plate Spline Motion Model for Image Animation.

Ggoalpwr @Dr. Furkan Gözükara trying to apply this video: https://www.youtube.com/watch?...

Furkan Gözükara SECourses•12/18/23, 10:51 AM

hello welcome. you downloaded from our patreon right?

Furkan Gözükara SECourses•12/18/23, 10:51 AM

can you tell me what do you see when you open cmd and type python?

LLorelai AI The old comfy install I had isn't booting up, I tried a fresh install and had th...

Furkan Gözükara SECourses•12/18/23, 10:51 AM

you can activate venv and install scipy manually

Furkan Gözükara SECourses•12/18/23, 10:52 AM

i also have auto installer : https://www.patreon.com/posts/92013455

PPflanzen die Tanzen First prototype.

Furkan Gözükara SECourses•12/18/23, 10:52 AM

so nice

Pflanzen die Tanzen•12/18/23, 10:53 AM

testing some additional styles on Mike and Willy

Pflanzen die Tanzen•12/18/23, 11:36 AM

PPflanzen die Tanzen First prototype.

Xylber•12/18/23, 12:08 PM

This is fantastic, I would like to do something like this when I finish my current project. Thanks for sharing.

Furkan Gözükara SECourses•12/18/23, 12:27 PM

nice

XXylber This is fantastic, I would like to do something like this when I finish my curre...

Pflanzen die Tanzen•12/18/23, 12:41 PM

let's see if I can document the process. There are several things that I learned along the way which greatly speed up the entire process.
All in all including the dataset and training 2 LoRAs this book took me ~15hrs.
From the current LoRA I could create new books in about 2 hrs. most of the time goes into generating and upscaling.

Though I'm hoping that with a dreambooth I might actually be able to create future stories on-demand via a small online platform.

Feels like we're really close to this (we've been planning and testing for ~6 months now, optimizing every aspect of the process)

Pflanzen die Tanzen•12/18/23, 12:59 PM

At least Mike put on a jacket. Must be freezing with those shorts

PPflanzen die Tanzen just a standard LoRA with my usual settings. It is v2. 1. version was trained o...

Deleted User•12/18/23, 1:12 PM

Will you explain the character sheet

DDeleted User Will you explain the character sheet

Pflanzen die Tanzen•12/18/23, 1:15 PM

not sure, what do you want to know?

PPflanzen die Tanzen not sure, what do you want to know?

Deleted User•12/18/23, 1:17 PM

I understand the mike, willy, mike and will, what does contain character sheet

Deleted User•12/18/23, 1:17 PM

I would like to try

Pflanzen die Tanzen•12/18/23, 1:31 PM

The very first thing I do is create a character

Jill the girl with short ginger hair and pale skin, a joyful character wearing a yellow sundress and black mary JanesJill the girl with short ginger hair and pale skin, a joyful character wearing a yellow sundress and black mary Janes in 1216x832

When I have a prompt that does what I'm aiming for I extend it

Jill the girl with short ginger hair and pale skin, a joyful character wearing a yellow sundress and black mary Janes, consistent (character sheet, multiple views, various expressions, character turnaround:1.2)

Jill the girl with short ginger hair and pale skin, a joyful character wearing a yellow sundress and black mary Janes, consistent (character sheet, multiple views, various expressions, character turnaround:1.2)

in 640x1536

I then run gens until I have several good starters. ( having a sheet generated is crucial for coherence)

You can then inpaint single poses and iterate until you have a decent turnaround and 2 - 3 faces.

side , diagonal, front , rear (all preferably from both sides)
smile, frown, scream, looking away...

All images then need to be fixed for perfection (photoshop and maybe inpainting or generate in Photoshop)
Now we create a new character sheet per hand and run that through img2img until we get some good coherence.
Now we upscale.
Upscale and highres fix the sheet,
slice the sheet and regenerate each pose, expression separately. (be careful to not loose coherence. loopbacks can help but I usually do each iteration by hand)

Finally, we have a dataset of 8 - 15 poses/expressions. All with exact same clothing, all on white background, (background removal is crucial at this point)

Good, we have a dataset: See screenshot with folder names. that's it, no captions, I just used the folder names on the first run, Keep it simple Stupid

I also added the character sheets to my dataset, to allow me to create better sheets with the first LoRA.

Then we move to version2. which is basically a rinse and repeat but with a lot better datasets, more poses, running, jumping, hand positions, but we still use white background on all images.
Get as much info in there as you can provide in best quality.
We still follow the rule: better less images than worse images

That's really the gist of it. No magic involved.

PPflanzen die Tanzen The very first thing I do is create a character `Jill the girl with short ginge...

Stefanoid9343•12/18/23, 1:34 PM

Hi, I read on another videos that captioning helps a lot for the accurate prompt

PPflanzen die Tanzen The very first thing I do is create a character `Jill the girl with short ginge...

Deleted User•12/18/23, 1:34 PM

Thank you very very much fir the explanation

Deleted User•12/18/23, 1:36 PM

I will try first i have to learn then goes deep

SStefanoid9343 Hi, I read on another videos that captioning helps a lot for the accurate prompt

Pflanzen die Tanzen•12/18/23, 1:39 PM

the first run is a waste LoRA. it is only use to get basic coherence. The second LoRa uses captioning.

Mike the boy, white backgroundMike the boy, white background
Mike the boy with hands in pocket, looking to side, white backgroundMike the boy with hands in pocket, looking to side, white background
Mike the boy with his dog Willy are running, white backgroundMike the boy with his dog Willy are running, white background
Willy the dog is barking, white backgroundWilly the dog is barking, white background
...

That's why it's crucial to use white background here. Put the weights where you need them, don't waste the data. The model will be able to separate this

You can add contextual images and scenery when you train a final model on dreambooth.

Accept limitations, emphasize the strength.

So far from all my trainings this is the best result by far.

As mentioned before, This is strictly for illustrations, For photography or actually real people I use a similar but in various points different approach

Pflanzen die Tanzen•12/18/23, 1:43 PM

the idea here is always the same: Describe what should be removable.
For Willy I never describe the blue collar and locket because it is part of Willy.
For Mike I never describe the clothing because they are part of Mike.

As I never want the character to have different clothing unless strictly prompted, this is crucial.

I will also add special images.

Mike the boy dressed as santa clausMike the boy dressed as santa claus
Mike the boy as astronaut in a space suitMike the boy as astronaut in a space suit

This will make outfit switching easier but keep his main outfit unless prompted otherwise

PPflanzen die Tanzen the idea here is always the same: Describe what should be removable. For Willy ...

Stefanoid9343•12/18/23, 2:01 PM

I didn’t have any good results with LORA but with the regular dreambooth yes

SStefanoid9343 I didn’t have any good results with LORA but with the regular dreambooth yes

Pflanzen die Tanzen•12/18/23, 3:15 PM

Yeah, I'm really curious to see the results of Dreambooth on a bigger dataset.

�

𝙻𝚎𝚠•12/18/23, 3:15 PM

Guys

�

𝙻𝚎𝚠•12/18/23, 3:15 PM

i found free image generation ai model with no barriers

�

𝙻𝚎𝚠•12/18/23, 3:15 PM

(sometimes)

PPflanzen die Tanzen the idea here is always the same: Describe what should be removable. For Willy ...

mikemenders•12/18/23, 3:56 PM

A familiar trick. So first you create a "character generator" Lora, which is not flexible, then you use the generated images to create a tagged database and train a flexible Lora.

FFurkan Gözükara SECourses i also have auto installer : https://www.patreon.com/posts/92013455

Lorelai AI•12/18/23, 4:15 PM

Thanks I was looking for that

Lorelai AI•12/18/23, 4:37 PM

This might be a dumb question, if you use dreambooth instead of Lora for the training, is the safetensors still used in the Lora folder like a Lora or the checkpoint folder? Or can you do either? And which is best if so?

FFurkan Gözükara SECourses can you tell me what do you see when you open cmd and type python?

goalpwr•12/18/23, 5:07 PM

@Dr. Furkan Gözükara thanks for the quick response, Yes the file was acquired from patreon, my patreon emails is different than my discord email: aljohns

(venv) N:\AI\Stable_Diffusion\ClassImages\SECourses\Deepface>python
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Mmikemenders A familiar trick. So first you create a "character generator" Lora, which is not...

Pflanzen die Tanzen•12/18/23, 5:35 PM

Yeah. Sort of. It’s all about efficiency. A good base should not take longer than a day including 2 trainings.

vp•12/18/23, 6:12 PM

Udemy also removed access to those that purchased and are offering a refund. Ridiculous.

Pflanzen die Tanzen•12/18/23, 7:36 PM

Some of you asked for it so I wrote a rather detailed article (warning dry content with no images, just plain detailed insights): https://blib.la/blog/mastering-character-consistency-in-stable-diffusion-sdxl-a-detailed-guide

Blibla | Mastering Character Consistency in Stable Diffusion SDXL

Creating consistent characters in AI art, especially with tools like Stable Diffusion SDXL, requires a blend of creativity, precision, and the right techniques.

I do a test render using ddim sampling method, 30 steps.. It looks like this, which I am ok with. Bu

Similar Threads

Similar Threads

Similar Threads