Software Engineering Courses (SECourses)•3y ago

@Dr. Furkan Gözükara have you tested xformers/gradient checkpointing as well as Memory Efficient Att

@Dr. Furkan Gözükara have you tested xformers/gradient checkpointing as well as Memory Efficient Attention? I am running without Gradient CH. and xformers but still have memory efficient attention on. Not sure if it decreases quality

rafrafOP•10/8/23, 8:38 PM

Also, wit A6000, on runpod, running 14 images dreambooth LoRA, at 4 batch size, I run into OOM, without gradient checkpointing and xformers. Using class images. Any thoughts if there are any memory optimizations better than these two? What do you recommend @Dr. Furkan Gözükara ??

SSandstormleader 220 professional photos, cropped but not down sized, trained at 768, xformers, 8...

Furkan Gözükara SECourses•10/8/23, 8:50 PM

220 pro photos is the key

NNeuromasters Hello, long time ago I made models and already forgot how to customize them in c...

Furkan Gözükara SECourses•10/8/23, 8:52 PM

i dont knoıw if colab supporting captions never tried

Furkan Gözükara SECourses•10/8/23, 8:52 PM

but colab is working

Furkan Gözükara SECourses•10/8/23, 8:52 PM

i did training few days ago

Furkan Gözükara SECourses•10/8/23, 8:52 PM

https://youtu.be/mnCY8uM7E50

YouTubeSECourses

Transform Your Selfie into a Stunning AI Avatar with Stable Diffusi...

Our Discord : https://discord.gg/HbqgGaZVmr. How to do free Stable Diffusion DreamBooth training on Google Colab for free. If I have been of assistance to you and you would like to show your support for my work, please consider becoming a patron on

https://www.patreon.com/SECourses

Playlist of Stable Diffusion Tutorials, Automatic1111 and Goo...

SSandstormleader @Dr. Furkan Gözükara anyone know if gradio link is working again yet?

Furkan Gözükara SECourses•10/8/23, 8:52 PM

i havent tested but it wasnt working like yesterday

NNarva sdxl gives very good results training specific objects, very versatile

Furkan Gözükara SECourses•10/8/23, 8:52 PM

it is super strong

SStefanoid9343 My training Starts! I jus change the server to 1 x RTX 4090 8 vCPU 31 GB RAM!

Furkan Gözükara SECourses•10/8/23, 8:53 PM

rtx 4090 sper strong on unix

Rrafraf @Dr. Furkan Gözükara have you tested xformers/gradient checkpointing as well as ...

Furkan Gözükara SECourses•10/8/23, 8:53 PM

i think memory efficient attention doing nothing

Rrafraf Also, wit A6000, on runpod, running 14 images dreambooth LoRA, at 4 batch size, ...

Furkan Gözükara SECourses•10/8/23, 8:53 PM

try this

Furkan Gözükara SECourses•10/8/23, 8:53 PM

dont enable gradient checkpointing

Furkan Gözükara SECourses•10/8/23, 8:53 PM

enable xformers

Furkan Gözükara SECourses•10/8/23, 8:53 PM

try to get max batch size

Furkan Gözükara SECourses•10/8/23, 8:53 PM

= max speed

rafrafOP•10/8/23, 8:54 PM

bs 4 i did gradient + xformers. Speed is great already, around 3h compared to 6 hours for batch 2. I will try bs4 with xformers only.

rafrafOP•10/8/23, 8:54 PM

quality is pretty awesome from epoch 1 already

rafrafOP•10/8/23, 8:56 PM

by the way, what is your understanding of max steps. If batch 1 is good at 8000 steps, when you go to batch 2, you have a total of 4000 steps. Do you have a formula already for how many extra steps to give the higher batch to compensate for loss in quality? Generalisation though seems to improve up to a certain number of batch increases.

FFurkan Gözükara SECourses try this

rafrafOP•10/8/23, 9:10 PM

ok, xformers only, no gradient checkpointing goes OOM

Sandstormleader•10/8/23, 9:29 PM

really impressed by how cinematic some of these are coming out

Rrafraf bs 4 i did gradient + xformers. Speed is great already, around 3h compared to 6 ...

Furkan Gözükara SECourses•10/8/23, 9:45 PM

there is no formula

Furkan Gözükara SECourses•10/8/23, 9:45 PM

but bs 2 means in 1 step it makes 2 steps

Furkan Gözükara SECourses•10/8/23, 9:45 PM

3 means 3 steps

Furkan Gözükara SECourses•10/8/23, 9:45 PM

Updated free kaggle notebook : https://twitter.com/GozukaraFurkan/status/1711134402380496953

Furkan Gözükara SECourses•10/8/23, 9:45 PM

https://www.patreon.com/posts/run-on-free-like-88714330

Furkan Gözükara SECourses•10/8/23, 9:46 PM

EyeSpyBekoAI•10/8/23, 10:13 PM

Hey gang, does anyone have experience on downloading safetensor files from runpod, to your local machine? 1 safetensor, which can be 5-7 gb, can take a half hour for me depending on the connection.

kesenteboregi•10/8/23, 10:14 PM

I have this error, I get this error when I train my own face. Does anyone know any solution?

kesenteboregi•10/8/23, 10:35 PM

and now

FFurkan Gözükara SECourses i dont knoıw if colab supporting captions never tried

Neuromasters•10/8/23, 11:46 PM

Are there any parameters that are not up to date?

SuperTurboHero•10/8/23, 11:47 PM

Guys. My RTX3060 12gb is very limited. Can’t really train SDXL models on it and SD1.5 dreambooth training takes overnight. Shoul I try some cloud services? How much vram does Kaggle give? Is it enough for SDXL model training? What do you guys use?

FFurkan Gözükara SECourses there is no formula

rafrafOP•10/8/23, 11:53 PM

cool

rafrafOP•10/9/23, 12:27 AM

by the way, @Dr. Furkan Gözükara you said text encoder is by default turned off, on kohya_ss sdxl. So, does it matter if i set text encoder LR to 0?

rafrafOP•10/9/23, 12:27 AM

Shouldnt affect the trainging correct?

Gerdes of Earth•10/9/23, 12:39 AM

Has anyone ran any tests on different optimizers using Kohya ss? to see whatg provides best results?

GGerdes of Earth Has anyone ran any tests on different optimizers using Kohya ss? to see whatg pr...

Deleted User•10/9/23, 12:54 AM

been testing for a week now, trained about 40 sdxl models, i'd say either go for adafactor or adamw8bit, I know some people have used Prodigy.

Here Test One with adamw8bit is white shirt sample
and test two adafactor is yellow shirt sample