Software Engineering Courses (SECourses)•3y ago

@ashleyk dataset looks great, i know it does cause i did the regular constant with warmup and did

@ashleyk dataset looks great, i know it does cause i did the regular constant with warmup and did 8bitadam and it worked great, but then i tried adan dadaption with the dadap scheduler and nothing

Zono50OP•6/7/23, 10:13 AM

i can post my json here if need be

Zono50OP•6/7/23, 10:13 AM

db_config.json3.99KB

FFurkan Gözükara SECourses https://twitter.com/GozukaraFurkan/status/1666213927716372486

ai_science_guy•6/7/23, 10:25 AM

That's incredible!
Looking forward to know more.

ZZono50 @ashleyk dataset looks great, i know it does cause i did the regular constant ...

ashley•6/7/23, 10:29 AM

Why don't you just stick to 8bit Adam if that is working for you? Why do you specifically want to use the D-Adaptation optimizers? They are slower. I can't help you with LoRA training unfortunately, I haven't trained any LoRA yet. Maybe that Russian guy was just lucky with his specific dataset he was training using the Facebook D-Adaptation optimizers because I have not been successful in getting them to work yet with regular Dreambooth training either. I guess we need to wait for the good Dr. to test them out and make a video for us.

teachtaha•6/7/23, 10:53 AM

@Dr. Furkan Gözükara how to resume training using last gpt.pth in DLAS cmd interface?

Zono50OP•6/7/23, 10:57 AM

ah ok gotcha

Zono50OP•6/7/23, 10:59 AM

@ashleyk quick question about class instances per image, is that based on the amount of images i'm using, or the amount of steps i'm taking? What would be a good number of class instances per image if i'm using 50 images?

Zono50OP•6/7/23, 11:01 AM

ok so i think i have everything set right now, using 50 images for my dataset this time

db_config.json3.98KB

Zono50OP•6/7/23, 11:02 AM

except for the xformers, i'm gonna change that to default

ZZono50 ok so i think i have everything set right now, using 50 images for my dataset th...

ashley•6/7/23, 11:02 AM

No clue on this, Anzhc told you yesterday, but I don't know myself.

Zono50OP•6/7/23, 11:05 AM

yeah so dataset yesterday was 15, and it used my prior preservation images, today i used a dataset of 50, with 50 class instances per image, and wanted to generate 1750 images, as soon as I dropped that to 10, it said my number of class instances per image is sufficient

Zono50OP•6/7/23, 11:06 AM

so the dadapt with lora only uses like 5 gb of vram, wow

Zono50OP•6/7/23, 11:06 AM

that's impressive

ZZono50 so the dadapt with lora only uses like 5 gb of vram, wow

ashley•6/7/23, 11:29 AM

Which D-Adapt optimizer specifically? In my testing Adan D-Adapt used 32GB of VRAM for regular Dreambooth training vs less than 24GB for any other optimizer.

Zono50OP•6/7/23, 11:45 AM

Not mine. I'm using adan dadapt

Zono50OP•6/7/23, 11:45 AM

And it uses 5gb for lora extended

Zono50OP•6/7/23, 11:46 AM

4.7 with xformers

teachtaha•6/7/23, 11:46 AM

@Dr. Furkan Gözükara how to resume training using last gpt.pth in DLAS cmd interface?

ZZono50 4.7 with xformers

ashley•6/7/23, 11:57 AM

Why would you enable xformers for training if you don't need it?

ashley•6/7/23, 11:58 AM

Guess the either:

1) The extensions implementation of the D-Adaptation Optimizers for full Dreambooth training is not correct.
2) There is a bug in the D-Adaptation Optimizers on Linux.

ashley•6/7/23, 11:59 AM

The Dreambooth extension has too many bugs, has too much code committed to dev branch that isn't even tested at all, and is just generally bad quality and has the worst SDLC of any project in the entire world so I have totally given up on it.

Zono50OP•6/7/23, 12:03 PM

well i mean it KINDA works, so i only used 15 images, and it was somewhat resembling my wife, but not 100%, so I upped the sample images to 51, and retesting

Zono50OP•6/7/23, 12:04 PM

plus it took 4 hours to do 1600 steps yestereday and didn't resemble my wife at all, i've actually gotten better results with xformers,

Zono50OP•6/7/23, 12:04 PM

i'm doing 5100 total steps this time, 50 epochs, 51 images dataset, will do an x/y/z and compare and see how it looks

Zono50OP•6/7/23, 12:05 PM

what's weird is even thouhg i set lora unet leaerning rate to 1 according to the article, it trains at 1.10e-6, but also his article says that adan dapation ignores l/r so i'm assumign thats' why

ZZono50 what's weird is even thouhg i set lora unet leaerning rate to 1 according to the...

ashley•6/7/23, 12:10 PM

The D-Adaptation optimizers adapt the Learning Rate constantly, hence their name. If you install Tensorboard, you will be able to view the graph of the LR and you will see that its not constant.

ashley•6/7/23, 12:12 PM

Here is a Tensorboard graph of the Adan D-Adaptation LR, the LR becomes constant eventually but it basically has a bit of a warm up at the beginning.