Software Engineering Courses (SECourses)•14mo ago

massed compute

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭massed compute

Furkan Gözükara SECourses•11/17/24, 8:51 PM

they rarely have bad machines

Furkan Gözükara SECourses•11/17/24, 8:51 PM

if you report machine id here

Furkan Gözükara SECourses•11/17/24, 8:51 PM

we can ping the owners

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 8:52 PM

ok thank you, I probably can't find that id now, when it's deleted?

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭ok thank you, I probably can't find that id now, when it's deleted?

Furkan Gözükara SECourses•11/17/24, 8:54 PM

you cant. when that happens report here and ping me :d

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 9:17 PM

thank you! btw, should I still use your downloader zip file for kohya when using massed compute? or just run kohya update then run kohya directly from massed compute?

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭thank you! btw, should I still use your downloader zip file for kohya when using...

Furkan Gözükara SECourses•11/17/24, 9:20 PM

use my zip files

Furkan Gözükara SECourses•11/17/24, 9:20 PM

best and most up to date

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 9:22 PM

excellent, was just about to start, 91 character training pics. 1 character. 200 epochs save at every 25 should be ok? my last character train with 66 pics, when comparing epocs for realism, 25, 50, 75, 100, 125, the last one, epoch 125 was the best, with 100 being the 2nd best.

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭excellent, was just about to start, 91 character training pics. 1 character. 200...

Furkan Gözükara SECourses•11/17/24, 9:26 PM

great

Furkan Gözükara SECourses•11/17/24, 9:26 PM

200 could be overfit for 91 images

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 9:26 PM

so 150?

Furkan Gözükara SECourses•11/17/24, 9:27 PM

150 good

Furkan Gözükara SECourses•11/17/24, 9:27 PM

and compare checkpoints

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 9:27 PM

yes, save every 25

Furkan Gözükara SECourses•11/17/24, 9:27 PM

but if you wna to be 100% sure 200 way to go

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 9:28 PM

sure, for research then

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 9:29 PM

200, save every 25, then do the amazing prompts and grid you taught us

thank you for the quick reply, no one works harder than you in AI, and we are grateful!

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭200, save every 25, then do the amazing prompts and grid you taught us 😄 thank ...

Furkan Gözükara SECourses•11/17/24, 9:29 PM

you can also save like every 20 or 15 depending on storage

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 10:58 PM

how much faster is the h100 compared to the a100 to complete fine tuning?

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 11:00 PM

i'm taking the h100 now, can try a100 next time.

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭how much faster is the h100 compared to the a100 to complete fine tuning?

Furkan Gözükara SECourses•11/17/24, 11:24 PM

Furkan Gözükara SECourses•11/17/24, 11:24 PM

but this looks wrong

Furkan Gözükara SECourses•11/17/24, 11:24 PM

wait

Furkan Gözükara SECourses•11/17/24, 11:24 PM

ok it looks true

Furkan Gözükara SECourses•11/17/24, 11:24 PM

L40s and H100 same

Furkan Gözükara SECourses•11/17/24, 11:25 PM

so L40s would be faster than a100

Furkan Gözükara SECourses•11/17/24, 11:26 PM

actually best case looks like comparing all 3

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 11:34 PM

ha, so more flops means higher speed. how about dual h100, would that make it faster or not?

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 11:48 PM

The speed of H100 GPU , 2.54s/it .

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭The speed of H100 GPU , 2.54s/it .

Furkan Gözükara SECourses•11/17/24, 11:53 PM

really good

Furkan Gözükara SECourses•11/17/24, 11:54 PM

but dont worth it i think

Furkan Gözükara SECourses•11/17/24, 11:54 PM

A6000 gets 6.3 second

Furkan Gözükara SECourses•11/17/24, 11:54 PM

so it is like 2.4 times faster

Furkan Gözükara SECourses•11/17/24, 11:54 PM

but how many times more expensive

Furkan Gözükara SECourses•11/17/24, 11:54 PM

try L40S

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 11:55 PM

awesome, I will

Yeah, I'm overpaying atm, but at least I can use my checkpoint tomorrow noon

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•11/17/24, 11:55 PM

L40S, got it, thanks!

Xcorpion•11/18/24, 2:37 AM

@Furkan Gözükara SECourses Is it possible to put a new optimizer? like this one?https://github.com/derrian-distro/LoRA_Easy_Training_scripts_Backend/blob/main/custom_scheduler/LoraEasyCustomOptimizer/came.py

I saw few people use this optimizer and another thing called REX

GitHub

LoRA_Easy_Training_scripts_Backend/custom_scheduler/LoraEasyCustomO...

The serverside backend created for use with the LoRA Easy Training Scripts Frontend - derrian-distro/LoRA_Easy_Training_scripts_Backend

Xcorpion•11/18/24, 2:37 AM

like this:- ss_optimizer": "LoraEasyCustomOptimizer.came.CAME

Xcorpion•11/18/24, 2:41 AM

creators who train characters use this, like to know how it different from others

Shamsi•11/18/24, 11:05 AM

@Furkan Gözükara SECourses doc, what's limiting my epochs to only 18 and 1600 steps? using the new 48gb configs in kohya.

XXcorpion <@205854764540362752> Is it possible to put a new optimizer? like this one?https...

Furkan Gözükara SECourses•11/18/24, 12:02 PM

you can request from kohya only he can do

SShamsi <@205854764540362752> doc, what's limiting my epochs to only 18 and 1600 steps? ...

Furkan Gözükara SECourses•11/18/24, 12:03 PM

bmaltais error

Furkan Gözükara SECourses•11/18/24, 12:03 PM

reported

Furkan Gözükara SECourses•11/18/24, 12:03 PM

i put a new bat file that switches back to older working commit

Furkan Gözükara SECourses•11/18/24, 12:03 PM

you can use it

Furkan Gözükara SECourses•11/18/24, 12:03 PM

cd kohya_ss

git checkout d47e3e664286314c6b9728bb38b1c7a576211572

Rockerx•11/18/24, 12:42 PM

@Furkan Gözükara SECourses one question for understanding. why would 16GB config be slower than 12gb config. isn't more VRAM better? I am comparing configs and only difference is blocks to swap. should it now slow down training if 12gb config has more blocks to swap ?

FFurkan Gözükara SECourses you can request from kohya only he can do

Rockerx•11/18/24, 12:43 PM

so for v14 it is necessary to use the bat file fix ? or is it only if this error is encountered

massed compute

Similar Threads

Similar Threads

Similar Threads