Software Engineering Courses (SECourses)•11mo ago

@Furkan Gözükara SECourses Hello I just started training with the 24 GB flux dev model but my storag

@Furkan Gözükara SECourses Hello I just started training with the 24 GB flux dev model but my storage was less only 50gb so i think it can create only 2 checkpoints but I checked the training status and the second checkpoint epoch has already strated even before checkpoint one epochs are totally complete.How to save one checkpoint to huggingface if the second one has also started simultaneously

phoenixOP•2/16/25, 1:34 PM

@Furkan Gözükara SECourses and also once the storage is full what error will I get and how to resume the training

phoenixOP•2/16/25, 1:35 PM

@Furkan Gözükara SECourses actually I have set 4 checkpoints instead of 2

Pphoenix @Furkan Gözükara SECourses Hello I just started training with the 24 GB flux dev...

Furkan Gözükara SECourses•2/16/25, 1:37 PM

i never could make it save to hugging face before automatically from kohya

Furkan Gözükara SECourses•2/16/25, 1:37 PM

once storage is full i dont remember behaviour sadly

phoenixOP•2/16/25, 1:42 PM

@Furkan Gözükara SECourses So can i just stop this process and increase the storage for the same gpu instance and the again start training

Pphoenix @Furkan Gözükara SECourses So can i just stop this process and increase the stor...

Furkan Gözükara SECourses•2/16/25, 1:45 PM

on runpod yes

phoenixOP•2/16/25, 1:45 PM

I am on masscompute

Pphoenix I am on masscompute

Furkan Gözükara SECourses•2/16/25, 1:46 PM

you cant there

Furkan Gözükara SECourses•2/16/25, 1:46 PM

get L40 machine it has way more storage

Furkan Gözükara SECourses•2/16/25, 1:46 PM

or multiple A6000 GPU

Pphoenix I am on masscompute

Furkan Gözükara SECourses•2/16/25, 1:46 PM

by the way

Furkan Gözükara SECourses•2/16/25, 1:46 PM

you can delete a lot of apps there

Furkan Gözükara SECourses•2/16/25, 1:46 PM

all will increase storage

Furkan Gözükara SECourses•2/16/25, 1:47 PM

you can delete all except kohya

FFurkan Gözükara SECourses all will increase storage

phoenixOP•2/16/25, 1:47 PM

As you had said in the long tutorial that checkpoints can be saved midway so I proceeded with 4 checkpoints

Pphoenix As you had said in the long tutorial that checkpoints can be saved midway so I p...

Furkan Gözükara SECourses•2/16/25, 1:48 PM

you can. but you have to monitor

Furkan Gözükara SECourses•2/16/25, 1:48 PM

as soon as saved upload to huggin gface

Furkan Gözükara SECourses•2/16/25, 1:48 PM

and delete :d

Original message was deleted

Furkan Gözükara SECourses•2/16/25, 1:51 PM

it saves the checkpoint

Furkan Gözükara SECourses•2/16/25, 1:51 PM

and continue training

FFurkan Gözükara SECourses and continue training

phoenixOP•2/16/25, 1:52 PM

So is checkpoint 1 completed here?

phoenixOP•2/16/25, 1:52 PM

In the screenshot

Pphoenix So is checkpoint 1 completed here?

Furkan Gözükara SECourses•2/16/25, 1:52 PM

yes saved

Furkan Gözükara SECourses•2/16/25, 1:53 PM

check folder

FFurkan Gözükara SECourses check folder

phoenixOP•2/16/25, 1:53 PM

but why the individual epochs are still in process none is 100% complete

phoenixOP•2/16/25, 1:53 PM

I am getting confused

Pphoenix but why the individual epochs are still in process none is 100% complete

Furkan Gözükara SECourses•2/16/25, 1:54 PM

what do you mean?

phoenixOP•2/16/25, 1:55 PM

i mean for checkpoint 1 the epochs status bar is not fully filled.does every epoch status need to be filled till 100% ?

Pphoenix i mean for checkpoint 1 the epochs status bar is not fully filled.does every epo...

Furkan Gözükara SECourses•2/16/25, 1:55 PM

nope

Furkan Gözükara SECourses•2/16/25, 1:55 PM

it shows entire progress

Furkan Gözükara SECourses•2/16/25, 1:56 PM

it is steps based training actually as you see

phoenixOP•2/16/25, 1:56 PM

ok ok got it so checkpoint is safe to upload to huggingface and deleted from the instance right?

phoenixOP•2/16/25, 1:56 PM

checkpoint 1

Pphoenix ok ok got it so checkpoint is safe to upload to huggingface and deleted from the...

Furkan Gözükara SECourses•2/16/25, 1:57 PM

yes

phoenixOP•2/16/25, 1:59 PM

ok thanks for clearing my doubt doctor

Pphoenix ok thanks for clearing my doubt doctor

Furkan Gözükara SECourses•2/16/25, 1:59 PM

sure no issues

phoenixOP•2/16/25, 2:09 PM

@Furkan Gözükara SECourses is there any way to pause the training for a while and resume again

phoenixOP•2/16/25, 2:09 PM

in masscompute

Pphoenix @Furkan Gözükara SECourses is there any way to pause the training for a while an...

Furkan Gözükara SECourses•2/16/25, 2:10 PM

sadly not

phoenixOP•2/16/25, 2:11 PM

whats the fastest way to upload 24 gb to huggingface ,or is possible to save to my pc local drive ?

Pphoenix whats the fastest way to upload 24 gb to huggingface ,or is possible to save to ...

Furkan Gözükara SECourses•2/16/25, 2:12 PM

we have notebook

Furkan Gözükara SECourses•2/16/25, 2:12 PM

https://www.patreon.com/posts/104672510

Patreon

Hugging Face Upload / Download Notebook - Supports Private Reposito...

Get more from SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts on Patreon

Furkan Gözükara SECourses•2/16/25, 5:44 PM

https://www.reddit.com/r/SECourses/comments/1iqxitr/started_10_new_trainings_on_flux_dev_model_to/

From the SECourses community on Reddit: Started 10 new trainings on...

Explore this post and more from the SECourses community

FFurkan Gözükara SECourses https://www.reddit.com/r/SECourses/comments/1iqxitr/started_10_new_trainings_on_...

thisisrahul.•2/16/25, 6:52 PM

What is different in this training?

Tthisisrahul.What is different in this training?

Furkan Gözükara SECourses•2/16/25, 7:06 PM

optimizer

Furkan Gözükara SECourses•2/16/25, 7:06 PM

we will see if they will perform better or not

FFurkan Gözükara SECourses optimizer

AiInfluence•2/16/25, 10:59 PM

Adam?

AAiInfluence Adam?

Furkan Gözükara SECourses•2/16/25, 11:17 PM

i am testing adam varients and lion

wardensc2•2/17/25, 2:08 AM

can you do some test about this, finetuning on this flux dev version

wardensc2•2/17/25, 2:08 AM

https://www.reddit.com/r/StableDiffusion/comments/1iqtoag/an_abliterated_version_of_flux1dev_that_reduces/

From the StableDiffusion community on Reddit: An abliterated versio...

Explore this post and more from the StableDiffusion community

@Furkan Gözükara SECourses Hello I just started training with the 24 GB flux dev model but my storag

Similar Threads

Similar Threads

Similar Threads