does anyone know what might be causing the training to fail?
does anyone know what might be causing the training to fail?
message.txt2.53KB
Await so, you recreate your UI results with the json api call with 40% succesrate ? SUS
DThe number of images must take at least 30 to catche
DYour catche must be finished inside 30 minutes
DYour training killed in 1577 images catched so resuce your training dataset images to 1550
DHow many training inages in your data set
Eon magic animate i get this error and it stuck like this :
pipeline_animation.py:624: FutureWarning: Accessing config attribute
num_channels_latents = self.unet.in_channels
0%| | 0/25 [00:00<?, ?it/s]
pipeline_animation.py:624: FutureWarning: Accessing config attribute
in_channels directly via 'UNet3DConditionModel' object attribute is deprecated. Please access 'in_channels' over 'UNet3DConditionModel's config object instead, e.g. 'unet.config.in_channels'.num_channels_latents = self.unet.in_channels
0%| | 0/25 [00:00<?, ?it/s]
DDuring extract lora from my checkpoint in kaggle i got error out of memory how to solve the problem and use gpu during the process
HI have 174 images in training dataset, although the regularization I think has 5200? if I re-run will it continue to cache them until it caches them all? or does it start caching from the start each time?
DIt will start catche from 0
Hthe error says 1577/5374 which I think makes sense. 5200 regularization images + 174 of my own training images. not anything to do with my internet upload speed since everything is already uploaded to Kaggle? it still seems quite a low number to timeout at (not even going through all the regularization images)
DReduce your training images to 50
DIt has nothing to do with regulationsimages
DJust reduce your training images to 50
DIf you reduce to 50 images it will do the job for you
Bwhat analysis does your script?
DAnyone will help me with
DWhen i do extract lora from my dreambooth checkpoint error comes that your notebook allocates more memory than available it is not utilising gpu
B1st. I create the payload with UI
2nd. I work on improving the payload, iteratively as described, by adjusting parameters: prompt, noise, CFG, additional scripts, etc.
The project workscope. >100 tuned models; >20 JSON payloads.
So that one payload might work well for a particular model, but poorly for another model.
Success rate (average 40%) - for all models, for all payloads. We generate thousands of images and remove 60%, manually. SD (stand dev) is quite high though. Can be improved, but I don't care that much.
2nd. I work on improving the payload, iteratively as described, by adjusting parameters: prompt, noise, CFG, additional scripts, etc.
The project workscope. >100 tuned models; >20 JSON payloads.
So that one payload might work well for a particular model, but poorly for another model.
Success rate (average 40%) - for all models, for all payloads. We generate thousands of images and remove 60%, manually. SD (stand dev) is quite high though. Can be improved, but I don't care that much.
Bpretty basic statistical analysis: tables with mean and sd
Hhmm reduced to 50 training images, it seems to be timing out at a similar point
message.txt2.42KB
Di used only 13
Deven at 18 it was timing out.. i was training lora sdxl
Dalso check in the argument in kohyass to remove "text_encoder_learner" something
Hshould I try reduce my training images further? (from 50 to 10?) and uncheck "text_encoder_learner"? (where do I find this?) feel like I am missing something fundamental...
D13 with 1024,1024 working good
DAs you are using 512,512 maybe more images around 15 to 20 will also work
AIm really struggling to explain my self, for that i am sorry, but:
-You describe the workflow, super nice!
-But are you saying if you could just use the UI you would hit 100% correct generations, and you are "fighting the results you are getting from UI vs the json payload"?
-You describe the workflow, super nice!
-But are you saying if you could just use the UI you would hit 100% correct generations, and you are "fighting the results you are getting from UI vs the json payload"?
Hthanks, is there anyway to increase the timeout or is this a limitation by Kaggle? ideally I want to use all my training data
DIt’s bug maybe they will solve the problem in near future
DYou are doing 512,512 training
Hyes I am doing 512x512
DTry 25 images
DOr 20
Hyep I'm trying 10, just to confirm reducing images helps the problem
DWhen they give you caching time
DCatching must be finish with in 30 minutes
DI have did 1024,1024 with 15 images too
D13 is best for 1024,1024
DMaybe with 512 more imgaes will also do the job for around 20
Hwill kaggle premium give more caching time? or is it a bug which will probably appear there too?
DI have never used premium
Athe cache time issue will be resolved if you just start running again right after the 30 min
VFurkan
AThat is what @Dr. Furkan Gözükara told me and it worked for me:
Furkan Gözükara SECourses — 11/30/2023 12:16 PM
ok easy to fix
click start training again
it will cache remaining files
Furkan Gözükara SECourses — 11/30/2023 12:16 PM
ok easy to fix
click start training again
it will cache remaining files
Vseeming huggingface changed, i tried to upload
Vbut yousee all models come to this place
Fload it from hugging face
Vit seems uploading to branch, i do not know how to download it from branch
Fi think it is about meta data. you message me about this from patreon?
Vyes i guess so, let me send it to you from patreon