Software Engineering Courses (SECourses)•15mo ago

whats the best way to character flux caption? replicate / use joy captioning or simply only use an i

whats the best way to character flux caption? replicate / use joy captioning or simply only use an instance prompt + class prompt ?

Ddsienra Update https://huggingface.co/nyanko7/flux-dev-de-distill Lora training, removin...

druhl•10/20/24, 5:06 AM

My experience of training with de_distill:

1) It adds a sharpness to the image. The images are more noisier, better realism, and harsh reality kind. TBH, that is not always a good thing. If you like or do not like an image, it is subjective>
2) The ability to modify CFG can give you vastly varying results for the same seed. Its CFG scale is much, much tamer than dev2pro and produces linear effects (as you would expect) if you increase or decrease it.
3) The additional time it adds to inference is very frustrating. Original dev said 60+ steps, and he is right. I got good results at Step 70. You can get the generation much faster on flux_dev with 25 to 42 steps. Adding steps adds time to an already slower generation speed.
4) You need to use extra settings during generations like thresholding, which adds additional complexity to the traditional step-cfg system.

druhl•10/20/24, 5:08 AM

Dreambooth on flux_dev:

druhl•10/20/24, 5:08 AM

Dreambooth on flux_dev_dedistill:

NNick44 whats the best way to character flux caption? replicate / use joy captioning or ...

Nick44OP•10/20/24, 6:19 AM

also lets say im training to make a consistent yet diverse race (ie different robots as opposed to a consistent singular character robot like Optimus Prime) for a flux lora / DB . its best to only use the instance prompt: "ohwx" + no captions + instance : "robot"

ar.1•10/20/24, 9:52 AM

what do i fill for class prompt and instance prompt for finetuning on photography images for realism style, not like a single character

Vinch•10/20/24, 10:04 AM

We all know from Dr. Furkan Gözükara's work that it typically takes 15 images, 3000 steps, and a learning rate of 4e-6 to train a model to capture a face. However, training something as complex as an alternate futuristic reality is an entirely different challenge. The dataset size, steps, and learning rate required for this are much more demanding.

In my case, with a dataset of 500 images for stage 1 and 1000 images for stage 2, I initially started with a low learning rate (4e-6), fearing overfitting. However, after running into issues like underfitting and wasting valuable resources, I learned an important lesson: starting with a high learning rate (like 4e-5) allows you to capture the core of the concept much faster, even if it overfits. After that, you can restart with a lower learning rate to fine-tune the details.

In fact, starting with a low learning rate to avoid overfitting ended up costing me both time and money. It takes longer to realize you're underfitting, and by the time you do, you've already invested significant resources. Allowing the model to overfit initially and then restarting with a lower LR is much more cost-effective and efficient.

That said, I’m still open to learning if this insight can be further optimized. I tend to favor restarting rather than fine-tuning because of my perfectionist tendencies, but there may be a balance that I’m missing. For now, the strategy of beginning with a high LR and adjusting downwards has proven to be the most effective approach for me.

VVinch We all know from **Dr. Furkan Gözükara's work** that it typically takes 15 image...

druhl•10/20/24, 10:45 AM

Makes sense.

druhl•10/20/24, 10:46 AM

With overfit, you have a bad product. But at least you have a product.

NNick44 whats the best way to character flux caption? replicate / use joy captioning or ...

Furkan Gözükara SECourses•10/20/24, 11:23 AM

prompt + class prompt

NNick44 also lets say im training to make a consistent yet diverse race (ie different ro...

Furkan Gözükara SECourses•10/20/24, 11:24 AM

yes but try to find a class prompt that will match better

Aar.1 what do i fill for class prompt and instance prompt for finetuning on photograph...

Furkan Gözükara SECourses•10/20/24, 11:25 AM

you can test different class prompts on dev model and see which one generates more resembaling images to yours

VVinch We all know from **Dr. Furkan Gözükara's work** that it typically takes 15 image...

Furkan Gözükara SECourses•10/20/24, 11:26 AM

it is true every dataset is different, but i tested my LR on both 15 and 256 images worked very good. are you doing fine tuning or lora? your LR difference is huge 10 times

FFurkan Gözükara SECourses you can test different class prompts on dev model and see which one generates mo...

ar.1•10/20/24, 11:27 AM

ye I just testing with something for now, but my biggest question is how many steps and epoch should it need with 10k images.

Aar.1 ye I just testing with something for now, but my biggest question is how many st...

Furkan Gözükara SECourses•10/20/24, 11:27 AM

10k images very likely not work

Furkan Gözükara SECourses•10/20/24, 11:27 AM

i tested 10k images here with our reg images : https://huggingface.co/MonsterMMORPG/Big_FLUX_Fine_Tuning_Experiments/tree/main

MonsterMMORPG/Big_FLUX_Fine_Tuning_Experiments at main

Furkan Gözükara SECourses•10/20/24, 11:27 AM

altough my attempt was general fine tuning

Furkan Gözükara SECourses•10/20/24, 11:27 AM

style may work

FFurkan Gözükara SECourses i tested 10k images here with our reg images : https://huggingface.co/MonsterMMO...

ar.1•10/20/24, 11:28 AM

did it work ?

Aar.1 did it work ?

Furkan Gözükara SECourses•10/20/24, 11:29 AM

well i had to use a very low LR

Furkan Gözükara SECourses•10/20/24, 11:29 AM

i did 5 epochs of 10k images

Furkan Gözükara SECourses•10/20/24, 11:29 AM

didnt see significant improvement or anything

Furkan Gözükara SECourses•10/20/24, 11:29 AM

regular LR destroyed the model

FFurkan Gözükara SECourses i did 5 epochs of 10k images

ar.1•10/20/24, 11:30 AM

maybe needs more

ar.1•10/20/24, 11:30 AM

with very low lr

ar.1•10/20/24, 11:30 AM

like 100 epoch

Aar.1 maybe needs more

Furkan Gözükara SECourses•10/20/24, 11:30 AM

yep

Furkan Gözükara SECourses•10/20/24, 11:30 AM

possibly

FFurkan Gözükara SECourses yep

ar.1•10/20/24, 11:30 AM

what was the reason you stopped at 5 ?

Aar.1 what was the reason you stopped at 5 ?

Furkan Gözükara SECourses•10/20/24, 11:31 AM

i had to stop to do trainings for fine tuning tutorial

FFurkan Gözükara SECourses i had to stop to do trainings for fine tuning tutorial 😄

ar.1•10/20/24, 11:32 AM

ah haha

ar.1•10/20/24, 11:32 AM

is there much benefit to using H100 vs A6000 for this

ar.1•10/20/24, 11:32 AM

like is the speed improvement worth it you think

Aar.1 like is the speed improvement worth it you think

Furkan Gözükara SECourses•10/20/24, 11:32 AM

well

ar.1•10/20/24, 11:33 AM

its almost 10x the cost

Furkan Gözükara SECourses•10/20/24, 11:33 AM

i think L40S and batch size 7 most sense making

Furkan Gözükara SECourses•10/20/24, 11:33 AM

but on runpod speed was

Furkan Gözükara SECourses•10/20/24, 11:33 AM

way lower than I expect

Furkan Gözükara SECourses•10/20/24, 11:33 AM

maybe it was that pod

Furkan Gözükara SECourses•10/20/24, 11:34 AM

i didnt test h100 speed

Furkan Gözükara SECourses•10/20/24, 11:34 AM

you should test effective step speed

FFurkan Gözükara SECourses i didnt test h100 speed

ar.1•10/20/24, 11:34 AM

ill test it later today and report back

Aar.1 ye I just testing with something for now, but my biggest question is how many st...

Vinch•10/20/24, 11:35 AM

I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e-5 (4e-5 was indeed too high, but better too high than too low), with 4000 steps, I also activated buckets for multiple aspect ratio's (where each bucket must have at least 7 images). The first 10% of steps are warmup and the LR decreases with a cosine. It's looking like 2e-5 is working the best for my major concept.

VVinch I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e...

ar.1•10/20/24, 11:39 AM

how many images total ?

Aar.1 how many images total ?

Vinch•10/20/24, 11:41 AM

I'm currently doing stage 1 that has 556 images. The images in stage 1 has high concept and low details, most of these images are AI generated to really get the concept. For stage 2, I have 2551 images, these images have less of the concept but are high in detail and quality, which are mostly production grade photographs and marketing material for movies and such.

Vinch•10/20/24, 11:41 AM

I don't know what the config will be for stage 2.

VVinch I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e...