Software Engineering Courses (SECourses)•10mo ago

Has anyone encountered this error while trying to train on an A100 or H100 in RunPod? I've started f

Has anyone encountered this error while trying to train on an A100 or H100 in RunPod? I've started four machines, and all of them give this error.

raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.

MMathew Has anyone encountered this error while trying to train on an A100 or H100 in Ru...

Furkan Gözükara SECourses•2/27/25, 2:41 PM

yes people had it

Furkan Gözükara SECourses•2/27/25, 2:41 PM

i think you need to get a new pod

Furkan Gözükara SECourses•2/27/25, 2:41 PM

i trained on h100 a100 h200

Furkan Gözükara SECourses•2/27/25, 2:41 PM

didnt have that error

Furkan Gözükara SECourses•2/27/25, 2:41 PM

which template you used?

FFurkan Gözükara SECourses which template you used?

MathewOP•2/27/25, 3:22 PM

This one

FFurkan Gözükara SECourses i think you need to get a new pod

MathewOP•2/27/25, 3:23 PM

strange, because I've tested on 4 pods already. I was checking if GPT could help me, but the solutions it provided didn't work

MathewOP•2/27/25, 3:24 PM

I ran it on my personal machine without any issues; it's only on RunPod that I encountered this error

MMathew This one

Furkan Gözükara SECourses•2/27/25, 3:34 PM

ye this is correct one

Furkan Gözükara SECourses•2/27/25, 3:35 PM

i recommend massed compute people never have errors there :d

FFurkan Gözükara SECourses i recommend massed compute people never have errors there :d

MathewOP•2/27/25, 3:40 PM

i'll try, thanks

JotaTerrasa•2/28/25, 8:07 AM

Hi there! Just wanted to share with everyone some results of my latest trained Finetuning.

If you want to try it! Here is the repo: https://huggingface.co/JotaTerrasa/BZ4X/tree/main

Trigger word is: BZ4X

101 Image dataset
100 Epoch

Hardware used for training and inference:

GPU: RTX 4090
RAM: 128GB DDR4
CPU: Ryzen 9 5900XT
Motherboard: MSI MPG B550 GAMING PLUS

If you have any tips for improvement, would be greatly appreciated.

Thanks for sharing your knowledge on this @Furkan Gözükara SECourses, you rock!

MetaKan•2/28/25, 10:58 AM

I've been running tests with various training approaches for days without successful results, and I'd like to share my goal to see if someone can help me.

I'm training a model for interior design. Essentially, I need a flexible model that allows me to train a variety of products in different environments. That is, multiple furniture designs applied to different spaces such as kitchens, bathrooms, dining rooms, etc.

My idea is to use captioning to specify what type of product appears in the image.
Then, I want to be able to generate images of that specific product applied in different settings.

For a single product, it's easy—I train a model with that specific item. But for multiple product families, do you think it's feasible to group them all into a single model?
Do you think that through captioning, the model would later be able to identify each product family correctly?

Thanks a lot!

JotaTerrasa•2/28/25, 12:36 PM

I would say that lora training would be suitable for this situation, for each product and stack them with Lora stack node & divide weights proportionally.

Then in your prompt you’ll need to call to each product trigger word.

JotaTerrasa•2/28/25, 12:38 PM

https://youtu.be/hZCTiZi-EPw?si=fTg65vTJCP2tqdxX the Lora training can be done easily from this workflow via ComfyUI, if not you can use Doc’s Lora Tutorial via Kohya GUI

JJotaTerrasa I would say that lora training would be suitable for this situation, for each pr...

JotaTerrasa•2/28/25, 12:40 PM

For the stacking, you can follow this idea by Mickmumpitz: https://youtu.be/MbQv8zoNEfY?si=gP9b6zPSW5_eBy3y

JJotaTerrasa For the stacking, you can follow this idea by Mickmumpitz: https://youtu.be/MbQv...

JotaTerrasa•2/28/25, 12:40 PM

Minute 11:47

JotaTerrasa•2/28/25, 12:43 PM

Lora training by Doc: https://youtu.be/nySGu12Y05k?si=yWTC8jR9x39YvIfZ you must master class prompt and understand what kind of dataset you need for your project.

YouTubeSECourses

FLUX LoRA Training Simplified: From Zero to Hero with Kohya SS GUI ...

Ultimate Kohya GUI FLUX LoRA training tutorial. This tutorial is product of non-stop 9 days research and training. I have trained over 73 FLUX LoRA models and analyzed all to prepare this tutorial video. The research still going on and hopefully the results will be significantly improved and latest configs and findings will be shared. Please wat...

JotaTerrasa•2/28/25, 12:44 PM

And for the captioning, I would suggest you to be us more precise as possible specially when training Lora with flux, just by natural language, but don’t over complex the semantics.

JotaTerrasa•2/28/25, 12:45 PM

Also, doctor has a really nice captioning Python app script that you can download if you subscribe to his Patreon

JotaTerrasa•2/28/25, 12:45 PM

I would strongly recommend you that

JJotaTerrasa Hi there! Just wanted to share with everyone some results of my latest trained F...

Furkan Gözükara SECourses•2/28/25, 3:56 PM

nice

MMetaKan I've been running tests with various training approaches for days without succes...

Furkan Gözükara SECourses•2/28/25, 3:57 PM

multiple certain things wont work with flux

Furkan Gözükara SECourses•2/28/25, 3:57 PM

it bleeds / collapses

Furkan Gözükara SECourses•2/28/25, 3:57 PM

but you can teach a single thing or a style

JJotaTerrasa Hi there! Just wanted to share with everyone some results of my latest trained F...

JotaTerrasa•3/1/25, 2:17 PM

https://civitai.green/models/1308468/toyota-bz4x-fine-tuning

spirit18•3/1/25, 8:52 PM

Unable to understand which tutorial to watch to do finetuning training on my 4090 Locally.
I tried watching the one with the Yellow Thumbnail.

I ran all 3 in order but GUI is not booting up

Sspirit18 Unable to understand which tutorial to watch to do finetuning training on my 409...

Furkan Gözükara SECourses•3/1/25, 9:00 PM

what is your python version?

spirit18•3/1/25, 9:01 PM

3.10 right now.

Sspirit18 3.10 right now.

Furkan Gözükara SECourses•3/1/25, 9:01 PM

then you must be doing something wrong somewhere

Furkan Gözükara SECourses•3/1/25, 9:01 PM

can you make a quick video how do you install

spirit18•3/1/25, 9:01 PM

Sure, Doing it immediately

spirit18•3/1/25, 9:06 PM

Here

Sspirit18 Here

Furkan Gözükara SECourses•3/1/25, 9:09 PM

it didnt work

Furkan Gözükara SECourses•3/1/25, 9:09 PM

you have error

Furkan Gözükara SECourses•3/1/25, 9:09 PM

Furkan Gözükara SECourses•3/1/25, 9:10 PM

your python installation is wrong

spirit18•3/1/25, 9:10 PM

How do I solve?

Furkan Gözükara SECourses•3/1/25, 9:10 PM

uninstall and reinstall as in this video exactly

Furkan Gözükara SECourses•3/1/25, 9:10 PM

https://youtu.be/DrhUHnYfwC0

YouTubeSECourses

How to Install Python, CUDA, cuDNN, C++ Build Tools, FFMPEG & Git T...

If you are interested in using AI, generative AI applications, open source applications in your computer, than this is the most fundamental and important tutorial that you need. In this tutorial I show and explain how to properly install appropriate Python versions accurately, how to switch between different Python versions, how to install diffe...