Software Engineering Courses (SECourses)•3y ago

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU 0; 47.54 GiB total

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU 0; 47.54 GiB total capacity; 42.79 GiB already allocated; 133.12 MiB free; 46.06 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Furkan Gözükara SECoursesOP•3/7/23, 9:05 PM

gonna test on 80 gb lets see

Sandstormleader•3/7/23, 9:12 PM

Furkan Gözükara SECoursesOP•3/7/23, 9:53 PM

80 failed too :d

Furkan Gözükara SECoursesOP•3/7/23, 9:53 PM

FFurkan Gözükara SECourses torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU...

Haashim•3/7/23, 10:07 PM

This do to dataset size ?

HHaashim This do to dataset size ?

Furkan Gözükara SECoursesOP•3/7/23, 10:07 PM

i think it is due to extension bug

Furkan Gözükara SECoursesOP•3/7/23, 10:07 PM

i opened a thread

Furkan Gözükara SECoursesOP•3/7/23, 10:08 PM

but this machine consumes a lot credits

FFurkan Gözükara SECourses i think it is due to extension bug

Haashim•3/7/23, 10:08 PM

This is why I am looking at an alternative to the extension for training.
I am not good enough with code to keep up with fixes

Furkan Gözükara SECoursesOP•3/7/23, 10:09 PM

true

dave5555•3/7/23, 10:12 PM

i have been using stable tuner for on vast or auto1111 on vast for training, currently both repos are broken for my setup

dave5555•3/7/23, 10:13 PM

can anyone help me translate the google colab kohya notebook to a jupyter notebook i can run on vast?

Ddave5555 i have been using stable tuner for on vast or auto1111 on vast for training, cu...

Haashim•3/7/23, 10:14 PM

Stabletuner is broken? Or cast giving issues?

Haashim•3/7/23, 10:14 PM

I was looking at everydream next possibly on Runpod

dave5555•3/7/23, 10:15 PM

i need to get pull stable tuner local to try the notebook export but dev mentioned in his discord he needed to fix something about notebooks

dave5555•3/7/23, 10:16 PM

i have a working notebook from stable tuner, i wish i could just add in offset noise

dave5555•3/7/23, 10:17 PM

is that possible to just add a little code to change the noise offset to my notebook?

Furkan Gözükara SECoursesOP•3/7/23, 10:17 PM

why dont you use dreambooth?

Furkan Gözükara SECoursesOP•3/7/23, 10:17 PM

on automatic1111

Furkan Gözükara SECoursesOP•3/7/23, 10:17 PM

it works great

dave5555•3/7/23, 10:18 PM

it works good for lora on my 1080, but i want to use vast to not tie up my comp , and also to do normal finetune not lora

dave5555•3/7/23, 10:18 PM

i have been able to run auto1111 on vast up until now i get error

dave5555•3/7/23, 10:19 PM

Traceback (most recent call last):
File "/workspace/sd/launch.py", line 361, in <module>
start()
File "/workspace/sd/launch.py", line 356, in start
webui.webui()
File "/workspace/sd/webui.py", line 240, in webui
app.add_middleware(GZipMiddleware, minimum_size=1000)
File "/workspace/env/lib/python3.10/site-packages/starlette/applications.py", line 135, in add_middleware
raise RuntimeError("Cannot add middleware after an application has started")
RuntimeError: Cannot add middleware after an application has started
[11:57 AM]
trying to run auto1111 on vast , its always worked before , but havent done it in a while ....... it loads the model weights , creates the remote gradio, all looks like usual , but then it adds on this code and kills the port

dave5555•3/7/23, 10:19 PM

i found a thread about this error and people mention changing fast api

dave5555•3/7/23, 10:20 PM

the requirements.txt has correct version

dave5555•3/7/23, 10:20 PM

i tried pip reinstall fast api and still get error

Ddave5555 i need to get pull stable tuner local to try the notebook export but dev mention...

Haashim•3/7/23, 10:21 PM

Have you noticed a difference in results using stabletuner vs a1111 for training ?

dave5555•3/7/23, 10:23 PM

its hard to say , some people in stable tuner mention its better quality, im not sure ...... for me the notebook runs more reliably and quicker setup than installing auto1111 to vast so thats why i used it

dave5555•3/7/23, 10:23 PM

i imagine its both using kohya and should be about same output right?

dave5555•3/7/23, 10:24 PM

i wish i could just learn how to adapt kohya code direct to jupyter notebook

dave5555•3/7/23, 10:25 PM

should be as simple as copying the colab code in to notebook right?

dave5555•3/7/23, 10:26 PM

i ran the first cell and got error, i think its because i need to install pytorch or some other depdndency first

dave5555•3/7/23, 10:27 PM

maybe i should try running the stabletuner notebook first , to install dependencies, and then switch to kohya copy paste

dave5555•3/7/23, 10:29 PM

so basically what stable tuner will do is take your dataset and creates a payload zip, and an ipynb notebook, u simply upload it to your instance , unzip the archive and run the notebook cells

dave5555•3/7/23, 10:33 PM

i will say that lora using offset noise i did on my 1080 seems to be really good , its hard to say how good on its own though because i merged it back in to my finetune merge im building (which is a bunch of finetunes ive made merged together)

dave5555•3/7/23, 10:33 PM

just this one dreambooth with offset noise blended in seems to improve the contrast of the whole model

Ddave5555 i imagine its both using kohya and should be about same output right?

Haashim•3/7/23, 10:47 PM

Oo, I didn’t know stabletuner was using kohya.
I couldn’t get that to work personally yet.

I haven’t had time to really get into finetuning yet but that’s next for me.

Haashim•3/7/23, 10:48 PM

You are a little farther on your journey

HHaashim You are a little farther on your journey

dave5555•3/7/23, 10:51 PM

ive been in to it for a while , 1 step forward 2 steps back it seems ....... the whole space makes me wish i learned basic stuff along time ago

dave5555•3/7/23, 10:52 PM

pretty sure khoya is the main source of the training code that ST and auto1111 use , someone correct me if im wrong

dave5555•3/7/23, 10:53 PM

i would just use colab but i dont even want to find out what a compute credit is, i imagine its probably 4x as expensive as vast

dave5555•3/7/23, 10:54 PM

in the beginning it seemed like all the SD communties were more active and helpful, i came here from reddit , video post

Ddave5555 ive been in to it for a while , 1 step forward 2 steps back it seems ....... the...

Haashim•3/7/23, 11:07 PM

Yeah I feel that. I think all the Knowlege you’ve acquired will pay off as the tools we use become more stable.

Dr Fs tutorials are the only ones that are thorough enough for me to continue learning. Most vids are just whatever the hot topic is at the time.

dave5555•3/7/23, 11:24 PM

i will check out the videos today ....... this new style transfer controlnet looks really cool

dave5555•3/7/23, 11:25 PM

its so insane what you can do with img2img and controlnet

dave5555•3/7/23, 11:27 PM

ive been doing stuff where i start with my photography as img2img and just get lost in layers of control nets using variations of photos and img2img and just lose track of what even goes in to an image so many different steps were taken

dave5555•3/7/23, 11:28 PM

style transfer looks like a cool way to make use of everything youve done in past to influence stuff in future

Thermidor•3/7/23, 11:31 PM

Is anyone having CUDA problems on runpod? I noticed Dr. Gozukara's pod template version in the tutorial videos is web-automatic-1.5.15 but now pods only display the new version web-automatic-2.1.18

Thermidor•3/7/23, 11:32 PM

The latest tutorial isn't working on these, I think

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU 0; 47.54 GiB total

Similar Threads

Similar Threads

Similar Threads