only in the config.. but i noticed config shows fp16 instead ofbf16?
only in the config.. but i noticed config shows fp16 instead ofbf16?
Fok it should be bf16
Fdoesnt work?
T
Fno not that one
Faccelerate
Fit is accurate
Fwell i dont know what else can be made need further debug. but your gpu also older gen i wonder if that makes difference
Fprobably best to rent a machine on massed compute
Tit was working before. I don't understand wht changed with the recent release 
Ftrain and use on your computer
Fyou can use older
Fjust make fresh install
Fdont run torch 2.5
Fit will stay as torch 2.4.1
Tit ran on torch 2.5 pre-release or beta..
Tis there a way to get that again?
Fah i see
Fit was release candidate
Fi dont know if you can get
TYeah around Config v6/v7 timeframe. everything was working. with torch 2.5 dev I updated to V9 with torch release, and it stopped working
FPyTorch Developer Mailing List
Final 2.5.0 RC for PyTorch core and Domain Libraries is available for download from pytorch-test channel. Reminder of key dates: M5: External-Facing Content Finalized (10/11/24) M6: Release Day (10/17/24) List of Issues included in the Patch Release 2.5.0 can be found here: 2.5.0 Milestone Cherry-Picks included in the Patch Release 2.5.0 ca...
Ftry
Funinstall xformers
Fand use SPDA
TTwill try.. thank you again. its strange if its only supposed to use 32.2GB. why does mine keep climbing.. what is it loading into memory more?
Fye that doesnt make sense
Ffull training without shared vram uses 28.200 MB : 48GB_GPU_28200MB_6.4_second_it_Tier_1
TIDK, it keeps going over 32.2GB then it gives out of memory, on SPDA. I'll keep trying other things.
TIts like its not following the 35 blocks swap
ZYou didn't enable save state for some reason, right?
TI have in some configs, will that use more memory just to save a state? But i was also testing clean configs from the v9 zip
ZNot sure, but I know that when I started, I had a similar issue
Zmemory shot up
ZZtried a couple days later
Zworked out the box
ZOnly thing I remember is checking save state, so I can resume
Zin the first
THmm okay, I'll make sure I'm not using it.. I was thinking to also resume, since this is on a slow GPU. But I'm confident i've been testing with clean configs.. just change model and training folders.
Zand that was even on 64GB ram
Zso I know that was not the issue, lol
TSo for you it just started working? My was working, then i upgraded and can't get it working again
Zright, for me, no update in between. just a day or two, different instance, no changes
Zin my behavior, except checking that save state
Z(at least I'd like to think that was it)
TI have been using it on my other machine with 3090 but will for sure keep it off while i try to get this resolved on my older GPU
Zhaha, yeah, I moved training to Massed Compute
Zbecause couldn't stand the thought of blocking my GPU