Software Engineering Courses (SECourses)•3y ago

83%|████████████████████████████████▌ | 1516/1818 [30:08<06:00, 1.19s/it] [E ProcessGroupNCCL.

83%|████████████████████████████████▌ | 1516/1818 [30:08<06:00, 1.19s/it]
[E ProcessGroupNCCL.cpp:828] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802663 milliseconds before timing out.

DR.SivaOP•11/27/23, 11:32 AM

DR.SivaOP•11/27/23, 11:40 AM

reduced to 13 images - 521 steps , lets see

DR.SivaOP•11/27/23, 11:41 AM

23 min now ..

DDR.Siva 83%|████████████████████████████████▌ | 1516/1818 [30:08<06:00, 1.19s/it] ...

Furkan Gözükara SECourses•11/27/23, 12:37 PM

when this happens

Furkan Gözükara SECourses•11/27/23, 12:37 PM

is your notebook killed?

Furkan Gözükara SECourses•11/27/23, 12:38 PM

or only gradio

Furkan Gözükara SECourses•11/27/23, 1:16 PM

https://www.linkedin.com/posts/furkangozukara_pixart-alpha-pixart-%CE%B1-looking-amazing-activity-7134892793662951424-LZdU?utm_source=share&utm_medium=member_desktop

Furkan Gözükara on LinkedIn: pixart-alpha (PIXART-α) looking amazin...

pixart-alpha (PIXART-α) looking amazing

prompt : a mighty lion running in new york city

styles

default, anime, digital art

hopefully full tutorial and auto…

Chelsea•11/27/23, 1:20 PM

Can anyone help please, ive created about 10 'deep fake videos so far with no issues at all.
But the past few times my pc just crashes out and restarts about 80% sometimes 64% etc into processing.

Stefanoid9343•11/27/23, 1:21 PM

@Dr. Furkan Gözükara I’m doing this as a hobby but there is ways to make money generating this images and continue the research? and cover the server cost? this is a nice discussion that I guess a lot of people will be interest

Chelsea•11/27/23, 1:22 PM

I cant get a screenshot of my pc because I never know when its going to crash and with it taking a long time for each video I can't sit and watch it constantly

ive screenshotted the YouTube tutorial.

CChelsea I cant get a screenshot of my pc because I never know when its going to crash an...

Chelsea•11/27/23, 1:23 PM

So I select my face, choose my target video, check many faces, and everything runs smoothly then as I say it gets to 60/80 sometimes 90% then crash computer turns off?

CChelsea Can anyone help please, ive created about 10 'deep fake videos so far with no is...

Furkan Gözükara SECourses•11/27/23, 1:24 PM

your PC specs is the problem

Furkan Gözükara SECourses•11/27/23, 1:24 PM

could be undervolted RAM

SStefanoid9343 @Dr. Furkan Gözükara I’m doing this as a hobby but there is ways to make money g...

Furkan Gözükara SECourses•11/27/23, 1:25 PM

sadly i dont know. but some people making money

FFurkan Gözükara SECourses could be undervolted RAM

Chelsea•11/27/23, 1:26 PM

I'm confused as it worked fine, till now

CChelsea I'm confused as it worked fine, till now

Chelsea•11/27/23, 1:30 PM

12th Gen I5
32GB Ram
GeFroce GTX 1660ti
this is a screenshot of a previous video that worked perfectly. Im face swapping on the same video each time

CChelsea I'm confused as it worked fine, till now

Furkan Gözükara SECourses•11/27/23, 1:31 PM

well try to increase ram voltage

Furkan Gözükara SECourses•11/27/23, 1:31 PM

a little bit

Deleted User•11/27/23, 1:50 PM

Is close images datasets is good for dreambooth training or full body or half body images data set is good for woman

DDeleted User Is close images datasets is good for dreambooth training or full body or half bo...

Furkan Gözükara SECourses•11/27/23, 1:56 PM

i prefer to add all

Furkan Gözükara SECourses•11/27/23, 1:56 PM

and after use adetailer during inference

Furkan Gözükara SECourses•11/27/23, 2:06 PM

https://www.patreon.com/posts/93614549?pr=true

FFurkan Gözükara SECourses https://www.linkedin.com/posts/furkangozukara_pixart-alpha-pixart-%CE%B1-looking...

Stefanoid9343•11/27/23, 2:10 PM

this is like a new stable difussion?

SStefanoid9343 this is like a new stable difussion?

Furkan Gözükara SECourses•11/27/23, 2:16 PM

yes

Furkan Gözükara SECourses•11/27/23, 2:16 PM

different model

Furkan Gözükara SECourses•11/27/23, 2:16 PM

can follow prompts better

FFurkan Gözükara SECourses when this happens

DR.SivaOP•11/27/23, 2:32 PM

can confirm when the estimated trime is beyond 30 min. I reduced the images to 13 , which made the caching latency time within 30min (23min to be exact) , it is working now

DR.SivaOP•11/27/23, 2:34 PM

epoch 1/2
steps: 20%|███ | 521/2600 [44:13<2:56:30, 5.09s/it, avr_loss=0.118]
saving checkpoint: /kaggle/temp/models/DB_Gowthu-step00000521.safetensors
steps: 40%|████▊ | 1042/2600 [1:30:07<2:14:45, 5.19s/it, avr_loss=0.117]
saving checkpoint: /kaggle/temp/models/DB_Gowthu-step00001042.safetensors
steps: 50%|██████ | 1300/2600 [1:53:08<1:53:08, 5.22s/it, avr_loss=0.113]
saving checkpoint: /kaggle/temp/models/DB_Gowthu-000001.safetensors

epoch 2/2
steps: 60%|███████▏ | 1563/2600 [2:16:08<1:30:19, 5.23s/it, avr_loss=0.115]
saving checkpoint: /kaggle/temp/models/DB_Gowthu-step00001563.safetensors
steps: 64%|███████▋ | 1677/2600 [2:27:02<1:20:55, 5.26s/it, avr_loss=0.114]

DR.SivaOP•11/27/23, 2:36 PM

So i think there should be a disclaimer for now in your video to limit the images to 13-15 i guess. someone should work on this and see how many images can be within 30min. I am sure 18 images crosses 30 min

DDR.Siva can confirm when the estimated trime is beyond 30 min. I reduced the images to 1...

Furkan Gözükara SECourses•11/27/23, 2:37 PM

ye maybe developer will fix it

FFurkan Gözükara SECourses is your notebook killed?

DR.SivaOP•11/27/23, 2:37 PM

luckily the notebook was on, i went into kohya gui and entered new data set(13 images) directory link and made changes to 521 steps and ran training

DR.SivaOP•11/27/23, 2:38 PM

is it possible to move the generated tensor files to dataset directly ?

DR.SivaOP•11/27/23, 2:38 PM

ie from temp files to dataset ?

DR.SivaOP•11/27/23, 2:39 PM

numbering of safetensor file is weird i guess ? 521/1042/1?/1563

FFurkan Gözükara SECourses https://www.linkedin.com/posts/furkangozukara_pixart-alpha-pixart-%CE%B1-looking...

umuthasanoglu•11/27/23, 2:40 PM

how is the speed of pixart-alpha?

DDeleted User Is close images datasets is good for dreambooth training or full body or half bo...

DR.SivaOP•11/27/23, 2:45 PM

Varied , all types are important , but make it absolutely clear , in none of the photos other than face, nothing else repeats. Last time i trained my lora for my wife , i had few photos with same dress and it made a great impact on the outwhen when generating images. this time