Training for days
I want to train my model for days using a single GPU. How do I keep my Jupyter Notebook session to persist even after I close my laptop so that training continues?
JIf its on runpod, then it should just keep running, but ur best bet in case jupyter notebook crash is use something like tmux to execute your script in the background, then u can turn off ur terminal and stuff, and not worry about the script just stopping.
https://youtu.be/nTqu6w2wc68?si=eT995bcH8hXZ7xEi
Make sure you backup some sort of checkpoint to a network drive or third party storage tho, every so often
https://youtu.be/nTqu6w2wc68?si=eT995bcH8hXZ7xEi
Make sure you backup some sort of checkpoint to a network drive or third party storage tho, every so often
YouTubeNetworkChuck
Spin up your next project with Linode: https://ntck.co/linode –You get a $100 Credit good for 60 days as a new user!
I just started using Tmux……it’s amazing! If you use a terminal or CLI in any capacity Tmux will 10x your productivity in 10 seconds. From creating multiple panes and windows with ease to leaving your terminal sessions active as...
I just started using Tmux……it’s amazing! If you use a terminal or CLI in any capacity Tmux will 10x your productivity in 10 seconds. From creating multiple panes and windows with ease to leaving your terminal sessions active as...
Jin case something goes wrong
SWill the notebook keep running if I get disconnected from WiFi or shutdown the laptop?
JIf it in runpod it shouldn't but you can give it a try, b/c you have to imagine
Jits essentially another computer
Jout there
Jjupyter notebook is just a local server running on runpod's infrastructure
Jso even if u disconnect or connect to it, doesn't affect whether it keeps running or not
Mfor long term training I recomend running with ssh and tmux
AYep, tmux is slightly more difficult to use than screen though, so screen may be easier.
