Runpod•9mo ago

Unable to run jupyter on custom docker image

I am basically trying to build my own custom container and run jupyter lab in it. I've copied the configuration from base rundpod image from github. Everything is fine there. The problem occurs when the pod is booted up as it is supposed to start jupyterlab but we get the jupyter command not found issue which is why port 8888 is not ready. First screenshot is from my local machine where i get into the docker and i see jupyter --help works. Second one is from the runpod instance where it complains that jupyter is not found but then when i install in my conda env it only updates certifi version (as you can see from the screenshot) and then after that jupyter command is available again. I'm puzzle by this behaviour. How can a same docker container behave differently in different environments?

19 Replies

Unknown User•9mo ago

Message Not Public

Fazil TOP•9mo ago

the first screenshot is from my docker container running locally. As you can see jupyter is installed. it works fine there. The command not found is happening only on runpod

Fazil TOP•9mo ago

I've digged deep into the start.sh that runpod has, it invokes jupyter if JUPYTER_PASSWORD set. https://github.com/runpod/containers/blob/main/container-template/start.sh#L75 However setting environment variables is happening after that: https://github.com/runpod/containers/blob/main/container-template/start.sh#L96 I'm wondering how that lines gets invoked. Right now i have to open a shell into my pod, do conda install -y jupyter and it just barely updates a file called certifi and then jupyter command can be found. so I launch it and connect it.

GitHub

containers/container-template/start.sh at main · runpod/containers

🐳 | Dockerfiles for the RunPod container images used for our official templates. - runpod/containers

Fazil TOP•9mo ago

Perhaps something to do with start jupyter notebook button that default runpod templates have during gpu deployment screen. I don't have that option since I have a custom template.

Fazil TOP•9mo ago

This is the checkbox i mean

Unknown User•9mo ago

Message Not Public

Fazil TOP•9mo ago

I have the port link but it always says port is not ready. When i click it's HTTP 502 which is why i have to open the web terminal, run 1-2 commands and then it becomes ready Here is my dockerfile for reference:

ARG MINICONDA_VERSION=24.4.0-0

FROM continuumio/miniconda3:${MINICONDA_VERSION}

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

ENV DEBIAN_FRONTEND=noninteractive
ENV SHELL=/bin/bash

WORKDIR /

RUN mkdir /workspace

# git, wget is installed in the base image
RUN apt-get update --yes && \
    apt-get upgrade --yes && \
    apt install --yes --no-install-recommends curl libgl1 software-properties-common openssh-server nginx && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen

COPY requirements/ requirements/

RUN conda env create -n py -f requirements/dev/environment-gpu-full.yml && \
    conda clean -afy

# Activate conda env inside
SHELL ["conda", "run", "-n", "py", "/bin/bash", "-c"]

# This line is necessary to run the container interactively
RUN echo "conda activate py" >> ~/.bashrc

# Remove existing SSH host keys
RUN rm -f /etc/ssh/ssh_host_*

# NGINX Proxy
COPY scripts/runpod/docker/nginx.conf /etc/nginx/nginx.conf
COPY scripts/runpod/docker/readme.html /usr/share/nginx/html/readme.html

# Start Scripts
COPY scripts/runpod/docker/start.sh /
RUN chmod +x /start.sh

COPY . /workspace

# Set the default command for the container
CMD [ "/start.sh" ]

ARG MINICONDA_VERSION=24.4.0-0

FROM continuumio/miniconda3:${MINICONDA_VERSION}

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

ENV DEBIAN_FRONTEND=noninteractive
ENV SHELL=/bin/bash

WORKDIR /

RUN mkdir /workspace

# git, wget is installed in the base image
RUN apt-get update --yes && \
    apt-get upgrade --yes && \
    apt install --yes --no-install-recommends curl libgl1 software-properties-common openssh-server nginx && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen

COPY requirements/ requirements/

RUN conda env create -n py -f requirements/dev/environment-gpu-full.yml && \
    conda clean -afy

# Activate conda env inside
SHELL ["conda", "run", "-n", "py", "/bin/bash", "-c"]

# This line is necessary to run the container interactively
RUN echo "conda activate py" >> ~/.bashrc

# Remove existing SSH host keys
RUN rm -f /etc/ssh/ssh_host_*

# NGINX Proxy
COPY scripts/runpod/docker/nginx.conf /etc/nginx/nginx.conf
COPY scripts/runpod/docker/readme.html /usr/share/nginx/html/readme.html

# Start Scripts
COPY scripts/runpod/docker/start.sh /
RUN chmod +x /start.sh

COPY . /workspace

# Set the default command for the container
CMD [ "/start.sh" ]

start.sh, nginx.conf is copied from the runpod github repo.

Unknown User•9mo ago

Message Not Public

Fazil TOP•9mo ago

I have done this on runpod (overriding the start cmd) bash -c 'source ~/.bashrc && conda activate py && conda install -y jupyter && ./start.sh' but somehow i still needed webterminal launch jupyter. 🤷‍♂️

Unknown User•9mo ago

Message Not Public

zfmoodydub•8mo ago

@Fazil T did you end up figuring this out? im also having the same problem

zfmoodydub•8mo ago

@Fazil T @nerdylive Start Jupyter Notebook boolean is not even appearing for me on the deployment configuration

zfmoodydub•8mo ago

added this inside my dockerfile:

RUN echo '#!/bin/bash \n\
echo "Starting Jupyter Lab..." \n\
cd / && \n\
conda run -n automasher jupyter lab --allow-root \n\
    --no-browser \n\
    --port=8888 \n\
    --ip=* \n\
    --FileContentsManager.delete_to_trash=False \n\
    --ContentsManager.allow_hidden=True \n\
    --ServerApp.terminado_settings='"'"'{"shell_command":["/bin/bash"]}'"'"' \n\
    --ServerApp.token="" \n\
    --ServerApp.password="" \n\
    --ServerApp.allow_origin=* \n\
    --ServerApp.preferred_dir=/workspace &> /workspace/logs/jupyter.log & \n\
echo "Jupyter Lab started" \n\
conda activate automasher \n\
tail -f /dev/null' > /start_jupyter.sh && \
    chmod +x /start_jupyter.sh

ENTRYPOINT ["/start_jupyter.sh"]

i must be doing somethign wrong

Unknown User•8mo ago

Message Not Public

Fazil TOP•8mo ago

You have to launch the pod as is and port will be available after a while if your configuration is correct. The boolean box is somehow for rundpod template so it won't appear for custom dockers

zfmoodydub•8mo ago

what does just hardcode it mean? also how do i know if my configuration is correct?

Unknown User•8mo ago

Message Not Public

zfmoodydub•8mo ago

got it to work, thanks guys

Madiator2011 (Work)•8mo ago

code server > jupiter

Gaming

Programming

Unable to run jupyter on custom docker image

Did you find this page helpful?