R
Railway11mo ago
Pedroo

save temporary archives

Can i save temporary pdfs to a back-end host ? I'm trying to save a pdf file from a client upload but can't, have any form to do that ?
195 Replies
Percy
Percy11mo ago
Project ID: N/A
Pedroo
Pedroo11mo ago
it's a front-end calling a back-end api, and fron't gives a pdf, back-end need to saves to camelot read, and after that give back a archive to front
Medim
Medim11mo ago
Use railway volumes, saving directly into your service will cause it to be deleted at every redeploy/deploy
Pedroo
Pedroo11mo ago
hmmmm nice, i will see that
Medim
Medim11mo ago
If u don't need it to be persistent, theres no problem with that
Pedroo
Pedroo11mo ago
noo it's like a temp archive ever persons to call will give the same name, temp_pdf... and exclude after i will try volumes
Medim
Medim11mo ago
volumes are meant for persistent files, and they cost an additional 0,25USD/Gb/month
Brody
Brody11mo ago
slow down here
Medim
Medim11mo ago
brody knows better
Brody
Brody11mo ago
I'm trying to save a pdf file from a client upload but can't
you have yet to tell us why you "can't"
Pedroo
Pedroo11mo ago
in my locally server i just save in the src directory like "./temp-pdf.pdf' and debugging the server in rail, the app just don't save this can't have access to this pdf
Brody
Brody11mo ago
this is a ruby on rails app?
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
my local is like this i trying to give a buffer or something
Brody
Brody11mo ago
oh rail = railway
Pedroo
Pedroo11mo ago
noo
Brody
Brody11mo ago
this is python
Pedroo
Pedroo11mo ago
aaa brody is you you helped me like 30 mins ago hahahah with the opencv2
Brody
Brody11mo ago
indeed okay so lets slow down here do you see a print 4 in the deployment logs
Pedroo
Pedroo11mo ago
yeees i will send some screens a sec
Brody
Brody11mo ago
and then i assume there is no error printed from the catch?
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
the 6 i did'nt see
Pedroo
Pedroo11mo ago
Brody
Brody11mo ago
but you dont see error logs?
Pedroo
Pedroo11mo ago
hmmm you are calling me ?
Brody
Brody11mo ago
no?
Pedroo
Pedroo11mo ago
wtf.. my discord is souding but i did'nt see the call sorry a sec hm i not debbug, only prints
Brody
Brody11mo ago
i have not called you, or have ever called anyone in #help
Pedroo
Pedroo11mo ago
sorry
Brody
Brody11mo ago
show me the deployment logs please, use this https://bookmarklet.up.railway.app/log-downloader/
Pedroo
Pedroo11mo ago
okaay i will download i need to drag this
Brody
Brody11mo ago
yes
Brody
Brody11mo ago
Brody
Brody11mo ago
you dont have a bookmarks bar
Pedroo
Pedroo11mo ago
oh i see that now, i will se a video, really don't know what is this i come back in some minutos for study this oh nice i will send the logs [2023-08-25 17:27:41 +0000] [1] [INFO] Starting gunicorn 20.0.4 [2023-08-25 17:27:41 +0000] [1] [INFO] Listening at: http://0.0.0.0:5737 (1) [2023-08-25 17:27:41 +0000] [1] [INFO] Using worker: sync [2023-08-25 17:27:41 +0000] [10] [INFO] Booting worker with pid: 10 1 2 3 4 5
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
this came with the txtr logs
Brody
Brody11mo ago
you didnt use the bookmarklet but okay
Pedroo
Pedroo11mo ago
sorry for long time response i use see
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
is coming the same message of direct consol
Brody
Brody11mo ago
cool but then you where just supposed to send that file here
Pedroo
Pedroo11mo ago
ok
Brody
Brody11mo ago
it doesnt matter now
Brody
Brody11mo ago
show me a screenshot of your service metrics
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
Brody
Brody11mo ago
i assume this code does work locally?
Pedroo
Pedroo11mo ago
yes, work it is an application that takes a pdf from an educational institution (transfers it from one to another) and treats it, the flow is basically this, it sends the pdf through an api, the back-end saves this pdf, treats it, and sends it to return an excel
Brody
Brody11mo ago
try adding this as a nixpacks.toml file in your project
[phases.setup]
aptPkgs = ['...', 'ghostscript', 'python3-tk']
[phases.setup]
aptPkgs = ['...', 'ghostscript', 'python3-tk']
Pedroo
Pedroo11mo ago
okay !
Pedroo
Pedroo11mo ago
nice, added stay in the 5 print
Brody
Brody11mo ago
send me the build logs please
Pedroo
Pedroo11mo ago
okay
Brody
Brody11mo ago
can you implement the Ghostscript and Tkinter checks this docs page shows you? (for ubuntu) https://camelot-py.readthedocs.io/en/master/user/install-deps.html#checks-to-see-if-dependencies-are-installed-correctly
Pedroo
Pedroo11mo ago
YES will do that a sec hmm... not correctly
Brody
Brody11mo ago
explain
Pedroo
Pedroo11mo ago
Giving back None
Brody
Brody11mo ago
show me the code you implemented
Pedroo
Pedroo11mo ago
Okay i give this in out of app and in
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
but my local return None too
Brody
Brody11mo ago
yeah because locally you are using windows?
Pedroo
Pedroo11mo ago
'-' will test with the windows sorry. D:\gs10.01.2\bin\gsdll64.dll yes, local is ok
Brody
Brody11mo ago
okay but that's irevlent, we already knew it worked locally
Pedroo
Pedroo11mo ago
will try to redeploy and see if it's work will print the current directory files too to see if it's saving the archive
Brody
Brody11mo ago
what version of python do you use locally
Pedroo
Pedroo11mo ago
yes.. it's saving
Pedroo
Pedroo11mo ago
locally i use... 3.11.4
Brody
Brody11mo ago
why do you have a phases.setup file
Pedroo
Pedroo11mo ago
when I was trying to solve it by searching the internet, I had created... it was an error i will delete
Brody
Brody11mo ago
try this instead
[variables]
NIXPACKS_PYTHON_VERSION = '3.11'

[phases.setup]
nixPkgs = ['...', 'ghostscript', 'python311Packages.tkinter']
aptPkgs = ['...', 'ghostscript', 'python3-tk']
[variables]
NIXPACKS_PYTHON_VERSION = '3.11'

[phases.setup]
nixPkgs = ['...', 'ghostscript', 'python311Packages.tkinter']
aptPkgs = ['...', 'ghostscript', 'python3-tk']
Pedroo
Pedroo11mo ago
okay have the "-" ? sending the log
Brody
Brody11mo ago
i updated it
Pedroo
Pedroo11mo ago
building
Pedroo
Pedroo11mo ago
will change version of pdftopng to 0.1.0 None 😦 nixPkgs = ['...', 'ghostscript', 'python311Packages.tkinter'] aptPkgs = ['...', 'ghostscript', 'python3-tk'] for use the version older like 39.13 3.9..13 in the python311packages can change it ?
Brody
Brody11mo ago
show me your current nixpacks.toml file
Pedroo
Pedroo11mo ago
Brody
Brody11mo ago
show me a screenshot of your service variables please
Pedroo
Pedroo11mo ago
Brody
Brody11mo ago
that is not what I asked for
Pedroo
Pedroo11mo ago
sorry.
Brody
Brody11mo ago
I asked for your railway service variables
Pedroo
Pedroo11mo ago
my english not too good okay i will print a sec
Pedroo
Pedroo11mo ago
Brody
Brody11mo ago
do you happen to have a runtime.txt file?
Pedroo
Pedroo11mo ago
didn't have
Brody
Brody11mo ago
show me a screenshot of your backends project files
Pedroo
Pedroo11mo ago
okay
Pedroo
Pedroo11mo ago
Brody
Brody11mo ago
can you make sure you have that temp.pdf file in your gitignore and that you go and remove that pdf file from your github repo
Pedroo
Pedroo11mo ago
okay done
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
changed this too lets see
Brody
Brody11mo ago
why do you not use 3.11 locally?
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
i use 3.11, i see venv and venv is in the 3.9.13
Brody
Brody11mo ago
why would your venv be a different version
Pedroo
Pedroo11mo ago
i really don't know... i just start the project with the pycharm, and will se the venv now it is in the 3.9 😢 okay now servers up
Pedroo
Pedroo11mo ago
the gs don't find
Brody
Brody11mo ago
I think it may be time to move to a Dockerfile
Pedroo
Pedroo11mo ago
Oh nice My tech lead saw the same thing :c i will study it Buuuut, for now, thank u so much, u are very smart men I will do a docker for this
Brody
Brody11mo ago
dockerfile* I can write one for you, I'm just eating right now
Pedroo
Pedroo11mo ago
i really don't know nothing from infra / docker
Brody
Brody11mo ago
^
Pedroo
Pedroo11mo ago
oh nice i'm doing one here, viewing what gonna result hahaha can i add u on disc ?
Brody
Brody11mo ago
no thank you
Pedroo
Pedroo11mo ago
nice hahahah i will do the docker thanks for the help men !!
Brody
Brody11mo ago
dockerfile* but once done, show me it please
Pedroo
Pedroo11mo ago
okay i be back
version: '3.4'
services:
anima:
image: node:14 # Use a versão adequada do Node.js
container_name: anima
ports:
- "3000:3000" # Mapeie a porta do container para a porta do host, se necessário
volumes:
- ./anima:/app # Mapeie o diretório do código-fonte do frontend para o container
working_dir: /app
command: sh -c "npm install && npm start" # Comando para iniciar o frontend

mvp-achievement-rates:
image: python:3.9 # Use a versão adequada do Python
container_name: mvp-achievement-rates
ports:
- "8000:8000" # Mapeie a porta do container para a porta do host, se necessário
volumes:
- ./mvp-achievement-rates:/app # Mapeie o diretório do código-fonte do backend para o container
working_dir: /app
command: sh -c "apt-get update && apt-get install -y ghostscript && pip install -r requirements.txt && python main.py" # Comando para iniciar o backend

database:
image: postgres:13 # Use a versão adequada do PostgreSQL
container_name: database
ports:
- "5432:5432" # Mapeie a porta do container para a porta do host, se necessário
environment:
POSTGRES_DB: Databases
POSTGRES_USER: postgres
POSTGRES_PASSWORD: 1789
volumes:
- postgres_data:/var/lib/postgresql/data # Volume para persistência dos dados do banco

volumes:
postgres_data: # Defina o nome do volume para persistência dos dados do PostgreSQL
version: '3.4'
services:
anima:
image: node:14 # Use a versão adequada do Node.js
container_name: anima
ports:
- "3000:3000" # Mapeie a porta do container para a porta do host, se necessário
volumes:
- ./anima:/app # Mapeie o diretório do código-fonte do frontend para o container
working_dir: /app
command: sh -c "npm install && npm start" # Comando para iniciar o frontend

mvp-achievement-rates:
image: python:3.9 # Use a versão adequada do Python
container_name: mvp-achievement-rates
ports:
- "8000:8000" # Mapeie a porta do container para a porta do host, se necessário
volumes:
- ./mvp-achievement-rates:/app # Mapeie o diretório do código-fonte do backend para o container
working_dir: /app
command: sh -c "apt-get update && apt-get install -y ghostscript && pip install -r requirements.txt && python main.py" # Comando para iniciar o backend

database:
image: postgres:13 # Use a versão adequada do PostgreSQL
container_name: database
ports:
- "5432:5432" # Mapeie a porta do container para a porta do host, se necessário
environment:
POSTGRES_DB: Databases
POSTGRES_USER: postgres
POSTGRES_PASSWORD: 1789
volumes:
- postgres_data:/var/lib/postgresql/data # Volume para persistência dos dados do banco

volumes:
postgres_data: # Defina o nome do volume para persistência dos dados do PostgreSQL
Brody
Brody11mo ago
that is not a dockerfile thats a docker-compose
Pedroo
Pedroo11mo ago
i'm doing a compose
Brody
Brody11mo ago
railway does not support docker-compose
Pedroo
Pedroo11mo ago
hmmmmm so i will do a dockerfile
Brody
Brody11mo ago
yeah i kept saying dockerfile everytime you said docker
Pedroo
Pedroo11mo ago
like this
Pedroo
Pedroo11mo ago
i will do in the front / back and i do no, tomorrow show to u
Brody
Brody11mo ago
that is not an ideal dockerfile but it is a dockerfile, ill give you that much
Pedroo
Pedroo11mo ago
nice, tomorrow send to u have a good night, cya
Brody
Brody11mo ago
you too!
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
running on railway now, but i can't acess the api
Pedroo
Pedroo11mo ago
Brody
Brody11mo ago
send me the dockerfile, I'll fix it for you
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
FROM python:3.9

RUN apt-get update && \
apt install libgl1-mesa-glx -y && \
apt-get install 'ffmpeg'\
'libsm6'\
'libxext6' -y && \
apt-get install -y ghostscript && \
apt-get clean

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

CMD ["python3", "main.py"]
FROM python:3.9

RUN apt-get update && \
apt install libgl1-mesa-glx -y && \
apt-get install 'ffmpeg'\
'libsm6'\
'libxext6' -y && \
apt-get install -y ghostscript && \
apt-get clean

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

CMD ["python3", "main.py"]
i try to do with the debian:bookworm but some error ocurred
Brody
Brody11mo ago
yeah i know
Pedroo
Pedroo11mo ago
and i do that with the base py:3.9
Brody
Brody11mo ago
ill fix it
Pedroo
Pedroo11mo ago
thank you
Brody
Brody11mo ago
send me your requirements.txt file too please
Pedroo
Pedroo11mo ago
aiohttp==3.8.5
aiosignal==1.3.1
async-timeout==4.0.2
attrs==23.1.0
blinker==1.6.2
camelot-py==0.11.0
certifi==2023.7.22
cffi==1.15.1
chardet==5.1.0
charset-normalizer==3.2.0
click==8.1.6
colorama==0.4.6
cryptography==41.0.2
distro==1.8.0
et-xmlfile==1.1.0
flask==2.3.3
Flask-Cors==4.0.0
frozenlist==1.4.0
ghostscript==0.7
greenlet==2.0.2
idna==3.4
importlib-metadata==6.8.0
itsdangerous==2.1.2
Jinja2==3.1.2
lxml==4.9.3
MarkupSafe==2.1.3
multidict==6.0.4
numpy==1.24.4
openai==0.27.8
opencv-python==4.8.0.74
openpyxl==3.1.2
pandas==2.0.3
pdfminer.six==20221105
pdftopng==0.2.3
Pillow==10.0.0
psycopg2==2.9.7
pycparser==2.21
pypdf==3.14.0
pypdf2==3.0.1
python-dateutil==2.8.2
python-docx==0.8.11
pytz==2023.3
requests==2.31.0
six==1.16.0
SQLAlchemy==2.0.20
tabula-py==2.7.0
tabulate==0.9.0
tqdm==4.65.0
typing-extensions==4.7.1
tzdata==2023.3
Unidecode==1.3.6
urllib3==2.0.4
werkzeug==2.3.7
yarl==1.9.2
zipp==3.16.2
aiohttp==3.8.5
aiosignal==1.3.1
async-timeout==4.0.2
attrs==23.1.0
blinker==1.6.2
camelot-py==0.11.0
certifi==2023.7.22
cffi==1.15.1
chardet==5.1.0
charset-normalizer==3.2.0
click==8.1.6
colorama==0.4.6
cryptography==41.0.2
distro==1.8.0
et-xmlfile==1.1.0
flask==2.3.3
Flask-Cors==4.0.0
frozenlist==1.4.0
ghostscript==0.7
greenlet==2.0.2
idna==3.4
importlib-metadata==6.8.0
itsdangerous==2.1.2
Jinja2==3.1.2
lxml==4.9.3
MarkupSafe==2.1.3
multidict==6.0.4
numpy==1.24.4
openai==0.27.8
opencv-python==4.8.0.74
openpyxl==3.1.2
pandas==2.0.3
pdfminer.six==20221105
pdftopng==0.2.3
Pillow==10.0.0
psycopg2==2.9.7
pycparser==2.21
pypdf==3.14.0
pypdf2==3.0.1
python-dateutil==2.8.2
python-docx==0.8.11
pytz==2023.3
requests==2.31.0
six==1.16.0
SQLAlchemy==2.0.20
tabula-py==2.7.0
tabulate==0.9.0
tqdm==4.65.0
typing-extensions==4.7.1
tzdata==2023.3
Unidecode==1.3.6
urllib3==2.0.4
werkzeug==2.3.7
yarl==1.9.2
zipp==3.16.2
Brody
Brody11mo ago
are you using all those?
Pedroo
Pedroo11mo ago
yees
Brody
Brody11mo ago
what variable did you give to your flask app? app or main?
Brody
Brody11mo ago
like this
Brody
Brody11mo ago
add gunicorn=21.2.0 to that list, but dont push changes yet
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
the same of yours olay okay
Brody
Brody11mo ago
give this dockerfile a shot
FROM python:3.9

ENV PYTHONUNBUFFERED="1"
ENV PIP_DISABLE_PIP_VERSION_CHECK="1"

RUN apt-get update && apt-get install -y --no-install-recommends \
libgl1-mesa-glx \
ffmpeg \
libsm6 \
libxext6 \
ghostscript \
&& apt-get clean

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

CMD gunicorn main:app
FROM python:3.9

ENV PYTHONUNBUFFERED="1"
ENV PIP_DISABLE_PIP_VERSION_CHECK="1"

RUN apt-get update && apt-get install -y --no-install-recommends \
libgl1-mesa-glx \
ffmpeg \
libsm6 \
libxext6 \
ghostscript \
&& apt-get clean

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

CMD gunicorn main:app
and once you changed your dockerfile for that one, push your changes
Pedroo
Pedroo11mo ago
okay
Brody
Brody11mo ago
my bad i edited the dockerfile message
Pedroo
Pedroo11mo ago
WOOOOOOOOOOOOOOOOOW this workssssss
Brody
Brody11mo ago
awesome
Pedroo
Pedroo11mo ago
thank you so much !
Brody
Brody11mo ago
no problem!
Pedroo
Pedroo11mo ago
will use docker file ever now, hahahahah, have a good night !! 😄
Brody
Brody11mo ago
you too!
Pedroo
Pedroo11mo ago
the finally thing, the guicorn have how to configure the works ?
Brody
Brody11mo ago
show me a screenshot of your metrics please
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
Brody
Brody11mo ago
Why'd you send that twice
Pedroo
Pedroo11mo ago
i do another request for view the actual
Brody
Brody11mo ago
nothing really changed
Pedroo
Pedroo11mo ago
k, sorry
Brody
Brody11mo ago
how long should an http request take locally?
Pedroo
Pedroo11mo ago
here an GPT response like 1~~2sec
Brody
Brody11mo ago
something in your code took longer than 30 seconds, that is the default timeout from gunicorn should a request ever take longer than 30 seconds?
Pedroo
Pedroo11mo ago
no, i have a treatment in responses that take more than 10sec, retry
Pedroo
Pedroo11mo ago
Pedroo
Pedroo11mo ago
i will try do to a time.sleep
Brody
Brody11mo ago
why would you want to make the function take longer??
Pedroo
Pedroo11mo ago
could it be giving an error, because it makes many requests quickly?
Brody
Brody11mo ago
should a request ever take longer than 30 seconds?
Pedroo
Pedroo11mo ago
because every request he makes takes around 1 second, and he makes maybe more than 1000 per execution no, no
Brody
Brody11mo ago
then there is a bug in your code
Pedroo
Pedroo11mo ago
from the user, requested the api yes this take like 4~~5 minutos
Brody
Brody11mo ago
30 seconds is the default timeout from gunicorn
Pedroo
Pedroo11mo ago
aaaaaaaah okay for the user
Brody
Brody11mo ago
i just asked you and you said no 😐
Pedroo
Pedroo11mo ago
i think you say in the application like, in the back-end i do only request like 12 sec, but muchs requests like 1000 1500 and return to the user the answer but it's take 5 minutes, i really don't undertand the starter question, sorry.
Brody
Brody11mo ago
running locally what is the total request time the client will see
Pedroo
Pedroo11mo ago
like 5 minutes
Brody
Brody11mo ago
is that normal?
Pedroo
Pedroo11mo ago
yees
Brody
Brody11mo ago
5 minutes for one single network request?
Pedroo
Pedroo11mo ago
yes, it's an AI project I'm working on, which treats school transcripts based on pdfs, and returns the answer of where it should go when changing course, or institution. There are many requests, treated with chatGPT in search of similarities in the school grade
Brody
Brody11mo ago
one single network request that the client does takes 5 minutes? are you sure? you have been unsure in the past
Pedroo
Pedroo11mo ago
yes, take, when i call the api the response for client, take like 5 minutes
Brody
Brody11mo ago
alright then add --timeout 360 to the end of the CMD command in the dockerfile, this sets the request max timeout to 6 minutes
Pedroo
Pedroo11mo ago
okay now, this work fine thank u agaaaain and all solved 😄
Brody
Brody11mo ago
for sure this time lol