R
Railway15mo ago
Pada

save temporary archives

Can i save temporary pdfs to a back-end host ? I'm trying to save a pdf file from a client upload but can't, have any form to do that ?
195 Replies
Percy
Percy15mo ago
Project ID: N/A
Pada
Pada15mo ago
it's a front-end calling a back-end api, and fron't gives a pdf, back-end need to saves to camelot read, and after that give back a archive to front
Medim
Medim15mo ago
Use railway volumes, saving directly into your service will cause it to be deleted at every redeploy/deploy
Pada
Pada15mo ago
hmmmm nice, i will see that
Medim
Medim15mo ago
If u don't need it to be persistent, theres no problem with that
Pada
Pada15mo ago
noo it's like a temp archive ever persons to call will give the same name, temp_pdf... and exclude after i will try volumes
Medim
Medim15mo ago
volumes are meant for persistent files, and they cost an additional 0,25USD/Gb/month
Brody
Brody15mo ago
slow down here
Medim
Medim15mo ago
brody knows better
Brody
Brody15mo ago
I'm trying to save a pdf file from a client upload but can't
you have yet to tell us why you "can't"
Pada
Pada15mo ago
in my locally server i just save in the src directory like "./temp-pdf.pdf' and debugging the server in rail, the app just don't save this can't have access to this pdf
Brody
Brody15mo ago
this is a ruby on rails app?
Pada
Pada15mo ago
Pada
Pada15mo ago
my local is like this i trying to give a buffer or something
Brody
Brody15mo ago
oh rail = railway
Pada
Pada15mo ago
noo
Brody
Brody15mo ago
this is python
Pada
Pada15mo ago
aaa brody is you you helped me like 30 mins ago hahahah with the opencv2
Brody
Brody15mo ago
indeed okay so lets slow down here do you see a print 4 in the deployment logs
Pada
Pada15mo ago
yeees i will send some screens a sec
Brody
Brody15mo ago
and then i assume there is no error printed from the catch?
Pada
Pada15mo ago
Pada
Pada15mo ago
the 6 i did'nt see
Pada
Pada15mo ago
Brody
Brody15mo ago
but you dont see error logs?
Pada
Pada15mo ago
hmmm you are calling me ?
Brody
Brody15mo ago
no?
Pada
Pada15mo ago
wtf.. my discord is souding but i did'nt see the call sorry a sec hm i not debbug, only prints
Brody
Brody15mo ago
i have not called you, or have ever called anyone in #help
Pada
Pada15mo ago
sorry
Brody
Brody15mo ago
show me the deployment logs please, use this https://bookmarklet.up.railway.app/log-downloader/
Pada
Pada15mo ago
okaay i will download i need to drag this
Brody
Brody15mo ago
yes
Brody
Brody15mo ago
Brody
Brody15mo ago
you dont have a bookmarks bar
Pada
Pada15mo ago
oh i see that now, i will se a video, really don't know what is this i come back in some minutos for study this oh nice i will send the logs [2023-08-25 17:27:41 +0000] [1] [INFO] Starting gunicorn 20.0.4 [2023-08-25 17:27:41 +0000] [1] [INFO] Listening at: http://0.0.0.0:5737 (1) [2023-08-25 17:27:41 +0000] [1] [INFO] Using worker: sync [2023-08-25 17:27:41 +0000] [10] [INFO] Booting worker with pid: 10 1 2 3 4 5
Pada
Pada15mo ago
Pada
Pada15mo ago
this came with the txtr logs
Brody
Brody15mo ago
you didnt use the bookmarklet but okay
Pada
Pada15mo ago
sorry for long time response i use see
Pada
Pada15mo ago
Pada
Pada15mo ago
Pada
Pada15mo ago
is coming the same message of direct consol
Brody
Brody15mo ago
cool but then you where just supposed to send that file here
Pada
Pada15mo ago
ok
Brody
Brody15mo ago
it doesnt matter now
Brody
Brody15mo ago
show me a screenshot of your service metrics
Pada
Pada15mo ago
Pada
Pada15mo ago
Pada
Pada15mo ago
Brody
Brody15mo ago
i assume this code does work locally?
Pada
Pada15mo ago
yes, work it is an application that takes a pdf from an educational institution (transfers it from one to another) and treats it, the flow is basically this, it sends the pdf through an api, the back-end saves this pdf, treats it, and sends it to return an excel
Brody
Brody15mo ago
try adding this as a nixpacks.toml file in your project
[phases.setup]
aptPkgs = ['...', 'ghostscript', 'python3-tk']
[phases.setup]
aptPkgs = ['...', 'ghostscript', 'python3-tk']
Pada
Pada15mo ago
okay !
Pada
Pada15mo ago
nice, added stay in the 5 print
Brody
Brody15mo ago
send me the build logs please
Pada
Pada15mo ago
okay
Brody
Brody15mo ago
can you implement the Ghostscript and Tkinter checks this docs page shows you? (for ubuntu) https://camelot-py.readthedocs.io/en/master/user/install-deps.html#checks-to-see-if-dependencies-are-installed-correctly
Pada
Pada15mo ago
YES will do that a sec hmm... not correctly
Brody
Brody15mo ago
explain
Pada
Pada15mo ago
Giving back None
Brody
Brody15mo ago
show me the code you implemented
Pada
Pada15mo ago
Okay i give this in out of app and in
Pada
Pada15mo ago
Pada
Pada15mo ago
but my local return None too
Brody
Brody15mo ago
yeah because locally you are using windows?
Pada
Pada15mo ago
'-' will test with the windows sorry. D:\gs10.01.2\bin\gsdll64.dll yes, local is ok
Brody
Brody15mo ago
okay but that's irevlent, we already knew it worked locally
Pada
Pada15mo ago
will try to redeploy and see if it's work will print the current directory files too to see if it's saving the archive
Brody
Brody15mo ago
what version of python do you use locally
Pada
Pada15mo ago
yes.. it's saving
Pada
Pada15mo ago
locally i use... 3.11.4
Brody
Brody15mo ago
why do you have a phases.setup file
Pada
Pada15mo ago
when I was trying to solve it by searching the internet, I had created... it was an error i will delete
Brody
Brody15mo ago
try this instead
[variables]
NIXPACKS_PYTHON_VERSION = '3.11'

[phases.setup]
nixPkgs = ['...', 'ghostscript', 'python311Packages.tkinter']
aptPkgs = ['...', 'ghostscript', 'python3-tk']
[variables]
NIXPACKS_PYTHON_VERSION = '3.11'

[phases.setup]
nixPkgs = ['...', 'ghostscript', 'python311Packages.tkinter']
aptPkgs = ['...', 'ghostscript', 'python3-tk']
Pada
Pada15mo ago
okay have the "-" ? sending the log
Brody
Brody15mo ago
i updated it
Pada
Pada15mo ago
building
Pada
Pada15mo ago
will change version of pdftopng to 0.1.0 None 😦 nixPkgs = ['...', 'ghostscript', 'python311Packages.tkinter'] aptPkgs = ['...', 'ghostscript', 'python3-tk'] for use the version older like 39.13 3.9..13 in the python311packages can change it ?
Brody
Brody15mo ago
show me your current nixpacks.toml file
Pada
Pada15mo ago
Brody
Brody15mo ago
show me a screenshot of your service variables please
Pada
Pada15mo ago
Brody
Brody15mo ago
that is not what I asked for
Pada
Pada15mo ago
sorry.
Brody
Brody15mo ago
I asked for your railway service variables
Pada
Pada15mo ago
my english not too good okay i will print a sec
Pada
Pada15mo ago
Brody
Brody15mo ago
do you happen to have a runtime.txt file?
Pada
Pada15mo ago
didn't have
Brody
Brody15mo ago
show me a screenshot of your backends project files
Pada
Pada15mo ago
okay
Pada
Pada15mo ago
Brody
Brody15mo ago
can you make sure you have that temp.pdf file in your gitignore and that you go and remove that pdf file from your github repo
Pada
Pada15mo ago
okay done
Pada
Pada15mo ago
Pada
Pada15mo ago
changed this too lets see
Brody
Brody15mo ago
why do you not use 3.11 locally?
Pada
Pada15mo ago
Pada
Pada15mo ago
i use 3.11, i see venv and venv is in the 3.9.13
Brody
Brody15mo ago
why would your venv be a different version
Pada
Pada15mo ago
i really don't know... i just start the project with the pycharm, and will se the venv now it is in the 3.9 😢 okay now servers up
Pada
Pada15mo ago
the gs don't find
Brody
Brody15mo ago
I think it may be time to move to a Dockerfile
Pada
Pada15mo ago
Oh nice My tech lead saw the same thing :c i will study it Buuuut, for now, thank u so much, u are very smart men I will do a docker for this
Brody
Brody15mo ago
dockerfile* I can write one for you, I'm just eating right now
Pada
Pada15mo ago
i really don't know nothing from infra / docker
Brody
Brody15mo ago
^
Pada
Pada15mo ago
oh nice i'm doing one here, viewing what gonna result hahaha can i add u on disc ?
Brody
Brody15mo ago
no thank you
Pada
Pada15mo ago
nice hahahah i will do the docker thanks for the help men !!
Brody
Brody15mo ago
dockerfile* but once done, show me it please
Pada
Pada15mo ago
okay i be back
version: '3.4'
services:
anima:
image: node:14 # Use a versão adequada do Node.js
container_name: anima
ports:
- "3000:3000" # Mapeie a porta do container para a porta do host, se necessário
volumes:
- ./anima:/app # Mapeie o diretório do código-fonte do frontend para o container
working_dir: /app
command: sh -c "npm install && npm start" # Comando para iniciar o frontend

mvp-achievement-rates:
image: python:3.9 # Use a versão adequada do Python
container_name: mvp-achievement-rates
ports:
- "8000:8000" # Mapeie a porta do container para a porta do host, se necessário
volumes:
- ./mvp-achievement-rates:/app # Mapeie o diretório do código-fonte do backend para o container
working_dir: /app
command: sh -c "apt-get update && apt-get install -y ghostscript && pip install -r requirements.txt && python main.py" # Comando para iniciar o backend

database:
image: postgres:13 # Use a versão adequada do PostgreSQL
container_name: database
ports:
- "5432:5432" # Mapeie a porta do container para a porta do host, se necessário
environment:
POSTGRES_DB: Databases
POSTGRES_USER: postgres
POSTGRES_PASSWORD: 1789
volumes:
- postgres_data:/var/lib/postgresql/data # Volume para persistência dos dados do banco

volumes:
postgres_data: # Defina o nome do volume para persistência dos dados do PostgreSQL
version: '3.4'
services:
anima:
image: node:14 # Use a versão adequada do Node.js
container_name: anima
ports:
- "3000:3000" # Mapeie a porta do container para a porta do host, se necessário
volumes:
- ./anima:/app # Mapeie o diretório do código-fonte do frontend para o container
working_dir: /app
command: sh -c "npm install && npm start" # Comando para iniciar o frontend

mvp-achievement-rates:
image: python:3.9 # Use a versão adequada do Python
container_name: mvp-achievement-rates
ports:
- "8000:8000" # Mapeie a porta do container para a porta do host, se necessário
volumes:
- ./mvp-achievement-rates:/app # Mapeie o diretório do código-fonte do backend para o container
working_dir: /app
command: sh -c "apt-get update && apt-get install -y ghostscript && pip install -r requirements.txt && python main.py" # Comando para iniciar o backend

database:
image: postgres:13 # Use a versão adequada do PostgreSQL
container_name: database
ports:
- "5432:5432" # Mapeie a porta do container para a porta do host, se necessário
environment:
POSTGRES_DB: Databases
POSTGRES_USER: postgres
POSTGRES_PASSWORD: 1789
volumes:
- postgres_data:/var/lib/postgresql/data # Volume para persistência dos dados do banco

volumes:
postgres_data: # Defina o nome do volume para persistência dos dados do PostgreSQL
Brody
Brody15mo ago
that is not a dockerfile thats a docker-compose
Pada
Pada15mo ago
i'm doing a compose
Brody
Brody15mo ago
railway does not support docker-compose
Pada
Pada15mo ago
hmmmmm so i will do a dockerfile
Brody
Brody15mo ago
yeah i kept saying dockerfile everytime you said docker
Pada
Pada15mo ago
like this
Pada
Pada15mo ago
i will do in the front / back and i do no, tomorrow show to u
Brody
Brody15mo ago
that is not an ideal dockerfile but it is a dockerfile, ill give you that much
Pada
Pada15mo ago
nice, tomorrow send to u have a good night, cya
Brody
Brody15mo ago
you too!
Pada
Pada15mo ago
Pada
Pada15mo ago
running on railway now, but i can't acess the api
Pada
Pada15mo ago
Brody
Brody15mo ago
send me the dockerfile, I'll fix it for you
Pada
Pada15mo ago
Pada
Pada15mo ago
FROM python:3.9

RUN apt-get update && \
apt install libgl1-mesa-glx -y && \
apt-get install 'ffmpeg'\
'libsm6'\
'libxext6' -y && \
apt-get install -y ghostscript && \
apt-get clean

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

CMD ["python3", "main.py"]
FROM python:3.9

RUN apt-get update && \
apt install libgl1-mesa-glx -y && \
apt-get install 'ffmpeg'\
'libsm6'\
'libxext6' -y && \
apt-get install -y ghostscript && \
apt-get clean

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

CMD ["python3", "main.py"]
i try to do with the debian:bookworm but some error ocurred
Brody
Brody15mo ago
yeah i know
Pada
Pada15mo ago
and i do that with the base py:3.9
Brody
Brody15mo ago
ill fix it
Pada
Pada15mo ago
thank you
Brody
Brody15mo ago
send me your requirements.txt file too please
Pada
Pada15mo ago
aiohttp==3.8.5
aiosignal==1.3.1
async-timeout==4.0.2
attrs==23.1.0
blinker==1.6.2
camelot-py==0.11.0
certifi==2023.7.22
cffi==1.15.1
chardet==5.1.0
charset-normalizer==3.2.0
click==8.1.6
colorama==0.4.6
cryptography==41.0.2
distro==1.8.0
et-xmlfile==1.1.0
flask==2.3.3
Flask-Cors==4.0.0
frozenlist==1.4.0
ghostscript==0.7
greenlet==2.0.2
idna==3.4
importlib-metadata==6.8.0
itsdangerous==2.1.2
Jinja2==3.1.2
lxml==4.9.3
MarkupSafe==2.1.3
multidict==6.0.4
numpy==1.24.4
openai==0.27.8
opencv-python==4.8.0.74
openpyxl==3.1.2
pandas==2.0.3
pdfminer.six==20221105
pdftopng==0.2.3
Pillow==10.0.0
psycopg2==2.9.7
pycparser==2.21
pypdf==3.14.0
pypdf2==3.0.1
python-dateutil==2.8.2
python-docx==0.8.11
pytz==2023.3
requests==2.31.0
six==1.16.0
SQLAlchemy==2.0.20
tabula-py==2.7.0
tabulate==0.9.0
tqdm==4.65.0
typing-extensions==4.7.1
tzdata==2023.3
Unidecode==1.3.6
urllib3==2.0.4
werkzeug==2.3.7
yarl==1.9.2
zipp==3.16.2
aiohttp==3.8.5
aiosignal==1.3.1
async-timeout==4.0.2
attrs==23.1.0
blinker==1.6.2
camelot-py==0.11.0
certifi==2023.7.22
cffi==1.15.1
chardet==5.1.0
charset-normalizer==3.2.0
click==8.1.6
colorama==0.4.6
cryptography==41.0.2
distro==1.8.0
et-xmlfile==1.1.0
flask==2.3.3
Flask-Cors==4.0.0
frozenlist==1.4.0
ghostscript==0.7
greenlet==2.0.2
idna==3.4
importlib-metadata==6.8.0
itsdangerous==2.1.2
Jinja2==3.1.2
lxml==4.9.3
MarkupSafe==2.1.3
multidict==6.0.4
numpy==1.24.4
openai==0.27.8
opencv-python==4.8.0.74
openpyxl==3.1.2
pandas==2.0.3
pdfminer.six==20221105
pdftopng==0.2.3
Pillow==10.0.0
psycopg2==2.9.7
pycparser==2.21
pypdf==3.14.0
pypdf2==3.0.1
python-dateutil==2.8.2
python-docx==0.8.11
pytz==2023.3
requests==2.31.0
six==1.16.0
SQLAlchemy==2.0.20
tabula-py==2.7.0
tabulate==0.9.0
tqdm==4.65.0
typing-extensions==4.7.1
tzdata==2023.3
Unidecode==1.3.6
urllib3==2.0.4
werkzeug==2.3.7
yarl==1.9.2
zipp==3.16.2
Brody
Brody15mo ago
are you using all those?
Pada
Pada15mo ago
yees
Brody
Brody15mo ago
what variable did you give to your flask app? app or main?
Brody
Brody15mo ago
like this
Brody
Brody15mo ago
add gunicorn=21.2.0 to that list, but dont push changes yet
Pada
Pada15mo ago
Pada
Pada15mo ago
the same of yours olay okay
Brody
Brody15mo ago
give this dockerfile a shot
FROM python:3.9

ENV PYTHONUNBUFFERED="1"
ENV PIP_DISABLE_PIP_VERSION_CHECK="1"

RUN apt-get update && apt-get install -y --no-install-recommends \
libgl1-mesa-glx \
ffmpeg \
libsm6 \
libxext6 \
ghostscript \
&& apt-get clean

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

CMD gunicorn main:app
FROM python:3.9

ENV PYTHONUNBUFFERED="1"
ENV PIP_DISABLE_PIP_VERSION_CHECK="1"

RUN apt-get update && apt-get install -y --no-install-recommends \
libgl1-mesa-glx \
ffmpeg \
libsm6 \
libxext6 \
ghostscript \
&& apt-get clean

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

CMD gunicorn main:app
and once you changed your dockerfile for that one, push your changes
Pada
Pada15mo ago
okay
Brody
Brody15mo ago
my bad i edited the dockerfile message
Pada
Pada15mo ago
WOOOOOOOOOOOOOOOOOW this workssssss
Brody
Brody15mo ago
awesome
Pada
Pada15mo ago
thank you so much !
Brody
Brody15mo ago
no problem!
Pada
Pada15mo ago
will use docker file ever now, hahahahah, have a good night !! 😄
Brody
Brody15mo ago
you too!
Pada
Pada15mo ago
the finally thing, the guicorn have how to configure the works ?
Brody
Brody15mo ago
show me a screenshot of your metrics please
Pada
Pada15mo ago
Pada
Pada15mo ago
Brody
Brody15mo ago
Why'd you send that twice
Pada
Pada15mo ago
i do another request for view the actual
Brody
Brody15mo ago
nothing really changed
Pada
Pada15mo ago
k, sorry
Brody
Brody15mo ago
how long should an http request take locally?
Pada
Pada15mo ago
here an GPT response like 1~~2sec
Brody
Brody15mo ago
something in your code took longer than 30 seconds, that is the default timeout from gunicorn should a request ever take longer than 30 seconds?
Pada
Pada15mo ago
no, i have a treatment in responses that take more than 10sec, retry
Pada
Pada15mo ago
Pada
Pada15mo ago
i will try do to a time.sleep
Brody
Brody15mo ago
why would you want to make the function take longer??
Pada
Pada15mo ago
could it be giving an error, because it makes many requests quickly?
Brody
Brody15mo ago
should a request ever take longer than 30 seconds?
Pada
Pada15mo ago
because every request he makes takes around 1 second, and he makes maybe more than 1000 per execution no, no
Brody
Brody15mo ago
then there is a bug in your code
Pada
Pada15mo ago
from the user, requested the api yes this take like 4~~5 minutos
Brody
Brody15mo ago
30 seconds is the default timeout from gunicorn
Pada
Pada15mo ago
aaaaaaaah okay for the user
Brody
Brody15mo ago
i just asked you and you said no 😐
Pada
Pada15mo ago
i think you say in the application like, in the back-end i do only request like 12 sec, but muchs requests like 1000 1500 and return to the user the answer but it's take 5 minutes, i really don't undertand the starter question, sorry.
Brody
Brody15mo ago
running locally what is the total request time the client will see
Pada
Pada15mo ago
like 5 minutes
Brody
Brody15mo ago
is that normal?
Pada
Pada15mo ago
yees
Brody
Brody15mo ago
5 minutes for one single network request?
Pada
Pada15mo ago
yes, it's an AI project I'm working on, which treats school transcripts based on pdfs, and returns the answer of where it should go when changing course, or institution. There are many requests, treated with chatGPT in search of similarities in the school grade
Brody
Brody15mo ago
one single network request that the client does takes 5 minutes? are you sure? you have been unsure in the past
Pada
Pada15mo ago
yes, take, when i call the api the response for client, take like 5 minutes
Brody
Brody15mo ago
alright then add --timeout 360 to the end of the CMD command in the dockerfile, this sets the request max timeout to 6 minutes
Pada
Pada15mo ago
okay now, this work fine thank u agaaaain and all solved 😄
Brody
Brody15mo ago
for sure this time lol
Want results from more Discord servers?
Add your server