R
Railway•9mo ago
yavuz

Files have been disappearing from mounted Volume

Project Id: dba6cd6e-5ea1-494c-b77f-81377e24fbde I first noticed this yesterday. Some of my images and some pdfs have been lost somehow. They come up as 404. I reloaded them thru admin. And, another image is lost this morning. I'm saying lost but I don't really know what happened and I cannot browse the mounted directory. I don't have a way of knowing for sure. Are there any logs for file operations? This incident made me question the trust issue since I cannot help to think what will happen after testing in real production. I cannot really find out what happened. Can someone look into this? To better understand the matter, I want to ask about the mounted volume. Is it flat with keys like AWS bucket or are there real directories there? Any help for better understanding this would be appreciated.
104 Replies
Percy
Percy•9mo ago
Project ID: dba6cd6e-5ea1-494c-b77f-81377e24fbde
Brody
Brody•9mo ago
its an actual disk mount, though nvme based to be exact, so nothing at all like an s3 bucket. but lets talk about the issue at hand, what is the app's tech stack?
yavuz
yavuz•9mo ago
What do you mean by tech stack? It's a FastAPI service for react native expo consumer. Thanks for help. You need other info?
Brody
Brody•9mo ago
what is your volume's mount point?
yavuz
yavuz•9mo ago
Well I chose to enter /assets and from what I understood it mounts at the root directory of the app. So it's either /assets or /app/assets assets have sub directories like posters/, pdfs/ I cannot browse it so I don't exactly know
Brody
Brody•9mo ago
whats likely happening is that your mount point is incorrect so you end up saving files to the ephemeral container storage, then those files are lost when you deploy code again
yavuz
yavuz•9mo ago
Yes it happened after deployment but I did not deploy anything since last night and one more image is gone or I don't know what happened
Brody
Brody•9mo ago
what is your current mount point?
yavuz
yavuz•9mo ago
I chose /assets
Brody
Brody•9mo ago
where is your code saving files?
yavuz
yavuz•9mo ago
Well it is using FastAPI FileSystem Storage. I have been using it problem free for a while
Brody
Brody•9mo ago
right and where have you configured that to save files
yavuz
yavuz•9mo ago
In the SqlAlchemy models. I'll post a snippet one sec. poster_storage = FileSystemStorage(path="app/assets/posters")
py PosterUrl = Column(FileType(storage=poster_storage))
py PosterUrl = Column(FileType(storage=poster_storage))
`
Brody
Brody•9mo ago
are you using a dockerfile to deploy your app?
yavuz
yavuz•9mo ago
no. but one was created I gues automatically
Brody
Brody•9mo ago
do you have a dockerfile in your project?
yavuz
yavuz•9mo ago
No
Brody
Brody•9mo ago
is this part of your actual code?
yavuz
yavuz•9mo ago
You mean the snippet above? yes I also have one for static files
Brody
Brody•9mo ago
now this is where you store your posters, but is everything you store within the assets folder?
yavuz
yavuz•9mo ago
py app.mount("/static/assets", StaticFiles(directory="app/assets"), name="static_assets")
py app.mount("/static/assets", StaticFiles(directory="app/assets"), name="static_assets")
` assets folder have html, pdf and some templates and js and css assets is the root of storage I have other folders there for videos, pdfs, images, html, css etc.. I have not other storage
Brody
Brody•9mo ago
you can only mount one volume as im sure know, so all your non static assets need their own folder, like media/posters etc then you want to mount the volume to /app/media
yavuz
yavuz•9mo ago
Yes. that is what I assumed. It was not very clear and you can not really browse so I dont exactly know. However ,that's what I tried to do. One volume /assets and everything inside the subfolders. FastAPI static files mount may be confusing the directory but I cannot know for sure
Brody
Brody•9mo ago
try my suggested changes, once done, upload a file, then redeploy the service, if the file is still there after, you are good to go
yavuz
yavuz•9mo ago
I'm sorry but I missed what you suggested. Should I remount and if so what should be the mount point? I have it as /assets now Mount it to /app/assets ? or /app/media?
Brody
Brody•9mo ago
my two suggestions where - save your uploaded media to a path like media/posters or media/images instead of the paths you correctly use - mount the volume to /app/media
yavuz
yavuz•9mo ago
OK. I'll try it Thanks I tried exactly what you suggested but I cannot load or find the images. So, I have mounted the drive as /app/media and my files system storage saves to app/media.. So what would be public address of my media folder now? I cannot browse the drive so I cannot see where it was saved. I am looking to find it at my app directory like <mysubdomain>/railway.app/media/posters/image.jpg
Brody
Brody•9mo ago
well first off, did the volume usage increase when you uploaded files?
yavuz
yavuz•9mo ago
let me check looks flat I mean the usage line
Brody
Brody•9mo ago
how big where the files that you uploaded, upload like 50mb of files
yavuz
yavuz•9mo ago
no it was an image jpg 171kb I did upload a 56mb video yesterday but that does not show, I assumed file size might be too big but images were loading fine I uploaded one image jpg to test
Brody
Brody•9mo ago
upload that 56mb video today
yavuz
yavuz•9mo ago
It may work today?
Brody
Brody•9mo ago
it you see a bump in the disk usage, then thats one step closer
yavuz
yavuz•9mo ago
I have this code at my app mount point for static files like html jss, they were uploaded along with and part of source code at /app/assets/html , app/assets/html/js and app/assets/html/css. app = FastAPI() app.mount("/static/assets", StaticFiles(directory="app/assets"), name="static_assets") This is not confusing the volume is it?
Brody
Brody•9mo ago
i would like to say this since you brought it up in your original post, nothing here is due to issues with railway, this is all just because the paths are wrong in your code please dont confuse static files like html, js, and css with the media uploads, those are completely separate
yavuz
yavuz•9mo ago
Well, that is what I thought but I wanted to make sure Well, I followed your document examples. You have to understand, I cannot see, browse the volume and the example in your docs was not super clear. My path was not wrong, I just don't understand it yet. So, I managed to get them up and running, I actually saw the ,images but then they were lost since I did not understand waht was going on
Brody
Brody•9mo ago
my examples are only based on my best guess since i know very little about your code
yavuz
yavuz•9mo ago
That is understandable.
Brody
Brody•9mo ago
so did you upload that 56mb video?
yavuz
yavuz•9mo ago
For security purposes, I cannot share whole a lot but enough to let you know, that is what I'm trying 🙂 I appreciate you helping me though. Thanks a lot yes yes no usage bump let me check the mount again
Brody
Brody•9mo ago
whats your current volume mount point
yavuz
yavuz•9mo ago
/app/media directly from settings
Brody
Brody•9mo ago
okay and can you show me the code that is responsible for saving uploaded videos to disk
yavuz
yavuz•9mo ago
Well, that is inside the SqlAlchemy admin which what I'm using. However, it works as intended. I tested it on other servers and locally. It saves a relative path. poster_storage = FileSystemStorage(path="app/media/posters") video_storage = FileSystemStorage(path="app/media/vid") The rest is FastAPI FileSystemStorage from fastapi_storages import FileSystemStorage
Brody
Brody•9mo ago
right so you did my second recommendation, but you haven't done my first
yavuz
yavuz•9mo ago
which was...
Brody
Brody•9mo ago
read the message please
yavuz
yavuz•9mo ago
save your uploaded media to a path like media/posters or media/images instead of the paths you correctly use mount the volume to /app/media ok I see, You mean the storage should be media/images not app/media
Brody
Brody•9mo ago
media/video, media/posters, media/whatever
yavuz
yavuz•9mo ago
I see.. I changed it to media but I missed taking out the app part. Trying now
Brody
Brody•9mo ago
once code changes are done and pushed, wait 1 minute, upload that 56mb video, wait 2 more minutes, check for bump in disk usage
yavuz
yavuz•9mo ago
Oki will do. Thanks again 🙂 Hopefully It'll work this time Unfortunately, no bump in the disk usage and file is 404. I will troubleshoot further and report back here. I have to go to doctor now. Thanks
Brody
Brody•9mo ago
file being 404 is normal for now, you haven't setup a file server, but the no disk usage bump is unfortunate
yavuz
yavuz•9mo ago
Thanks. I will try to unmount and mount again after I get back from doctor. Thanks for troubleshooting with me
Brody
Brody•9mo ago
it's 2:40am for me so I likely won't be available when you're back
yavuz
yavuz•9mo ago
Using Volumes | Railway Docs https://docs.railway.app/guides/volumes
Railway Docs
Using Volumes | Railway Docs
Documentation for Railway
yavuz
yavuz•9mo ago
No problems. I am reading and re-reading to make sure I have not missed anything. Thanks
Brody
Brody•9mo ago
can you send me the docs for this FileSystemStorage stuff? I'll see if I can get a demo running
Brody
Brody•9mo ago
perfect, ill see if i can get a little example app working
yavuz
yavuz•9mo ago
That would be wonderful. However even though I do not know if I'm safe with this but I found a work around. I point the static-files ext of FastAPI and point that to the volume mount on railway Admin saves there and copies the relative url I add it to staic files base and make a auto-url. Since they are just static files and if you store the path correctly then it works this way now
Brody
Brody•9mo ago
so what does your current FileSystemStorage look like?
yavuz
yavuz•9mo ago
poster_storage = FileSystemStorage(path="app/assets/posters")
Brody
Brody•9mo ago
and whats the volume mount point?
yavuz
yavuz•9mo ago
app.mount("/app/assets", StaticFiles(directory="app/assets"), name="static_assets") so this static files line makes it possible for admin to write the correct url to db After all they are static image files as long as db has correct rel path
Brody
Brody•9mo ago
whats the volume mount point?
yavuz
yavuz•9mo ago
/app/assets
Brody
Brody•9mo ago
do you see a bump in the disk usage?
yavuz
yavuz•9mo ago
the same assets folder also has an html subfolder for css and js they work as well yes
Brody
Brody•9mo ago
guess this isnt a relative path then
yavuz
yavuz•9mo ago
It takes it from the route of the server outside the app folder
No description
yavuz
yavuz•9mo ago
I thought the static files and ( {url_for} ) was confusing it but no
Brody
Brody•9mo ago
do the files persist across redeploys?
yavuz
yavuz•9mo ago
I tried once with on image it was fine. I will test it again but scared a little :))
Brody
Brody•9mo ago
to me, what you have seems like it shoulnt work, but i could be wrong
yavuz
yavuz•9mo ago
I know it may not be the best practice but we'll see True but if you think about it, static files ext is there for serving static files like css and js However, if you put more sub dirs in it, it serves them as well Some of the come from deployments but they are not in assets/posters they are in assets/html We'll see I will redeploy for testing tomorrow.
Brody
Brody•9mo ago
hey you can't leave me wondering until tomorrow
yavuz
yavuz•9mo ago
client react app works fine with fastapi on railway
yavuz
yavuz•9mo ago
ok, i'll do it in a short while 🙂 I am scared that it won't work and I'll loose images but if that is the case I'll re enter after fixing
Brody
Brody•9mo ago
sounds good
yavuz
yavuz•9mo ago
I'll report back here
yavuz
yavuz•9mo ago
No description
yavuz
yavuz•9mo ago
Unfortunately no :))) They did not survive this time I had tested one image but it was. may be too soon or cached. For many images no :)) So, I guess I'll have to serve them right
Brody
Brody•9mo ago
the paths are still wrong, unfortunately i was right 😦
yavuz
yavuz•9mo ago
OK, so now I'll wait for you to make an example?
Brody
Brody•9mo ago
yep 🙂
yavuz
yavuz•9mo ago
oki, 🙂 Back in 90s I worked a lot as software developer but I'm a jazz musician and this will be for teaching 🙂 I played most NYC clubs back then 🙂
Brody
Brody•9mo ago
lots has changed in software dev from the 90s for sure
yavuz
yavuz•9mo ago
Yes, I found that out :))
from fastapi import FastAPI, UploadFile
from sqlalchemy import Column, Integer, create_engine
from sqlalchemy.orm import Session, declarative_base
from fastapi_storages import FileSystemStorage
from fastapi_storages.integrations.sqlalchemy import FileType

app = FastAPI()
Base = declarative_base()
engine = create_engine("sqlite:///test.db")


class Example(Base):
__tablename__ = "example"

id = Column(Integer, primary_key=True)
file = Column(FileType(storage=FileSystemStorage(path="/tmp")))


# Create database and table
Base.metadata.create_all(engine)


@app.post("/upload/")
def create_upload_file(file: UploadFile):
example = Example(file=file)
with Session(engine) as session:
session.add(example)
session.commit()
return {"filename": example.file.name}
from fastapi import FastAPI, UploadFile
from sqlalchemy import Column, Integer, create_engine
from sqlalchemy.orm import Session, declarative_base
from fastapi_storages import FileSystemStorage
from fastapi_storages.integrations.sqlalchemy import FileType

app = FastAPI()
Base = declarative_base()
engine = create_engine("sqlite:///test.db")


class Example(Base):
__tablename__ = "example"

id = Column(Integer, primary_key=True)
file = Column(FileType(storage=FileSystemStorage(path="/tmp")))


# Create database and table
Base.metadata.create_all(engine)


@app.post("/upload/")
def create_upload_file(file: UploadFile):
example = Example(file=file)
with Session(engine) as session:
session.add(example)
session.commit()
return {"filename": example.file.name}
This might be the right example may be you saw it already
Brody
Brody•9mo ago
first ill try to get this working https://aminalaee.dev/fastapi-storages/usage/#filesystemstorage are you using sqlite?
yavuz
yavuz•9mo ago
I’m using MySQL
Brody
Brody•9mo ago
okay cool got an crude example working, media uploads and serving the media files, uploaded media persists across multiple deploys. getting it in a repo now
yavuz
yavuz•9mo ago
Thank you. I’ll try first thing tomorrow and report back here. Thanks a lot:))
Brody
Brody•9mo ago
no problem!
yavuz
yavuz•9mo ago
Hey Brody 🙂 I started working on the repo you posted yesterday. I noticed a railway.json It has a different deploy / start command than mine. It looks like it's irrelevant for what I'm doing so I don't need it right? I also wanted to ask you about the mounting volume director. Is it one folder or can we create subdirectories. I have different file types and it looks like I have to create 5 different file uploads and It would nice to have them on separate dirs. OK, thanks Brody 🙂 I got it working. You're probably sleeping, considering the time zone difference. I got it working ( all five different directories ) This upload methods on the main file was not sth I wanted so I tried to change it to a router and found out that you cannot mount static files on routers. However, it does work on apps. So I created an app for uploads and mounted it on the main app with app.mount.. Then of course you need to give the app you're mounting a path as well then... it did not work. I either mixed up the paths however I tried to consider app mounting point vs media folder mounting point etc., I could not get it working yet but if I do I'll report back. Anyways, basic way of things are working now. Thanks a lot for this 🙂 OK, I tried with right paths and could not get it working with a sub app mounting to main app with "/upload" mount point. So I gave up on that. However, starting from your example, I managed to get it working mounted on the main app. Thanks 🙂
Brody
Brody•9mo ago
I use hypercorn, but I'm sure you use uvicorn (at least I hope) so it's irrelevant, they both do the same job you can create sub directorys, but the code would do that for you automatically as I've noticed I have confidence that with my example you'll be able to get everything working as you want! and always make sure to redeploy the app just as a sanity check to make sure the images stick
yavuz
yavuz•9mo ago
Yes, I have double checked and the files persist 🙂 I am hopeful that I can get it working the way I want too but I have to clean up a few things first. Besides, it was just a cosmetic thing, I just did not want main to be crowded that's all 🙂 Thanks a lot for this 🙂 I really appreciate the help
Brody
Brody•9mo ago
awesome, happy to help!
yavuz
yavuz•9mo ago
Hey check out my recent music video if or when you have time 🙂
yavuz
yavuz•9mo ago
YAVUZ AKYAZICI
YouTube
Motor Helmet On - Official Music Video
Motor Helmet On Music Video Funda Akyazıcı Vocal ( @fundaakyazici ) Yavuz Akyazıcı Guitar, Slide Guitar, Musical Arrangement Kevin Fitzgerald Burke Vocal ( @kevinfitzgeraldburke ) Enver Muhamedi Bass Cengiz Baysal Drums Songwriters: Funda & Yavuz Akyazıcı Complementary back vocal lyrics: Kevin Fitzgerald Burke Websites: https://yavuzakyazici....
Brody
Brody•9mo ago
cool, I will
Want results from more Discord servers?
Add your server