Railway•5mo ago

Files have been disappearing from mounted Volume

Project Id: dba6cd6e-5ea1-494c-b77f-81377e24fbde I first noticed this yesterday. Some of my images and some pdfs have been lost somehow. They come up as 404. I reloaded them thru admin. And, another image is lost this morning. I'm saying lost but I don't really know what happened and I cannot browse the mounted directory. I don't have a way of knowing for sure. Are there any logs for file operations? This incident made me question the trust issue since I cannot help to think what will happen after testing in real production. I cannot really find out what happened. Can someone look into this? To better understand the matter, I want to ask about the mounted volume. Is it flat with keys like AWS bucket or are there real directories there? Any help for better understanding this would be appreciated.

Solution:

heres the repo https://github.com/brody192/fastapi-FileSystemStorage

Jump to solution

104 Replies

Percy•5mo ago

Project ID: dba6cd6e-5ea1-494c-b77f-81377e24fbde

Brody•5mo ago

its an actual disk mount, though nvme based to be exact, so nothing at all like an s3 bucket. but lets talk about the issue at hand, what is the app's tech stack?

yavuz•5mo ago

What do you mean by tech stack? It's a FastAPI service for react native expo consumer. Thanks for help. You need other info?

Brody•5mo ago

what is your volume's mount point?

yavuz•5mo ago

Well I chose to enter /assets and from what I understood it mounts at the root directory of the app. So it's either /assets or /app/assets assets have sub directories like posters/, pdfs/ I cannot browse it so I don't exactly know

Brody•5mo ago

whats likely happening is that your mount point is incorrect so you end up saving files to the ephemeral container storage, then those files are lost when you deploy code again

yavuz•5mo ago

Yes it happened after deployment but I did not deploy anything since last night and one more image is gone or I don't know what happened

Brody•5mo ago

what is your current mount point?

yavuz•5mo ago

I chose /assets

Brody•5mo ago

where is your code saving files?

yavuz•5mo ago

Well it is using FastAPI FileSystem Storage. I have been using it problem free for a while

Brody•5mo ago

right and where have you configured that to save files

yavuz•5mo ago

In the SqlAlchemy models. I'll post a snippet one sec. poster_storage = FileSystemStorage(path="app/assets/posters")

py PosterUrl = Column(FileType(storage=poster_storage))

py PosterUrl = Column(FileType(storage=poster_storage))

Brody•5mo ago

are you using a dockerfile to deploy your app?

yavuz•5mo ago

no. but one was created I gues automatically

Brody•5mo ago

do you have a dockerfile in your project?

yavuz•5mo ago

Brody•5mo ago

is this part of your actual code?

yavuz•5mo ago

You mean the snippet above? yes I also have one for static files

Brody•5mo ago

now this is where you store your posters, but is everything you store within the assets folder?

yavuz•5mo ago

py app.mount("/static/assets", StaticFiles(directory="app/assets"), name="static_assets")

py app.mount("/static/assets", StaticFiles(directory="app/assets"), name="static_assets")

` assets folder have html, pdf and some templates and js and css assets is the root of storage I have other folders there for videos, pdfs, images, html, css etc.. I have not other storage

Brody•5mo ago

you can only mount one volume as im sure know, so all your non static assets need their own folder, like media/posters etc then you want to mount the volume to /app/media

yavuz•5mo ago

Yes. that is what I assumed. It was not very clear and you can not really browse so I dont exactly know. However ,that's what I tried to do. One volume /assets and everything inside the subfolders. FastAPI static files mount may be confusing the directory but I cannot know for sure

Brody•5mo ago

try my suggested changes, once done, upload a file, then redeploy the service, if the file is still there after, you are good to go

yavuz•5mo ago

I'm sorry but I missed what you suggested. Should I remount and if so what should be the mount point? I have it as /assets now Mount it to /app/assets ? or /app/media?

Brody•5mo ago

my two suggestions where - save your uploaded media to a path like media/posters or media/images instead of the paths you correctly use - mount the volume to /app/media

yavuz•5mo ago

OK. I'll try it Thanks I tried exactly what you suggested but I cannot load or find the images. So, I have mounted the drive as /app/media and my files system storage saves to app/media.. So what would be public address of my media folder now? I cannot browse the drive so I cannot see where it was saved. I am looking to find it at my app directory like <mysubdomain>/railway.app/media/posters/image.jpg

Brody•5mo ago

well first off, did the volume usage increase when you uploaded files?

yavuz•5mo ago

let me check looks flat I mean the usage line

Brody•5mo ago

how big where the files that you uploaded, upload like 50mb of files

yavuz•5mo ago

no it was an image jpg 171kb I did upload a 56mb video yesterday but that does not show, I assumed file size might be too big but images were loading fine I uploaded one image jpg to test

Brody•5mo ago

upload that 56mb video today

yavuz•5mo ago

It may work today?

Brody•5mo ago

it you see a bump in the disk usage, then thats one step closer

yavuz•5mo ago

I have this code at my app mount point for static files like html jss, they were uploaded along with and part of source code at /app/assets/html , app/assets/html/js and app/assets/html/css. app = FastAPI() app.mount("/static/assets", StaticFiles(directory="app/assets"), name="static_assets") This is not confusing the volume is it?

Brody•5mo ago

i would like to say this since you brought it up in your original post, nothing here is due to issues with railway, this is all just because the paths are wrong in your code please dont confuse static files like html, js, and css with the media uploads, those are completely separate

yavuz•5mo ago

Well, that is what I thought but I wanted to make sure Well, I followed your document examples. You have to understand, I cannot see, browse the volume and the example in your docs was not super clear. My path was not wrong, I just don't understand it yet. So, I managed to get them up and running, I actually saw the ,images but then they were lost since I did not understand waht was going on

Brody•5mo ago

my examples are only based on my best guess since i know very little about your code

yavuz•5mo ago

That is understandable.

Brody•5mo ago

so did you upload that 56mb video?

yavuz•5mo ago

For security purposes, I cannot share whole a lot but enough to let you know, that is what I'm trying 🙂 I appreciate you helping me though. Thanks a lot yes yes no usage bump let me check the mount again

Brody•5mo ago

whats your current volume mount point

yavuz•5mo ago

/app/media directly from settings

Brody•5mo ago

okay and can you show me the code that is responsible for saving uploaded videos to disk

yavuz•5mo ago

Well, that is inside the SqlAlchemy admin which what I'm using. However, it works as intended. I tested it on other servers and locally. It saves a relative path. poster_storage = FileSystemStorage(path="app/media/posters") video_storage = FileSystemStorage(path="app/media/vid") The rest is FastAPI FileSystemStorage from fastapi_storages import FileSystemStorage

Brody•5mo ago

right so you did my second recommendation, but you haven't done my first

yavuz•5mo ago

which was...

Brody•5mo ago

read the message please

yavuz•5mo ago

save your uploaded media to a path like media/posters or media/images instead of the paths you correctly use mount the volume to /app/media ok I see, You mean the storage should be media/images not app/media

Brody•5mo ago

media/video, media/posters, media/whatever

yavuz•5mo ago

I see.. I changed it to media but I missed taking out the app part. Trying now

Brody•5mo ago

once code changes are done and pushed, wait 1 minute, upload that 56mb video, wait 2 more minutes, check for bump in disk usage

yavuz•5mo ago

Oki will do. Thanks again 🙂 Hopefully It'll work this time Unfortunately, no bump in the disk usage and file is 404. I will troubleshoot further and report back here. I have to go to doctor now. Thanks

Brody•5mo ago

file being 404 is normal for now, you haven't setup a file server, but the no disk usage bump is unfortunate

yavuz•5mo ago

Thanks. I will try to unmount and mount again after I get back from doctor. Thanks for troubleshooting with me

Brody•5mo ago

it's 2:40am for me so I likely won't be available when you're back

yavuz•5mo ago

Using Volumes | Railway Docs https://docs.railway.app/guides/volumes

Railway Docs

Using Volumes | Railway Docs

Documentation for Railway

yavuz•5mo ago

No problems. I am reading and re-reading to make sure I have not missed anything. Thanks

Brody•5mo ago

can you send me the docs for this FileSystemStorage stuff? I'll see if I can get a demo running

yavuz•5mo ago

https://aminalaee.dev/fastapi-storages/usage/

Usage - FastAPI Storages

FastAPI Storages

yavuz•5mo ago

https://aminalaee.dev/sqladmin/working_with_files/

Working with Files and Images - SQLAlchemy Admin

Flexible admin dashboard for SQLAlchemy.

Brody•5mo ago

perfect, ill see if i can get a little example app working

yavuz•5mo ago

That would be wonderful. However even though I do not know if I'm safe with this but I found a work around. I point the static-files ext of FastAPI and point that to the volume mount on railway Admin saves there and copies the relative url I add it to staic files base and make a auto-url. Since they are just static files and if you store the path correctly then it works this way now

Brody•5mo ago

so what does your current FileSystemStorage look like?

yavuz•5mo ago

poster_storage = FileSystemStorage(path="app/assets/posters")

Brody•5mo ago

and whats the volume mount point?

yavuz•5mo ago

app.mount("/app/assets", StaticFiles(directory="app/assets"), name="static_assets") so this static files line makes it possible for admin to write the correct url to db After all they are static image files as long as db has correct rel path

Brody•5mo ago

whats the volume mount point?

yavuz•5mo ago

/app/assets

Brody•5mo ago

do you see a bump in the disk usage?

yavuz•5mo ago

the same assets folder also has an html subfolder for css and js they work as well yes

Brody•5mo ago

guess this isnt a relative path then

yavuz•5mo ago

It takes it from the route of the server outside the app folder

yavuz•5mo ago

I thought the static files and ( {url_for} ) was confusing it but no

Brody•5mo ago

do the files persist across redeploys?

yavuz•5mo ago

I tried once with on image it was fine. I will test it again but scared a little :))

Brody•5mo ago

to me, what you have seems like it shoulnt work, but i could be wrong

yavuz•5mo ago

I know it may not be the best practice but we'll see True but if you think about it, static files ext is there for serving static files like css and js However, if you put more sub dirs in it, it serves them as well Some of the come from deployments but they are not in assets/posters they are in assets/html We'll see I will redeploy for testing tomorrow.

Brody•5mo ago

hey you can't leave me wondering until tomorrow

yavuz•5mo ago

client react app works fine with fastapi on railway

yavuz•5mo ago

ok, i'll do it in a short while 🙂 I am scared that it won't work and I'll loose images but if that is the case I'll re enter after fixing

Brody•5mo ago

sounds good

yavuz•5mo ago

I'll report back here

yavuz•5mo ago

Unfortunately no :))) They did not survive this time I had tested one image but it was. may be too soon or cached. For many images no :)) So, I guess I'll have to serve them right

Brody•5mo ago

the paths are still wrong, unfortunately i was right 😦

yavuz•5mo ago

OK, so now I'll wait for you to make an example?

Brody•5mo ago

yep 🙂

yavuz•5mo ago

oki, 🙂 Back in 90s I worked a lot as software developer but I'm a jazz musician and this will be for teaching 🙂 I played most NYC clubs back then 🙂

Brody•5mo ago

lots has changed in software dev from the 90s for sure

yavuz•5mo ago

Yes, I found that out :))

from fastapi import FastAPI, UploadFile
from sqlalchemy import Column, Integer, create_engine
from sqlalchemy.orm import Session, declarative_base
from fastapi_storages import FileSystemStorage
from fastapi_storages.integrations.sqlalchemy import FileType

app = FastAPI()
Base = declarative_base()
engine = create_engine("sqlite:///test.db")


class Example(Base):
    __tablename__ = "example"

    id = Column(Integer, primary_key=True)
    file = Column(FileType(storage=FileSystemStorage(path="/tmp")))


# Create database and table
Base.metadata.create_all(engine)


@app.post("/upload/")
def create_upload_file(file: UploadFile):
    example = Example(file=file)
    with Session(engine) as session:
        session.add(example)
        session.commit()
        return {"filename": example.file.name}

from fastapi import FastAPI, UploadFile
from sqlalchemy import Column, Integer, create_engine
from sqlalchemy.orm import Session, declarative_base
from fastapi_storages import FileSystemStorage
from fastapi_storages.integrations.sqlalchemy import FileType

app = FastAPI()
Base = declarative_base()
engine = create_engine("sqlite:///test.db")


class Example(Base):
    __tablename__ = "example"

    id = Column(Integer, primary_key=True)
    file = Column(FileType(storage=FileSystemStorage(path="/tmp")))


# Create database and table
Base.metadata.create_all(engine)


@app.post("/upload/")
def create_upload_file(file: UploadFile):
    example = Example(file=file)
    with Session(engine) as session:
        session.add(example)
        session.commit()
        return {"filename": example.file.name}

This might be the right example may be you saw it already

Brody•5mo ago

first ill try to get this working https://aminalaee.dev/fastapi-storages/usage/#filesystemstorage are you using sqlite?

yavuz•5mo ago

I’m using MySQL

Brody•5mo ago

okay cool got an crude example working, media uploads and serving the media files, uploaded media persists across multiple deploys. getting it in a repo now

Solution

Brody•5mo ago

heres the repo https://github.com/brody192/fastapi-FileSystemStorage

yavuz•5mo ago

Thank you. I’ll try first thing tomorrow and report back here. Thanks a lot:))

Brody•5mo ago

no problem!

yavuz•5mo ago

Hey Brody 🙂 I started working on the repo you posted yesterday. I noticed a railway.json It has a different deploy / start command than mine. It looks like it's irrelevant for what I'm doing so I don't need it right? I also wanted to ask you about the mounting volume director. Is it one folder or can we create subdirectories. I have different file types and it looks like I have to create 5 different file uploads and It would nice to have them on separate dirs. OK, thanks Brody 🙂 I got it working. You're probably sleeping, considering the time zone difference. I got it working ( all five different directories ) This upload methods on the main file was not sth I wanted so I tried to change it to a router and found out that you cannot mount static files on routers. However, it does work on apps. So I created an app for uploads and mounted it on the main app with app.mount.. Then of course you need to give the app you're mounting a path as well then... it did not work. I either mixed up the paths however I tried to consider app mounting point vs media folder mounting point etc., I could not get it working yet but if I do I'll report back. Anyways, basic way of things are working now. Thanks a lot for this 🙂 OK, I tried with right paths and could not get it working with a sub app mounting to main app with "/upload" mount point. So I gave up on that. However, starting from your example, I managed to get it working mounted on the main app. Thanks 🙂

Brody•5mo ago

I use hypercorn, but I'm sure you use uvicorn (at least I hope) so it's irrelevant, they both do the same job you can create sub directorys, but the code would do that for you automatically as I've noticed I have confidence that with my example you'll be able to get everything working as you want! and always make sure to redeploy the app just as a sanity check to make sure the images stick

yavuz•5mo ago

Yes, I have double checked and the files persist 🙂 I am hopeful that I can get it working the way I want too but I have to clean up a few things first. Besides, it was just a cosmetic thing, I just did not want main to be crowded that's all 🙂 Thanks a lot for this 🙂 I really appreciate the help

Brody•5mo ago

awesome, happy to help!

yavuz•5mo ago

Hey check out my recent music video if or when you have time 🙂

yavuz•5mo ago

https://www.youtube.com/watch?v=kSwg2HKxY10

YAVUZ AKYAZICI

YouTube

Motor Helmet On - Official Music Video

Motor Helmet On Music Video Funda Akyazıcı Vocal ( @fundaakyazici ) Yavuz Akyazıcı Guitar, Slide Guitar, Musical Arrangement Kevin Fitzgerald Burke Vocal ( @kevinfitzgeraldburke ) Enver Muhamedi Bass Cengiz Baysal Drums Songwriters: Funda & Yavuz Akyazıcı Complementary back vocal lyrics: Kevin Fitzgerald Burke Websites: https://yavuzakyazici....

Brody•5mo ago

cool, I will

Gaming

Programming

Files have been disappearing from mounted Volume