Railway•8mo ago

Does Railway blocks memory for files on runtime?

Im having problem with save file on buffer to process, the stack is Node + Fastify, and the code works on localhost perfectly, but on Railway just stop on

file.toBuffer()

file.toBuffer()

that saves file in memory for usage.

const fileName = file.filename;
const fileData = await file.toBuffer() <- Stops here

const workbook = xlsx.read(fileData, { type: 'buffer' });
const sheetName = workbook.SheetNames[0];
const sheet = workbook.Sheets[sheetName];
const jsonData = xlsx.utils.sheet_to_json(sheet) as IBaseBTGFile[];

const fileName = file.filename;
const fileData = await file.toBuffer() <- Stops here

const workbook = xlsx.read(fileData, { type: 'buffer' });
const sheetName = workbook.SheetNames[0];
const sheet = workbook.Sheets[sheetName];
const jsonData = xlsx.utils.sheet_to_json(sheet) as IBaseBTGFile[];

27 Replies

Percy•8mo ago

Project ID: af0ccfe5-cb9d-43cd-9c80-2ba515dc53b2

Gasavo•8mo ago

af0ccfe5-cb9d-43cd-9c80-2ba515dc53b2

Gasavo•8mo ago

Brody•8mo ago

I've use an in memory buffer and in memory file system before without issues what node version are you using locally and what node version is railway using?

Gasavo•8mo ago

How do i find the version in production?

Brody•8mo ago

should be in the build table at the top of the build logs

Gasavo•8mo ago

I didn't find it, but it must be 18 default

Brody•8mo ago

show a screenshot of the build table please

Gasavo•8mo ago

o, finded, it is 18

Brody•8mo ago

what version are you using locally

Gasavo•8mo ago

18.18

Brody•8mo ago

is file.toBuffer() a new api in node 18.18?

Gasavo•8mo ago

nop, is from fastify

Brody•8mo ago

then this would be a code issue

Gasavo•8mo ago

hmmm ok Not seem like a code issue, now im trying to write a file on the app root directory and the promise never ends

async saveFileOnDisk(file: MultipartFile) {
        return new Promise<string>((resolve, reject) => {
            if (!fs.existsSync('files')) {
                console.log("-> Dir 'files' não existe, criando...")
                fs.mkdirSync('files')
            }

            const fileName = file.filename
            console.log("-> Salvando arquivo", fileName)

            const filePath = `files/${fileName}`

            console.log(file) <- File ok

            const fileStream = fs.createWriteStream(filePath)
            file.file.pipe(fileStream)


            fileStream.on("finish", () => {
                console.log(`-> Arquivo ${fileName} salvo em disco temporariamente com sucesso.`)
                resolve(filePath)
            })

            fileStream.on("error", (error) => {
                console.log(`-> Erro ao salvar o arquivo ${fileName}:`, error)
                reject(error)
            })
        })
    }

async saveFileOnDisk(file: MultipartFile) {
        return new Promise<string>((resolve, reject) => {
            if (!fs.existsSync('files')) {
                console.log("-> Dir 'files' não existe, criando...")
                fs.mkdirSync('files')
            }

            const fileName = file.filename
            console.log("-> Salvando arquivo", fileName)

            const filePath = `files/${fileName}`

            console.log(file) <- File ok

            const fileStream = fs.createWriteStream(filePath)
            file.file.pipe(fileStream)


            fileStream.on("finish", () => {
                console.log(`-> Arquivo ${fileName} salvo em disco temporariamente com sucesso.`)
                resolve(filePath)
            })

            fileStream.on("error", (error) => {
                console.log(`-> Erro ao salvar o arquivo ${fileName}:`, error)
                reject(error)
            })
        })
    }

Change the strategy to not use buffers in memory, again, not works The MultipartFile appears normally in the log

Brody•8mo ago

this is an issue with your app, not railway

Gasavo•8mo ago

i dont wanna intrigue, but testing on others plataform the service run as expected,this logs is from render I just wanna continue using Railway 😢

Brody•8mo ago

give me a minimal reproducible example and I will see what I can find out and side question, how big are these xlsx files you are trying to handle the upload of?

ThallesComH•8mo ago

btw, can't you use stream? XLSX seems to support streams

ThallesComH•8mo ago

and if the file you're receiving is a File web api, you can just call await file.stream(); or just save the file to disk and then use fs to create a read stream. railway have ephemeral disk anyways so no need for a volume.

Gasavo•8mo ago

+/- 700kb

Gasavo•8mo ago

I tried to include as much detail as possible in the code without exporting what is sensitive but still leaving it as close to my structure as possible.

MyProblem.md

Brody•8mo ago

im looking for a repo that i can simply just deploy and get the same error

Gasavo•7mo ago

Hello Thalles, In fact I was doing exactly as you said, the problem was the same, for the promise that converting the upload file into a buffer or stream (I tried both), never ends. Good morning guys, any news on the subject? I don't have the knowledge about infrastructure to test what the error could be... In other providers, both the strategy of storing the file in memory as Buffer or saving a local copy are working, in Railway it still doesn't finalize the promises...

Brody•7mo ago

I'm sorry but this wouldn't be an issue with railway however, this offer is still open

Gasavo•7mo ago

I understood that this description was enough, in this case, just copying the code into this folder structure should work https://discord.com/channels/713503345364697088/1166349559665012846/1166794385866899559

Brody•7mo ago

help me, help you let me know if that's something you'd be interested in providing, otherwise there's not much I can do for you here unfortunately

Gaming

Programming

Does Railway blocks memory for files on runtime?