R
Railway13mo ago
Gustavo

Does Railway blocks memory for files on runtime?

Im having problem with save file on buffer to process, the stack is Node + Fastify, and the code works on localhost perfectly, but on Railway just stop on
file.toBuffer()
file.toBuffer()
that saves file in memory for usage.
const fileName = file.filename;
const fileData = await file.toBuffer() <- Stops here

const workbook = xlsx.read(fileData, { type: 'buffer' });
const sheetName = workbook.SheetNames[0];
const sheet = workbook.Sheets[sheetName];
const jsonData = xlsx.utils.sheet_to_json(sheet) as IBaseBTGFile[];
const fileName = file.filename;
const fileData = await file.toBuffer() <- Stops here

const workbook = xlsx.read(fileData, { type: 'buffer' });
const sheetName = workbook.SheetNames[0];
const sheet = workbook.Sheets[sheetName];
const jsonData = xlsx.utils.sheet_to_json(sheet) as IBaseBTGFile[];
27 Replies
Percy
Percy13mo ago
Project ID: af0ccfe5-cb9d-43cd-9c80-2ba515dc53b2
Gustavo
Gustavo13mo ago
af0ccfe5-cb9d-43cd-9c80-2ba515dc53b2
Gustavo
Gustavo13mo ago
No description
Brody
Brody13mo ago
I've use an in memory buffer and in memory file system before without issues what node version are you using locally and what node version is railway using?
Gustavo
Gustavo13mo ago
How do i find the version in production?
Brody
Brody13mo ago
should be in the build table at the top of the build logs
Gustavo
Gustavo13mo ago
I didn't find it, but it must be 18 default
Brody
Brody13mo ago
show a screenshot of the build table please
Gustavo
Gustavo13mo ago
o, finded, it is 18
Brody
Brody13mo ago
what version are you using locally
Gustavo
Gustavo13mo ago
18.18
Brody
Brody13mo ago
is file.toBuffer() a new api in node 18.18?
Gustavo
Gustavo13mo ago
nop, is from fastify
Brody
Brody13mo ago
then this would be a code issue
Gustavo
Gustavo13mo ago
hmmm ok Not seem like a code issue, now im trying to write a file on the app root directory and the promise never ends
async saveFileOnDisk(file: MultipartFile) {
return new Promise<string>((resolve, reject) => {
if (!fs.existsSync('files')) {
console.log("-> Dir 'files' não existe, criando...")
fs.mkdirSync('files')
}

const fileName = file.filename
console.log("-> Salvando arquivo", fileName)

const filePath = `files/${fileName}`

console.log(file) <- File ok

const fileStream = fs.createWriteStream(filePath)
file.file.pipe(fileStream)


fileStream.on("finish", () => {
console.log(`-> Arquivo ${fileName} salvo em disco temporariamente com sucesso.`)
resolve(filePath)
})

fileStream.on("error", (error) => {
console.log(`-> Erro ao salvar o arquivo ${fileName}:`, error)
reject(error)
})
})
}
async saveFileOnDisk(file: MultipartFile) {
return new Promise<string>((resolve, reject) => {
if (!fs.existsSync('files')) {
console.log("-> Dir 'files' não existe, criando...")
fs.mkdirSync('files')
}

const fileName = file.filename
console.log("-> Salvando arquivo", fileName)

const filePath = `files/${fileName}`

console.log(file) <- File ok

const fileStream = fs.createWriteStream(filePath)
file.file.pipe(fileStream)


fileStream.on("finish", () => {
console.log(`-> Arquivo ${fileName} salvo em disco temporariamente com sucesso.`)
resolve(filePath)
})

fileStream.on("error", (error) => {
console.log(`-> Erro ao salvar o arquivo ${fileName}:`, error)
reject(error)
})
})
}
Change the strategy to not use buffers in memory, again, not works The MultipartFile appears normally in the log
Brody
Brody13mo ago
this is an issue with your app, not railway
Gustavo
Gustavo13mo ago
i dont wanna intrigue, but testing on others plataform the service run as expected,this logs is from render I just wanna continue using Railway 😢
No description
Brody
Brody13mo ago
give me a minimal reproducible example and I will see what I can find out and side question, how big are these xlsx files you are trying to handle the upload of?
ThallesComH
ThallesComH13mo ago
btw, can't you use stream? XLSX seems to support streams
No description
ThallesComH
ThallesComH13mo ago
and if the file you're receiving is a File web api, you can just call await file.stream(); or just save the file to disk and then use fs to create a read stream. railway have ephemeral disk anyways so no need for a volume.
Gustavo
Gustavo13mo ago
+/- 700kb
Gustavo
Gustavo13mo ago
I tried to include as much detail as possible in the code without exporting what is sensitive but still leaving it as close to my structure as possible.
Brody
Brody13mo ago
im looking for a repo that i can simply just deploy and get the same error
Gustavo
Gustavo12mo ago
Hello Thalles, In fact I was doing exactly as you said, the problem was the same, for the promise that converting the upload file into a buffer or stream (I tried both), never ends. Good morning guys, any news on the subject? I don't have the knowledge about infrastructure to test what the error could be... In other providers, both the strategy of storing the file in memory as Buffer or saving a local copy are working, in Railway it still doesn't finalize the promises...
Brody
Brody12mo ago
I'm sorry but this wouldn't be an issue with railway however, this offer is still open
Gustavo
Gustavo12mo ago
I understood that this description was enough, in this case, just copying the code into this folder structure should work https://discord.com/channels/713503345364697088/1166349559665012846/1166794385866899559
Brody
Brody12mo ago
help me, help you let me know if that's something you'd be interested in providing, otherwise there's not much I can do for you here unfortunately
Want results from more Discord servers?
Add your server