Send large data (~5MB) to trpc backend
I need to build a feature that reads sqlite files in memory, my idea is to upload the file, parse it and pass it to my trpc backend that will create an in memory sqlite db to perform some queries.
However, it seems passing large payloads takes ages (never finished) even locally, im not sure if trpc is not equipped to handle huge payloads.
Please share if you have a solution or even a better way to do it.
NOTE: I cant have a single db - each use can upload their own sqlite db file
TLDR: how to upload a sqlite dump & perform queries on it
7 Replies
either use a regular next.js api endpoint
or reconsider your architecture (this is what id advise)
@cje will try with a regular api endpoint.
What do you mean by archi? is it the tech stack or the actual approach I thought of?
My main concern is that I cant instantiate sqlite in mem db on client side, hence the need to try to do it on backend (in theory?)
what are you doing that requires you to send an entire sqlite database back and forth
what happens if the user closes the tab or loses their connection before they can send it back
it seems like a recipe for disaster
It doesnt really matter if the data is volatile - Im building a save editor for a game - users upload their save, I show them some data, they can update some data and I overwrite the sql, then send them back the save file - data is meant to be volatile
save file is a binary file that also contains the sqlite db dump
(i built this as a CLI, but trying the web approach for non-techy users :D)
update: yeah even with api endpoint, request is stuck at pending :/
I guess the next sensible approach is to try a file upload rather than send large payload through api - i will see where that leads, might be faster to upload file then just read that, and somehow delete on session close or periodically
ok quick update again: this fixed the delay https://github.com/vercel/next.js/discussions/12517#discussioncomment-2929922 - I also had to add
responseLimit: false
to the config - huge WGitHub
Get Body if bodyParser is Disabled · Discussion #12517 · vercel/nex...
Hi guys, How to get request body if the bodyParser is Disabled ?
Hmm, how about not uploading anything and doing everything at client side(in the browser). I guess there are some libraries for this. A quick search leads to https://github.com/sql-js/sql.js/
GitHub
GitHub - sql-js/sql.js: A javascript library to run SQLite on the web.
A javascript library to run SQLite on the web. . Contribute to sql-js/sql.js development by creating an account on GitHub.
I will recommend using nextjs regular endpoint
It will save you lot of stress
I will share you my upload code snippet
Backend
Front-end