Runpod•2y ago

How should I store/load my data for network storage?

Hi,

I've been keeping my data in an sql database which is excruciatingly slow on runpod with a network storage.
But I don't see any obvious alternative..

In what type of file could my data live in on the disk,
in order for it to be loaded fast in the network storage scenario of runpod?

J.•2/16/24, 4:09 PM

How much data are u loading?

J.•2/16/24, 5:12 PM

how are u loading it? why not a CSV / JSON file? Or something? Must it be SQL? Is this millions of rows? etc.

ashley•2/16/24, 5:25 PM

Anything read from network storage is as slow as constipation, network storage is garbage and unusable

panos.firbasOP•2/16/24, 5:25 PM

hi, my data is not text it's genomics so right now i'm saving numpy arrays in an sql

panos.firbasOP•2/16/24, 5:25 PM

it's about 200G

J.•2/16/24, 5:25 PM

Yeah, the way I see network storage is as a External Hard drive lol

Ppanos.firbas hi, my data is not text it's genomics so right now i'm saving numpy arrays in an...

J.•2/16/24, 5:25 PM

Do u need to load all 200G? at once?

J.•2/16/24, 5:25 PM

Does the data change?

ashley•2/16/24, 5:25 PM

An external hard drive is MUCH faster than network storage

panos.firbasOP•2/16/24, 5:26 PM

it does not change, i need to read it once per epoch, ideally in random order

ashley•2/16/24, 5:26 PM

Network storage is like sending stuff via a pigeon, but then again a pigeon will probably even be faster

Ppanos.firbas it does not change, i need to read it once per epoch, ideally in random order

J.•2/16/24, 5:26 PM

How much data are u loading per epoch?

panos.firbasOP•2/16/24, 5:27 PM

all of it eventually

J.•2/16/24, 5:28 PM

How are u reading it? is ur data indexes properly?

J.•2/16/24, 5:28 PM

So just to confirm:
You have a SQLLite database for about 200GB, you are randomly selecting sizes of data for epoch cycles?

ashley•2/16/24, 5:29 PM

You can maybe try chatting with @JM about getting some kind of custom storage for your use case, but you may have to commit to a certain level of spend or something.

panos.firbasOP•2/16/24, 5:29 PM

i did a test, a script that just loads all the records from the db. In my system it's doing something like 6000/second in runpod it was doing 2/second

J.•2/16/24, 5:30 PM

Honestly, you could probably just divide your data straight up into files of N x Epoch_Size in your network storage. Create a hashmap of all the file names. Randomly pick one, remove from hashmap.

Load the file for the next X cycles. And in the background you can also in a different thread load into a variable the next file for the next epoch cycle

panos.firbasOP•2/16/24, 5:30 PM

I shuffle the main index and query the db index by index. Every new epoch is a new shuffle

Ppanos.firbas i did a test, a script that just loads all the records from the db. In my system...

ashley•2/16/24, 5:30 PM

Not surprising, Network storage is unbarably slow

Ppanos.firbas I shuffle the main index and query the db index by index. Every new epoch is a n...

J.•2/16/24, 5:30 PM

I feel, that something about this feels wrong...

J.•2/16/24, 5:31 PM

What does shuffle the main index mean?

JJ.Honestly, you could probably just divide your data straight up into files of N x...

J.•2/16/24, 5:31 PM

But if it was me, I'd do this

J.•2/16/24, 5:31 PM

u might not even need to load something in the background

J.•2/16/24, 5:31 PM

But i'd just straight up divide to the pure text format, and avoid the SQLite

J.•2/16/24, 5:31 PM

SQLite, with all the query time could also be straight up hurting u just due to the way the searching would be working over 200GBs

ashley•2/16/24, 5:31 PM

Both have to load from disk, so I don't see how it will make any difference

panos.firbasOP•2/16/24, 5:32 PM

essentially
order = np.arrange(lenofdatabase) +1
np.shuffle(order)

then
for i in order:
SELECT WHERE i
fetch one

Aashley Both have to load from disk, so I don't see how it will make any difference

J.•2/16/24, 5:32 PM

SQLite, has to keep a tree of his data as he goes indexes for a shuffle

Ppanos.firbas essentially order = np.arrange(lenofdatabase) +1 np.shuffle(order) then for i i...

J.•2/16/24, 5:32 PM

Yea this is probably bad

JJ.But i'd just straight up divide to the pure text format, and avoid the SQLite

panos.firbasOP•2/16/24, 5:32 PM

but my data is not text!

J.•2/16/24, 5:32 PM

its looks like it doing a full table scan

J.•2/16/24, 5:32 PM

Ah what is the data tho?

J.•2/16/24, 5:33 PM

its just np arrays right?

ashley•2/16/24, 5:33 PM

numpy arrays he said

J.•2/16/24, 5:33 PM

u can serialize it into something

panos.firbasOP•2/16/24, 5:33 PM

yep numpy arrays

J.•2/16/24, 5:33 PM

Honestly, you could do some sort of optimization around that sql query to do like preloaded batches of randomness

panos.firbasOP•2/16/24, 5:33 PM

that can't be faster than loading an array from the disk !

J.•2/16/24, 5:33 PM

this looks like a random selection every time

J.•2/16/24, 5:33 PM

over the full database

Ppanos.firbas that can't be faster than loading an array from the disk !

J.•2/16/24, 5:34 PM

Well, if ur loading from a SQLite database, or a text, its both from disk, but yeah, i get the point. I think we can go the route of fixing hte SQL Query

panos.firbasOP•2/16/24, 5:34 PM

arent databases incredibly fast at fetching record i from a table when i is a proper index? that's what i'm doing, in a specific order

J.•2/16/24, 5:34 PM

Let me read over this sql query some more and think xD been a bit since ive worked with it

J.•2/16/24, 5:34 PM

what ur trying to do

ashley•2/16/24, 5:35 PM

Network storage is so slow you are probably better off storing your data in a cloud mongodb or something and fetching from there, I am sure it will be much faster

panos.firbasOP•2/16/24, 5:35 PM

and as i said, this system is blazing fast on my system

panos.firbasOP•2/16/24, 5:35 PM

so what if I don't use network storage

panos.firbasOP•2/16/24, 5:35 PM

but instead spawn a pod with the needed space ?

panos.firbasOP•2/16/24, 5:35 PM

and just move the data in there in 'runtime' ?

How should I store/load my data for network storage?

Similar Threads

Similar Threads

Similar Threads