How to add more RAM to the Pod
Hi! Is there a way to set RAM manually? I mean, the current options is limited for example max 283gb RAM. But I need 1TB. How do I add more RAM?
Thank you.
Thank you.
Juse clusters with more GPUs
Jor cpu pods, with larger ram
AIs this what you mean? It's empty don't know why..
CPU pods only limited to about 250 gb RAM.
CPU pods only limited to about 250 gb RAM.
Jthey have larger vram because they have more gpu's yes
J(in one cluster)
Jbtw clusters are expensive tho
Jnvm you can go here (look below in my screenshot) and increase more gpus and still get 1tb of ram
JThese setups allows you to get 1tb of ram
Jthese are in pods
AAh okayy, finally understand the parameter for it. Thank you so much..
So.... with about $7/hour, I can try use kimi k2 for my self? hahahaha. And try the quantized version with so much cheaper price? :D:D
Anyway thank youu
So.... with about $7/hour, I can try use kimi k2 for my self? hahahaha. And try the quantized version with so much cheaper price? :D:D
Anyway thank youu
JHmm or use the amd ones
JTry apis if you don't wanna self host just for your self (if it's cheaper that way why not)
Aapis? Do you mean like OpenRouter, Chutes?
JYe openrtr tgther ai
JUnless you need high throughput, that is effecient on cloud gpus
AI see... I thought of using Runpod because of hit rate limit on free one. and if paying, I wonder if assumed 1 hour on OpenRouter kimi usage can be cheaper than I use runpod vs speed. Basically it gets back to price value.. If runpod is cheaper for 1 hour full usage then it's really worth it ( based on my tests a couple of days and for my use cases ).
What do you think?
What do you think?
AIf you said openrouter will be cheaper then I won't bother trying since it's also kind of a work to make the template first for gguf one. << Never try this, I also want to know if gguf one (from unsloth) is enough for roocode usage.
JHmm depends on what you use
JHow much tokens estimate
JMost likely if you're alone, using the full kimi k2 it's more Exp in runpod or wherever it is, paying more than. .8 per hr
Awow... This is really a hard question hahaahah. Honestly I don't know, but I assume it is actually so much because when I try using gemini pro, it's taking maybe about 80M token within 3 hours, so it's around 28M token per hour or more depending on the task.
I see that are cases that is not using much token, but there is using so much token that I'm not even except it is possible.
I see that are cases that is not using much token, but there is using so much token that I'm not even except it is possible.
ABased on your comment, I think it's wise for me to just try using runpod for full 1 hour paying to see how much it is ( I don't want to do this at first ). hahahah. But still, on OpenRouter, total token matter so much when runpod is no, that's also some things to consider. If we're using maximum token fully 1 hour, of course runpod will be cheaper right?
I need maybe 50t/s since when I use roocode, I got that speed ( based on what chutes provider said on openrouter )
I need maybe 50t/s since when I use roocode, I got that speed ( based on what chutes provider said on openrouter )
JThe runpod costs are estimatable if you're using pods, even without using it
JJust multiply (the price per hour + cost of your storage/hr) with your usage
JWow that's quite heavy use
AI also shocked by it.. I don't know how gemini cli works at that time, but it really burns the token so much.. hahaha.
JYeah sure try the perf on runpod
JOh gemini cli? Maybe it's agentic use
JIt explores many files, thinking
JTool calling
JNot sure if kimi k2 can support cli like use
JIf it's not fine tuned for it
ANo no.. after I got billed about $35 within 4 days, I try roocode instantly, and try for the free, and honestly it's good. But about these 2 days, somehow I get limited so much ( maybe the chutes doing the limiting, since openrouter says 1000 request / day, so it should be okay but no ).
That's why, if I must pay, I think I want it to be no restriction and of course low cost since it's all for my personal research.
That's why, if I must pay, I think I want it to be no restriction and of course low cost since it's all for my personal research.
Jahh ic
JHow much are you willing to pay?
Jmy friend said he managed to run it in 4x or 5x mi300x (its low stock currently) with 4k context 50t/s, usually you'd want a higher context tho
if you're okay with ram offloading ig it fits with other gpus
if you're okay with ram offloading ig it fits with other gpus
AThis is also hard to say since this is relative type question. Like if it's 50t/s, I feel this already pretty pretty good, I'm okay with paying $1.5/hour ( Note: I'm not like a crazy person that turns this on 24 hours HAHAHA. I use it when I'm only on computer too.. So maximum is 3 hours a day ). If 50t/s with 64K token, I already feel that's enough. Even if it's 30t/s, I also already okay.
So let say I can even get it on much much lower rate, I also okay with 15t/s
. Sorry to be like a cheapskate, but I'm not rich enough to burn money.
Is it too much to ask?
So let say I can even get it on much much lower rate, I also okay with 15t/s
. Sorry to be like a cheapskate, but I'm not rich enough to burn money. Is it too much to ask?
Jwoah
J1.5$ an hour?
HWhen I said 50t/s I mean processing
ADepends on t/s. If it's acceptable, of course I'm okay offloading to ram. On my local PC, I try it with offloading, but I only can run up to 14B model Q4, and when using high context, I already got soooo poor speed.
Jso to run kimi you need around 1tb+ of mixed ram and vram?
Jis t hat correct
HIf you load a full 64K prompt at 50t/s pp it would take 21 mins
HBut currently testing how fast it is on nvidia, the 50 was on AMD
Ayeah hahahaah! That's why I want to try the quantized one by unsloth team ( the 280gb size one ).
Ji guess $1.5 nots going to run it (the full one)
HWhich quant is that?
J
