R
RunPodACiDGRiM

secure connections

I want to ensure all traffic between my app and the server less backend is encrypted. Does the endpoint decrypt the traffic from the internet and transmit in plaintext to the server less container? Specifically, is the data in my prompt In clear text even in memory before it reaches the container?
Solution:
In theory you could make own worker that input would be encrypted file and it would be decoded on container itself though you would need make that code yourself
M
Madiator201143d ago
It’s mostly depends how you write your worker. All endpoint use https proxy so traffic is encrypted. Usually you send JSON object to API and that info is used to start the script for the app after job is done serverless do not store input information only the link for output. For full privacy you are advised to use own S3 bucket as otherwise you might get b64 encoded image.
A
ACiDGRiM43d ago
OK so the best way to obscure my prompt would be to encrypt the message in the api req "message" body and decrypt it in the container? It sounds like the app endpoint handles plaintext json, rather than transparently forwarding a request to the container, TLS intact. Or does the url endpoint determine which container to forward to? In other words, which TLS cert is seen by curl? Api.runpod.io or "my custom cert"
A
ashleyk43d ago
You can't use custom certs, it is always the RunPod cert
A
ACiDGRiM43d ago
OK I thought so. I want to get see embedding of all my personal documents. So I'm just trying to find a way to feel comfortable sending the text to another computer or the internet. I don't mind them being in ram for the inference, but I don't want them exposed between the api and the container. Maybe that's the server option, I'm just trying to save a buck
Solution
M
Madiator201143d ago
In theory you could make own worker that input would be encrypted file and it would be decoded on container itself though you would need make that code yourself
A
ashleyk43d ago
The serverless workers are all in secure cloud though and data transmitted over TLS so I don't see any issue.
A
ACiDGRiM42d ago
OK that's what I'm going to do, but just encrypt the payload. Traffic out of the Contrainer isn't proxied, so I can download the decryption key from my network? I'm sure your secure against most threats but if I'm not sending my documents to Google, I want to limit my exposure to other 3rd parties out of principal until I can afford an l40s at least If I don't control the keys when it's my private I fo it's not secure. You guys have good infra, but I have no idea who you are or who's server the worker is on I'll accept that my files being in vram for a few minutes is acceptable Please confirm this will work, I have a PoC of a feature to send an encrypted body to the run api endpoint, and then decrypt and pass to a typical pyTorch workload, and then encrypt the response and send it If I send the data you have listed in your sync and async endpoints documentation
curl -X POST https://api.runpod.ai/v2/{endpoint_id}/run \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer ${API_KEY}' \
-d '{"input": {"prompt": "Your prompt"}}
curl -X POST https://api.runpod.ai/v2/{endpoint_id}/run \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer ${API_KEY}' \
-d '{"input": {"prompt": "Your prompt"}}
but with the data/body
{"encrypted": "asdf3wqcm84wmt87v4e7mtasrhcrdgdc"}
{"encrypted": "asdf3wqcm84wmt87v4e7mtasrhcrdgdc"}
will the endpoint forward it to my endpoint ID as is, or do you sanitize for proper prompts? Also for the stream endpoint, the encrypted body will completely change after every returned token is encrypted, this should be transparent when streaming the decrypted original body responce, but I''m not sure if your api will handle that"
M
Madiator201142d ago
I mean you would need to adjust your worker code to handle decryption
A
ACiDGRiM41d ago
yes, I have a proof of concept that does this, I just want to make sure your api doesn't do any sanitization on the data, just passes it direclty to the worker
A
ashleyk41d ago
As long as its valid JSON it should be fine Also the body must be have:
{
"input": ...
}
{
"input": ...
}
Serverless doesn't work without input. So if you want to use encrypted as a key in the JSON, put it inside input.
A
ACiDGRiM41d ago
Thanks, that's the piece of info I wanted to know. I'll modify my promptProxy to work accordingly
Want results from more Discord servers?
Add your server
More Posts
server less capability checkI want to add runpod into a tier of load balanced llm models behind an app like openrouter.ai, but tGPU memory usage is at 99% when starting the task.I started to notice some GPU OOM failure today, and it's specific to this instance: A40 - 44adfw5inhShould i wait for the worker to pull my imageI have a large image (100 GB), should i wait for worker to pull the image before starting any inferePossible memory leak on ServerlessWe're testing different mistral models (cognitivecomputations/dolphin-2.6-mistral-7b and TheBloke/doare we able to run DinD image for GPU pods?Hi, anyone tried running DinD in GPU pods?Runpod error starting container2024-03-07T14:40:19Z error starting container: Error response from daemon: failed to create task forRunpod SD ComfyUI Template missing??Where did the "Runpod SD ComfyUI" template go? Can anyone help? I've been using it extensively for aDockerless dev and deploy, async handler need to use async ?handler.py in HelloWorld project, there is not 'async' before def handler(job): . But in serverlesSomething broken at 1am UTCSomething was broken at 1am UTC which caused a HUGE spike in my cold start and delay times.Should I use Data Centers or Network Volume when confige serverless endpoint ?My project is an AI portrait app targeting global users. The advantage of using data centers is the Pod OutageCurrently taking 100x longer to pull the docker image and when it eventually builds I have an API seAre stream endpoints not working?This is a temp endpoint just to show you all. /stream isn't available, what's up?Cuda - Out of Memory error when the 2nd GPU not utilizedI have a pod with 2 x 80 GB PCIe and I am trying to load and run Smaug-72B-v0.1 LLM. The problem is,Postman returns either 401 Unauthorized, or when the request can be sent it returns as Failed, errorPostman reads the following, when I send runsync request from runpod tutorial (from generativelabs) Backdrop Build V3 Credits missingHi team, I hope this message finds you well. I am writing to follow up on the recent offer I receivText-generation-inference on serverless endpointsHi, I don't have much experience neither with llms nor with python, so I always just use this image When on 4000 ADA, it's RANDOMLY NOT DETECTING GPU!When on 4000 ADA, it's RANDOMLY NOT DETECTING GPU! Yesterday I set it up and it's okay. Today I set Cold Start Time is too longWhen i test a HelloWorld project, run , it take too much time. Worker Configuration as attachment, IWhat happened to the webhook graph?There was a webhook graph for serverless but I can't seem to find it anymore. Was it removed for soHow i can use more than 30 workers?i've tested my task with 30 workers and realized that i need more) is it possible to get 40 or more?