'Background' options for Pod Initiated file transfer
I'm trying to scope out if there's a solution to have a runpod send me back a small .db/txt file on completion of task, or of progress before closing due to being outbid and closed (Community pods)
I've been looking at rsync, runpodctl, SSH, and they all seem to require transfer to be 'initiated' from the recipient machine
I'm looking at the google drive API, which I think is going to be my best bet for an 'always ready to receive' solution.
Anyone else already have experience working around something similar?
I've been looking at rsync, runpodctl, SSH, and they all seem to require transfer to be 'initiated' from the recipient machine
I'm looking at the google drive API, which I think is going to be my best bet for an 'always ready to receive' solution.
Anyone else already have experience working around something similar?
Solution
You might need something like this, detect the signal and do something:
import signal
import boto3
import os
from time import sleep
S3_BUCKET = 'your-s3-bucket-name'
FILE_TO_UPLOAD = '/path/to/your/file.txt'
S3_KEY = 'uploaded-file.txt'
def upload_to_s3():
s3_client = boto3.client('s3')
try:
s3_client.upload_file(FILE_TO_UPLOAD, S3_BUCKET, S3_KEY)
print(f"File {FILE_TO_UPLOAD} uploaded to S3 bucket {S3_BUCKET} as {S3_KEY}")
except Exception as e:
print(f"Failed to upload file: {e}")
def handle_signal(signal_number, frame):
print(f"Received signal {signal_number}. Saving file to S3.")
upload_to_s3()
exit(0)
signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)
print("Running... Press Ctrl+C to exit.")
try:
while True:
sleep(1)
except KeyboardInterrupt:
handle_signal(signal.SIGINT, None)
import signal
import boto3
import os
from time import sleep
S3_BUCKET = 'your-s3-bucket-name'
FILE_TO_UPLOAD = '/path/to/your/file.txt'
S3_KEY = 'uploaded-file.txt'
def upload_to_s3():
s3_client = boto3.client('s3')
try:
s3_client.upload_file(FILE_TO_UPLOAD, S3_BUCKET, S3_KEY)
print(f"File {FILE_TO_UPLOAD} uploaded to S3 bucket {S3_BUCKET} as {S3_KEY}")
except Exception as e:
print(f"Failed to upload file: {e}")
def handle_signal(signal_number, frame):
print(f"Received signal {signal_number}. Saving file to S3.")
upload_to_s3()
exit(0)
signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)
print("Running... Press Ctrl+C to exit.")
try:
while True:
sleep(1)
except KeyboardInterrupt:
handle_signal(signal.SIGINT, None)
