Runpod•2y ago

How to authenticate with Google Cloud in a Docker container running Serverless?

I'm trying to authenticate using a service account json key file running a Docker container so I can store objects to GCS. I've added the json file's content as a Secret but without success. Am I missing something here or how would you advise me to authenticate? Update: Looks like the entire json file's content doesn't fit as a Secret which explains why it doesn't work. Still, I'd like to find a way to authenticate.

16 Replies

zkreutzjanz•2y ago

I've had the same problem. Have not found a safe workaround yet

Unknown User•2y ago

Message Not Public

sallok00OP•2y ago

Thanks @nerdylive. I'm using GOOGLE_APPLICATION_CREDENTIALS the way you suggest during local development. However, for deployment this would require me to keep the JSON file inside the Docker container which is not great practice from a security perspective. This is how I read the credentials and upload the objects:

from google.cloud.storage import Client, transfer_manager
GOOGLE_APPLICATION_CREDENTIALS = os.environ.get('GOOGLE_APPLICATION_CREDENTIALS', None)

storage_client = Client.from_service_account_json(
    json_credentials_path=GOOGLE_APPLICATION_CREDENTIALS,
)

# Upload the files
transfer_manager.upload_many(
    file_blob_pairs,
    worker_type="thread",
    max_workers=workers,
)

from google.cloud.storage import Client, transfer_manager
GOOGLE_APPLICATION_CREDENTIALS = os.environ.get('GOOGLE_APPLICATION_CREDENTIALS', None)

storage_client = Client.from_service_account_json(
    json_credentials_path=GOOGLE_APPLICATION_CREDENTIALS,
)

# Upload the files
transfer_manager.upload_many(
    file_blob_pairs,
    worker_type="thread",
    max_workers=workers,
)

@zacksparrow would it be possible to extend the Secret length to 4000 characters similar to the Registry Credentials? That way I would be able to read the credentials directly as follows:

storage_client = Client.from_service_account_info(
    info=GOOGLE_APPLICATION_CREDENTIALS,
)

storage_client = Client.from_service_account_info(
    info=GOOGLE_APPLICATION_CREDENTIALS,
)

Unknown User•2y ago

Message Not Public

sallok00OP•2y ago

I'm trying the service account approach as I explained above but it's not working. Workload identity federation seem a little bit trickier (and it's not clear whether that key is shorter) so I think I'll just wait to see if they can extend the Secret length to 4k instead - it seems like a potentially easier solution 🙂

Unknown User•2y ago

Message Not Public

sallok00OP•2y ago

Friendly ping @flash-singh, is this something you could help out with? Similar to discussed here: https://discord.com/channels/912829806415085598/1195092101671690260/1195092101671690260

flash-singh•2y ago

thanks will bring this up, should be easy, does it let you save the secret or does it save the secret and truncates some parts?

zkreutzjanz•2y ago

truncates lets you save but value is truncated when you retrieve

MuddyRumbles•2y ago

I save a base64 encoded json credentials file (ala service-account.json) in a secret (previously env var). I opened a ticket a while back to have the limit increased to support my use case. You can look for and decode the file in your entrypoint. If you have questions or anything please @ me because I start work soon and might not see your messages.

def get_service_account_credentials(file="service-account.json"):
    """Get service account from base64 encoded environment variable, if a local file is not found"""
    if os.path.exists(file):
        # This local file will be used by the google client library
        logger.info("Local service account file found, using it for authentication.")
        return
    else:
        # Get the service account from the base64 encoded environment variable
        encoded_service_account = os.environ.get("SERVICE_ACCOUNT_JSON", None)
        if encoded_service_account is None:
            raise ValueError("SERVICE_ACCOUNT_JSON environment variable not set")
        else:
            # Write the service account to the local file for use by the google client library
            logger.info("Writing service account to local file for authentication.")
            with open(file, "w") as f:
                f.write(base64.b64decode(encoded_service_account).decode("utf-8"))

def get_service_account_credentials(file="service-account.json"):
    """Get service account from base64 encoded environment variable, if a local file is not found"""
    if os.path.exists(file):
        # This local file will be used by the google client library
        logger.info("Local service account file found, using it for authentication.")
        return
    else:
        # Get the service account from the base64 encoded environment variable
        encoded_service_account = os.environ.get("SERVICE_ACCOUNT_JSON", None)
        if encoded_service_account is None:
            raise ValueError("SERVICE_ACCOUNT_JSON environment variable not set")
        else:
            # Write the service account to the local file for use by the google client library
            logger.info("Writing service account to local file for authentication.")
            with open(file, "w") as f:
                f.write(base64.b64decode(encoded_service_account).decode("utf-8"))

This code snippet might help. Its easy to accomplish in any language. Could be done in straight bash easily. @sallok00 Let me know if this helps

sallok00OP•2y ago

Thanks @MuddyRumbles, and sorry for my slow reply. Looks like I somehow missed your ping. I've also tried to save it as a secret but it gets truncated. For now I'm doing something similar to what you do above - saving the .json file in the container and read it directly. This is not ideal on longer term though due to the security risk.

zkreutzjanz•2y ago

The truncation is fixed in runpod secrets

Unknown User•2y ago

Message Not Public

zkreutzjanz•2y ago

Not sure, my gcp service creds fit now though

flash-singh•2y ago

we have increased the field, it was a mysql thing and kept truncating the data

sallok00OP•2y ago

Amazing - thanks a lot!

Gaming

Programming

How to authenticate with Google Cloud in a Docker container running Serverless?

Did you find this page helpful?