Crawlee & Apify•14mo ago

Download images and store in apify

Hi, I read the documentation on keyvalue pair storage, but did not quite really fiure out how to use it to store images, videos or audio using python sdk. Is there any example code that I can reference to?

6 Replies

MEE6•14mo ago

@sabin9848 just advanced to level 1! Thanks for your contributions! 🎉

Mantisus•14mo ago

Hey You may want to look at this discussion. https://discord.com/channels/801163717915574323/1202619785884467231 And so I've prepared a little sample code

async def download_image(url):
    response = await client.get(url)
    key_store = await Actor.open_key_value_store(name='images')
    await key_store.set_value("test", response.content)
    image = await key_store.get_value("test")
    with open("test.jpg", "wb") as f:
        f.write(image)

async def download_image(url):
    response = await client.get(url)
    key_store = await Actor.open_key_value_store(name='images')
    await key_store.set_value("test", response.content)
    image = await key_store.get_value("test")
    with open("test.jpg", "wb") as f:
        f.write(image)

afraid-scarletOP•13mo ago

thank you for the help, I figured it out. Hello again, I successfully stored image in apify using key value storage, and when I tried to store audio, by sending its base64 value, when I go to the public url of the keyvalue it is not playing https://api.apify.com/v2/key-value-stores/2E6MJPpFo0qpUEC3v/records/test This is the code: async def upload_to_apify(self, value): key_store = await Actor.open_key_value_store(name='audio') await key_store.set_value('test',value, content_type='audio/mpeg') public_url = await key_store._get_public_url_internal('test')
print(public_url)

Mantisus•13mo ago

Hey ) But you encoded it as base64 and specified the data type as audio/mpeg... If you download your audio and decode Base64 and save it to a file, it will play correctly.

import requests
import base64 

response = requests.get("https://api.apify.com/v2/key-value-stores/2E6MJPpFo0qpUEC3v/records/test")

decoded_bytes = base64.b64decode(response.text)

with open("test.mpeg", "wb") as f:
    f.write(decoded_bytes)

import requests
import base64 

response = requests.get("https://api.apify.com/v2/key-value-stores/2E6MJPpFo0qpUEC3v/records/test")

decoded_bytes = base64.b64decode(response.text)

with open("test.mpeg", "wb") as f:
    f.write(decoded_bytes)

afraid-scarletOP•13mo ago

oh I thought we need to send base64 value, but now I sent the whole audio file and it is working properly. Thank you.

absent-sapphire•13mo ago

its your own choice: https://docs.apify.com/sdk/python/reference/class/KeyValueStore#set_value and "content_type" any of https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types just make sure that content really fits type

Gaming

Programming

Download images and store in apify

Did you find this page helpful?