Crawlee & Apify•2y ago

Keyvaluestore file extensions

Hi, How do I configure Keyvalue store to have the .mhtml file extension? Using the code below seems to always set it to .bin extension

await KeyValueStore.setValue("some-name", data, {
  contentType: `application/x-mimearchive` \\ multipart/related doesn't work either,
});

// data above is either text or buffer

await KeyValueStore.setValue("some-name", data, {
  contentType: `application/x-mimearchive` \\ multipart/related doesn't work either,
});

// data above is either text or buffer

The goal is to have a file written to KV similar to some-name.mhtml Thanks!

12 Replies

conscious-sapphireOP•2y ago

Hi @Andrey Bykov can you please assist me with this? 🙂

eastern-cyan•2y ago

have you tried using multipart/related as mimetype?

conscious-sapphireOP•2y ago

Yes I did, it saved the file as some-name.bin which is really weird

MEE6•2y ago

@microworlds just advanced to level 3! Thanks for your contributions! 🎉

eastern-cyan•2y ago

oh sorry missed the comment above could you send the whole minimum repro?

conscious-sapphireOP•2y ago

Sure!

eastern-cyan•2y ago

(I mean the page url, how you grab the content, etc)

conscious-sapphireOP•2y ago

// Inside the requestHandler 
const cdpSession = await page.target().createCDPSession();
      await cdpSession.send('Page.enable');
      const { data } = await cdpSession.send('Page.captureSnapshot', { format: 'mhtml' });

      const key = `file-${Math.floor(
        Math.random() * 0xffffff
      ).toString(16)}.mhtml`;
      await KeyValueStore.setValue(key, data, {
        contentType: `multipart/related`,
      });

// Inside the requestHandler 
const cdpSession = await page.target().createCDPSession();
      await cdpSession.send('Page.enable');
      const { data } = await cdpSession.send('Page.captureSnapshot', { format: 'mhtml' });

      const key = `file-${Math.floor(
        Math.random() * 0xffffff
      ).toString(16)}.mhtml`;
      await KeyValueStore.setValue(key, data, {
        contentType: `multipart/related`,
      });

@Andrey Bykov URL: http://example.com/

eastern-cyan•2y ago

I'd say immediately you can't do anything about it. If I remember correctly - mime-types and content-type packages are used by crawlee, and .mht/.mhtml do not return any matching mime type, therefore when you provide this miime-type - it falls back to .bin. I'll pass it to the team, but would not expect that it will be changed/updated straight away

conscious-sapphireOP•2y ago

Thanks for your help! 😄

eastern-cyan•2y ago

I just tried to run it through above packages, and indeed it does not return any mime-type. So it basically needs to be added to the packages itself... But it someone from the team will come with some solution - I will write back

conscious-sapphireOP•2y ago

Alright, thanks !

Gaming

Programming

Keyvaluestore file extensions

Did you find this page helpful?