CA
Crawlee & Apify•7mo ago
optimistic-gold

await a promise set in a pre navigation hook

Hi All, I have a pre navigation hook that listens for requests and if they return images saves them to the cloud 
  return async (context) => {
    if (context.request.label == requestLabels.article) {
      context.page.on('request', async (req) => {
        if (req.resourceType() == 'image') {
          const response = await req.response();
          // extra processing and save to cloud
  return async (context) => {
    if (context.request.label == requestLabels.article) {
      context.page.on('request', async (req) => {
        if (req.resourceType() == 'image') {
          const response = await req.response();
          // extra processing and save to cloud
This works in 95% of cases however there are some where the main request hander completes before each res.response() can be collected. This causes an error since the browser context closes with the main request handler. My question is how can i best get around this? One idea I had was that I could put a promise in the page userdata that resolves when the page has no outstanding images. After reading the docs however im not sure if this is possible since userdata needs to be serialisable? Has anyone else encountered this type of issue and how have they got around it?
3 Replies
Hall
Hall•7mo ago
Someone will reply to you shortly. In the meantime, this might help:
Pepa J
Pepa J•7mo ago
Hi @Crafty If I understand it correctly, the simplest solution would be to have global variable and match it with the request. 
const requestPromises = new Map();
const requestPromises = new Map();
// preNavHook
async (context) => {
   requestPromises.set(context.request.id, [ /* ... Promises ... */ ]);
}
// preNavHook
async (context) => {
   requestPromises.set(context.request.id, [ /* ... Promises ... */ ]);
}
...
requestHandler: async (context) => {
  // my routeHandler code

  await Promise.all(requestPromises.get(context.request.id));
  requestPromises.delete(context.request.id);
}
...
...
requestHandler: async (context) => {
  // my routeHandler code

  await Promise.all(requestPromises.get(context.request.id));
  requestPromises.delete(context.request.id);
}
...
Sometimes I see people passing their own variables to the context which is then passed to the routeHandler 
// preNavHook
async (context) => {
   context.myPromises = [ ... ];
}
// preNavHook
async (context) => {
   context.myPromises = [ ... ];
}
...
requestHandler: async (context) => {
  // my routeHandler code

  await Promise.all(context.myPromises);
}
...
...
requestHandler: async (context) => {
  // my routeHandler code

  await Promise.all(context.myPromises);
}
...
optimistic-gold
optimistic-goldOP•7mo ago
Hi, thanks for this. It should solve my problem nicely. 🙂

Did you find this page helpful?