CA
Crawlee & Apify•3y ago
metropolitan-bronze

How to modify Start URLs dynamically in Apify Web Scraper (V3)

Hi from Melbourne, Australia. This is my first post.. I'm using the Apify Web Scraper (V3) to crawl a list of Start URLs - And I need to modify some of those URLs on the fly and append URL query paramaters to them. i.e. from 'starturl.com' to 'starturl.com?modified=onthefly' I searched through the docs and thinking it may involve the request object inside preNavigationHooks function, but I'm quite new to the platform and only using it in a basic way so not really sure where to start.. Any suggestions or examples would be much appreciated. Cheers! #apify-platform
1 Reply
dependent-tan
dependent-tan•3y ago
Yes, you should be able to change the URL with the pre navigation hooks - 
[
    async (crawlingContext, gotoOptions) => {
        crawlingContext.request.url = crawlingContext.request.url + '?modified=true'
    },
]
[
    async (crawlingContext, gotoOptions) => {
        crawlingContext.request.url = crawlingContext.request.url + '?modified=true'
    },
]
You can also enqueu more links dynamically with context.enqueueLinks

Did you find this page helpful?