Firecrawl•2mo ago

🤔 There's a Cookie popup in my scrape

Hello. Question: Is there a Firecrawl config setting I can use to prevent popups from getting into my scrapes? All of a sudden to today I am getting "cookie popup content" when I scrape documentation files. I would like to prevent this if possible. Attached: - Screenshot of the cookie and site scraped - Scraped file that now has the "cookie info" - lines [1-44] were added today, previous scrapes it wasn't there. Here is my config (Firecrawl Python SDK):

    result = firecrawl.scrape(
        url,
        formats=["markdown"],
        only_main_content=True,  # Excl. nav menu, footer, sidebars, etc.
        remove_base64_images=True,  # Removes base64 strings (keeps alt text)
        wait_for=3000,  # Wait to capture dynamic content (3 seconds)
        max_age=86400000,  # Use cached content for speed (24 hours)
    )

    result = firecrawl.scrape(
        url,
        formats=["markdown"],
        only_main_content=True,  # Excl. nav menu, footer, sidebars, etc.
        remove_base64_images=True,  # Removes base64 strings (keeps alt text)
        wait_for=3000,  # Wait to capture dynamic content (3 seconds)
        max_age=86400000,  # Use cached content for speed (24 hours)
    )

Thanks in advance.

ai-sdk.md

2 Replies

Gaurav Chadha•2mo ago

Oh, so for this, I think you're going to want to use custom actions (e.g. click, wait). This will instruct Firecrawl how to interact with the webpage before the page is scraped. This gives you a lot of control when the default behavior doesn't work as expected on a particular site!

MichelleOP•2mo ago

Hello @micah.stairs - appologies for the belated "thank you" - I've just seen your post now. Thank you, I will give it a try!

Gaming

Programming

🤔 There's a Cookie popup in my scrape

Did you find this page helpful?