F
Firecrawl•9mo ago
Frnz

Need Help: Scrape Actions doesn't seem to work / apply

I'm trying to use scroll actions to navigate through a website (Kickstarter), but they aren't working at all. I confirmed this by taking a screenshot, which shows that the page remains at the top even after multiple scroll-down actions. To rule out a site-specific issue, I tested other websites, but the problem persists. Clicking and text input actions also don't work. However, I do receive an error when a clickable element isn't found, which suggests that my syntax is correct and that the checks are being applied. Additionally, the example actions from the documentation doesn't work out of the box ( did the website change? ): https://docs.firecrawl.dev/launch-week#example Here is the code I used: 
from firecrawl import FirecrawlApp

# Initialize the app
app = FirecrawlApp(api_key = "fc-XXXXX")


# URL to scrape
url = 'https://www.kickstarter.com/discover/advanced?category_id=35&sort=popularity&page=1'


# Define the actions to be performed
actions = [
    {"type": "wait", "selector": ".load_more"},
    {"type": "scroll", "direction": "down", "amount": 2000 },
    {"type": "click", "selector": ".load_more"},
    {"type": "scroll", "direction": "down", "amount": 2000 },
    {"type": "screenshot", "fullPage": True},
    {"type": "scrape"}
]


# Perform actions and scrape
result = app.scrape_url(url, 
    params={
        'formats': ['html', 'screenshot'],
        'actions': actions,
        'timeout': 999999
    })

# show screenshot
print(result['screenshot'])
from firecrawl import FirecrawlApp

# Initialize the app
app = FirecrawlApp(api_key = "fc-XXXXX")


# URL to scrape
url = 'https://www.kickstarter.com/discover/advanced?category_id=35&sort=popularity&page=1'


# Define the actions to be performed
actions = [
    {"type": "wait", "selector": ".load_more"},
    {"type": "scroll", "direction": "down", "amount": 2000 },
    {"type": "click", "selector": ".load_more"},
    {"type": "scroll", "direction": "down", "amount": 2000 },
    {"type": "screenshot", "fullPage": True},
    {"type": "scrape"}
]


# Perform actions and scrape
result = app.scrape_url(url, 
    params={
        'formats': ['html', 'screenshot'],
        'actions': actions,
        'timeout': 999999
    })

# show screenshot
print(result['screenshot'])
The expected screenshot and html should contain more than 20 games, since we scrolled down and pressed on the "load more" button, but the screenshot shows only the top side of the page, and the html contains only the first 13 games.
7 Replies
Frnz
FrnzOP•9mo ago
PD Feedback for the Docs: I would honestly love more information on the docs about the actual values of any parameter used by the library, for example, I didn't knew at first that result['screenshot'] was just an url to the actual screenshot, and when i asked for an example on the doc's LLM, it hallucinated and used base64 decode on the field. This has happened multiple times with different options and parameters. @Moderator Sorry for the ping, I saw that most of the threads are stale, but mods have helped recently with different issues in the general chat, and i just need some help with this, if you see anything wrong with the previous code snippet.
mogery
mogery•9mo ago
for the screenshot of the actions, you'll want result.actions.screenshots[0]
Frnz
FrnzOP•9mo ago
Would the same apply for the scrapping results?
mogery
mogery•9mo ago
yeah but you do not need those actions tbh the screenshot and the scrape is already done after the actions
Frnz
FrnzOP•9mo ago
I was having issues trying to obtain the resulting html after applying the actions from result["html"] I'll try this asap, and i'll let you know, thank you :) I just get a no attribute error for the actions: AttributeError: 'dict' object has no attribute 'actions' using the same code as above
mogery
mogery•9mo ago
did you remove the screenshot action too?
Frnz
FrnzOP•9mo ago
I didn't remove anything, its the same actions as shown above, should i remove the screenshot and scrape actions? I just tried that and i get the same error

Did you find this page helpful?