Crawlee & Apify•3y ago

How can you find HTML element that can be clicked on to use dropdown?

I have managed to find element for some dropdowns but for a specific dropdown I cant find it. Is there any HTML attribute I can look at to determine which can be clicked? There does not seem to be a onClick() or similar

9 Replies

harsh-harlequin•3y ago

Got a link to the page and a screenshot of what you want to click?

fair-roseOP•3y ago

fair-roseOP•3y ago

https://www.europarl.europa.eu/RegistreWeb/search/simpleSearchHome.htm?languages=EN&sortAndOrder=DATE_DOCU_DESC

Document search page

Search by criteria (reference, author, date, etc.)

fair-roseOP•3y ago

I want to click on "FULL" dropdown Forgot to mention you need to click on the top element on the page before it is shown as displayed

harsh-harlequin•3y ago

There is page.selectOption(): https://playwright.dev/docs/api/class-page#page-select-option This selector seems to match all the dropdowns on the page that you want:

select[name*="FULL"]

select[name*="FULL"]

Since all the select elements are already PRESENT in the DOM (just not visible until the container is clicked), you can do:

page.selectOption('selector-here', 'NONE', { force: true });

page.selectOption('selector-here', 'NONE', { force: true });

force set to true means the element doesn't necessarily have to be visible (see more here: https://playwright.dev/docs/actionability) But since you want to click all of them, you'll need to loop through all elements that that selector resolves to and run the selecting logic for each one.

Page | Playwright

* extends: [EventEmitter]

harsh-harlequin•3y ago

Hope this helps 😄

fair-roseOP•3y ago

Thanks I will try that out. Propably better than just selecting each element and trying to click it 😛 How would the corresponding code look like in chrome dev tools?

harsh-harlequin•3y ago

So I was wrong about the page. The title MUST be clicked for the <select> to be added to the DOM. Try running this code:

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    headless: false,
    requestHandler: async ({ page }) => {
        // Wait for the requests for the data to be made
        await page.waitForLoadState('networkidle');

        const titles = await page.$$('h3[class*="results-item-title"]');

        // Click all title elements to display the dropdown on the page
        for (const title of titles) {
            await title.click();
        }

        const dropdowns = await page.$$('select[name*="FULL"]');

        // Choose the NONE option on all dropdowns
        for (const dropdown of dropdowns) {
            await dropdown.selectOption('NONE');
        }

        // give 20 seconds to check out the page
        await page.waitForTimeout(20e3);
    },
});

await crawler.run(['https://www.europarl.europa.eu/RegistreWeb/search/simpleSearchHome.htm?languages=EN&sortAndOrder=DATE_DOCU_DESC']);

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    headless: false,
    requestHandler: async ({ page }) => {
        // Wait for the requests for the data to be made
        await page.waitForLoadState('networkidle');

        const titles = await page.$$('h3[class*="results-item-title"]');

        // Click all title elements to display the dropdown on the page
        for (const title of titles) {
            await title.click();
        }

        const dropdowns = await page.$$('select[name*="FULL"]');

        // Choose the NONE option on all dropdowns
        for (const dropdown of dropdowns) {
            await dropdown.selectOption('NONE');
        }

        // give 20 seconds to check out the page
        await page.waitForTimeout(20e3);
    },
});

await crawler.run(['https://www.europarl.europa.eu/RegistreWeb/search/simpleSearchHome.htm?languages=EN&sortAndOrder=DATE_DOCU_DESC']);

fair-roseOP•3y ago

thanks I will test it soon

Gaming

Programming

How can you find HTML element that can be clicked on to use dropdown?

Did you find this page helpful?