Crawlee & Apify•3y ago

PlaywrightCrawler - how often browser fingerprints are changed?

Are browser fingerprints changed - every request? - every 1 min? - every... I do not what else )) And how changing browser fingerprints related to using or not using PlaywrightCrawler.launchContext.useIncognitoPages ? I am asking this because I saw a situation when two attempts to open a bot detection site https://fingerprint.com/demo/ result in same "ID" - in other words they were able to identify me! Screenshots attached. Interval between requests - 3 min. Different IPs (from the pool of "rotating" IP's). Without incognito

Technical Demo - Fingerprint Pro

Fingerprint Pro is the 99.5% accurate device fingerprinting solution. FingerprintJS is the top open-source browser fingerprinting library. Prevent fraud, spam, and account takeovers. Available for web, iOS, and Android.

6 Replies

yappiest-sapphire•3y ago

Following

exotic-emeraldOP•3y ago

I used the site [1] to test PlaywrightCrawler with different setting. Here what I saw: 1. Without incognito mode Without incognito mode - as described above - the site [1] assigns the same ID even when different IP's are used. 2. Incognito mode turned on

    launchContext: {
        experimentalContainers: false,
        useIncognitoPages: true,
        launcher: firefox,
    },

    launchContext: {
        experimentalContainers: false,
        useIncognitoPages: true,
        launcher: firefox,
    },

With incognito mode turned on the site [1] can not detect Crawlee - it assigns a different ID for every request. 3. With experimentalContainers turned on:

    launchContext: {
        experimentalContainers: true,
        useIncognitoPages: false,
        launcher: firefox,
    },

    launchContext: {
        experimentalContainers: true,
        useIncognitoPages: false,
        launcher: firefox,
    },

the site [1] assign the same ID even when different IP's are used. In all cases - no session pool and persistent cookies:

    useSessionPool: false,
    persistCookiesPerSession: false,

    useSessionPool: false,
    persistCookiesPerSession: false,

[1] https://fingerprint.com/demo/ So is it "as designed" or my test is wrong or something else?

Technical Demo - Fingerprint Pro

equal-aqua•3y ago

Fingerprint and proxy IP should be attached to a browser. Depending on your browserPoolOptions, it will open some number of pages per browser. By default it is 100 requests per browser I think. Those should have same fingerprint and proxy IP. Each new browser should attach to different IP and fingerprint. Incognito and experimental containers should on the other hand use different fingerprint/IP for each request. But of course, there might be bugs Thanks for all the testing

exotic-emeraldOP•3y ago

Incognito and experimental containers should on the other hand use different fingerprint/IP for each request.

This seems to be true for incognito - different ID (on fingerprint.com/demo/) assigned every time And something is wrong in case of experimental containers - same ID all the time!

exotic-emeraldOP•3y ago

I did additional tests with bot detection/fingerprinting sites. Used PlaywrightCrawler with this configuration:

    launchContext: {
        experimentalContainers: false,
        useIncognitoPages: true,
        launcher: firefox,
    },

    launchContext: {
        experimentalContainers: false,
        useIncognitoPages: true,
        launcher: firefox,
    },

No session pool and persistent cookies. PlaywrightCrawler.browserPoolOptions.useFingerprints = true - Test results with [1] are same as before - it assigns a different ID for every request. Fine. - Test results with [2] and [3] - SAME ID FOR EVERY REQUEST. The IP was different with each request. On the screenshot from site [2] -attached below- look at "FP ID" on the top and on "visits". "Visits: 4" - it means this site knows - it is my visit number 4. Screenshots from site [3] are self-explaining: huge "Your fingerpring:" in the middle. Fingerprint remains the same when I visit it from different IP. Not a good news. So, what can be done against it? The FingerprintGenerator is there, how it can be used against [2] and [3] ??? [1] https://fingerprint.com/demo/ [2] https://abrahamjuliot.github.io/creepjs/ [3] https://noscriptfingerprint.com/

Technical Demo - Fingerprint Pro

No-JS fingerprinting

This demo illustrates that fingerprinting is possible even without JavaScript and cookies

exotic-emeraldOP•3y ago

we could use the following trick (to fool bot detection [2] and [3]): Use the prePageCreateHooks and set randomized pageOptions there. I already doing this for locale+timezoneId, and it works:

    browserPoolOptions: {
   
        prePageCreateHooks: [
            (pageId, browserController, pageOptions) => {
                pageOptions.locale      = myLocaleSingle;
                pageOptions.timezoneId  = myTimezone;
            },
        ],

    browserPoolOptions: {
   
        prePageCreateHooks: [
            (pageId, browserController, pageOptions) => {
                pageOptions.locale      = myLocaleSingle;
                pageOptions.timezoneId  = myTimezone;
            },
        ],

According to Playwright documentation https://playwright.dev/docs/api/class-browser#browser-new-page we can set these things (only the most interesting options listed here): colorScheme deviceScaleFactor extraHTTPHeaders ??? screen.width screen.height timezoneId locale userAgent !!! viewport.width viewport.height How to get these values from the Fingerprint Generator? (being in the PlaywrightCrawler) I would like to try, pls help...

Browser | Playwright

* extends: [EventEmitter]

Gaming

Programming

PlaywrightCrawler - how often browser fingerprints are changed?

Did you find this page helpful?