Crawlee & Apify•3y ago

Saving fingerprints and cookies in database

Hello there! Is it possible, to store fingerprints, cookies, etc in database? save it automatically and load when needed?

10 Replies

absent-sapphire•3y ago

Hi! You can use the FingerPrintGenerator package to generate fingerprints, then store them in a database for later usage

frozen-sapphireOP•3y ago

How can I set fingerprint manually for some request?

absent-sapphire•3y ago

https://www.npmjs.com/package/fingerprint-generator

npm

fingerprint-generator

NodeJS package for generating realistic browser fingerprints.. Latest version: 2.0.8, last published: 17 days ago. Start using fingerprint-generator in your project by running npm i fingerprint-generator. There are 5 other projects in the npm registry using fingerprint-generator.

frozen-sapphireOP•3y ago

I know this library🙂 I just asking about how to use fingerprint generator with browser pool, to set fingerprint to new request manually

absent-sapphire•3y ago

Check this out: Referring to this lesson in the Academy: https://developers.apify.com/academy/anti-scraping/mitigation/generating-fingerprints#injecting-fingerprints You can do this to inject custom fingerprints:

import { FingerprintGenerator } from 'fingerprint-generator';
import { FingerprintInjector } from 'fingerprint-injector';
import { PlaywrightCrawler } from 'crawlee';

const fingerprintInjector = new FingerprintInjector();

const fingerprintGenerator = new FingerprintGenerator({
    browsers: [{ name: 'firefox', minVersion: 80 }],
    devices: ['desktop'],
    operatingSystems: ['windows'],
});

const crawler = new PlaywrightCrawler({
    browserPoolOptions: {
        postLaunchHooks: [
            async (_, controller) => {
                const promises = controller.browser.contexts().map((context) => {
                    const generated = fingerprintGenerator.getFingerprint();

                    context.setExtraHTTPHeaders({
                        ...generated.headers,
                    });

                    return fingerprintInjector.attachFingerprintToPlaywright(context, generated);
                });

                await Promise.all(promises);
            },
        ],
    },
    requestHandler: () => {},
});

import { FingerprintGenerator } from 'fingerprint-generator';
import { FingerprintInjector } from 'fingerprint-injector';
import { PlaywrightCrawler } from 'crawlee';

const fingerprintInjector = new FingerprintInjector();

const fingerprintGenerator = new FingerprintGenerator({
    browsers: [{ name: 'firefox', minVersion: 80 }],
    devices: ['desktop'],
    operatingSystems: ['windows'],
});

const crawler = new PlaywrightCrawler({
    browserPoolOptions: {
        postLaunchHooks: [
            async (_, controller) => {
                const promises = controller.browser.contexts().map((context) => {
                    const generated = fingerprintGenerator.getFingerprint();

                    context.setExtraHTTPHeaders({
                        ...generated.headers,
                    });

                    return fingerprintInjector.attachFingerprintToPlaywright(context, generated);
                });

                await Promise.all(promises);
            },
        ],
    },
    requestHandler: () => {},
});

Apify

Generating fingerprints · Apify Developers

Learn how to use two super handy NPM libraries to easily generate fingerprints and inject them into a Playwright or Puppeteer page.

absent-sapphire•3y ago

With this configuration, each browser context will have a newly generated fingerprint

frozen-sapphireOP•3y ago

Got it. But how I can use non-random fingerprint? I mean, for each request for some profile (I.e. profile#123) use same fingerprint (I.e. fingerprint#123)? I want to generate Fingerprint once, save it and use it every time for same profile

absent-sapphire•3y ago

So you can generate one single fingerprint and attach the same one each time, it doesn’t have to be generated each time

frozen-sapphireOP•3y ago

Can I attach cookies and proxies this way?

absent-sapphire•3y ago

The object generated by generator.getFingerprint returns an object with two properties - headers, and fingerprint. You could extend this object and store more data on it such as proxy info etc. If you’d like Then store it for later usage, or do whatever you want with it

Gaming

Programming

Saving fingerprints and cookies in database

Did you find this page helpful?