Crawlee+PlaywrightCrawler+proxy - original IP leaking through WebRTC
I'm running this simple program from a server in German datacenter with IP 167.235...
This program uses US residential proxies (rotating every 1min).
And I see that pixelscan.net is able to detect my original IP: 167.235...
On the attached screenshot you can find it under "WebRTC address"
So how to avoid this?
P.S:
and another problem I see - "Plugins Length", it is discussed here https://discord.com/channels/801163717915574323/1059483872271798333
15 Replies
unwilling-turquoise•3y ago
What type of proxy are you using? Is it an 'Elite' proxy?
sunny-greenOP•3y ago
well, I do not know what an "Elite proxy" is :-)
i am using proxies from smartproxy.com
these are "residential", "rotating" proxies.
The US proxies availiable under "us.smartproxy.com:<port>" (they have proxies for almost every country and the global pool)
I think for $12.5 per GB, "pay as you go" - they are not bad.
Anyway, I am open to alternatives
P.S. lets discuss different proxy services here: https://discord.com/channels/801163717915574323/1060179502392684594
harsh-harlequin•3y ago
Apify has the same price for residentials and they aggregate multiple providers, you can try on free plan. Will check the original issue
@petrpatek. Check the Web RTC leak
sunny-greenOP•3y ago
by the way, this: https://browserleaks.com/webrtc might be helpfull.
and, in the Firefox/Linux I use every day for browsing - all fields on that page are either "n/a" or "false" - and I do not even see that green START button (twicked my Firefox to prevent fingerprinting + "Privacy Badger" extension + "Canvas Blocker" extension)
I think all features related to WebRTC could be just disabled
BrowserLeaks
WebRTC Leak Test
IP address detection using JavaScript. Starting work on WebRTC API, the web browser communicates with the STUN server and shares information about local and public IP addresses even if you are behind NAT and use a VPN or Proxy. This tool will show if your real public IP address is leaking out.
sunny-greenOP•3y ago
To disable WebRTC:
media.peerconnection.enabled (WebRTC) = false
Taken from this article
https://restoreprivacy.com/firefox-privacy/
I think, it make sense to change configuration of headless Firefox. To set privacy.resistFingerprinting and some other flags described there.
I did it in my desktop browser - no WebRTC leak more.
So, who knows how to apply this to headless Firefox?
UPDATE:
https://stackoverflow.com/questions/53017490/how-can-i-disable-webrtc-local-ip-leak-with-puppeteer
looks like solution for Puppeteer...Sven Taylor
Restore Privacy
Firefox Privacy - The Complete How-To Guide for 2023
This new and updated Firefox privacy guide shows you how to tweak and configure Firefox for maximum privacy and security in 2023.
Stack Overflow
How can I disable webRTC local IP leak with puppeteer?
I tried:
const browser = await puppeteer.launch({args: ['--enable-webrtc-stun-origin=false', '--enforce-webrtc-ip-permission-check=false']});
But this is not working.
Next I tried:
const target...
fair-rose•3y ago
I believe that Firefox and Chrome have this setting disabled by default. I remember this was a huge security issue because you could access the IP addresses of other devices on the same network as the computer. I tested it on my browsers and have no leaks when using VPN or proxy. This might be because your proxy is sending x-forwarded headers. Let me know if you have a repro. I am curious to find out what is happening.
sunny-greenOP•3y ago
Firefox and Chrome have this setting disabled by default.this is not true for Firefox! I changed the
media.peerconnection.enabled according to above-mentioned article -- these settings was enabled by default.
This might be because your proxy is sending x-forwarded headers.Who knows what the hell my proxy is doing... @petrpatek. - how to check for x-forwarded headers? Just make request through my proxy to this address https://httpbin.org/get and look fo "x-forwarded" ?
fair-rose•3y ago
What is your Firefox version?
Yes the httpbin should work.
sunny-greenOP•3y ago
firefox used to run these tests:
/home/<my-user-name>/.cache/ms-playwright/firefox-1357/
(thats on a machine in datacenter)
firefox I use on desktop (to read this forum for example ) is: 108.02 (64bit)
sunny-greenOP•3y ago
this screenshot made in headless mode, httpbin.org/get was accessed through proxy.
and - the same program accessing pixelscan.net - it shows the WebRTC leak
fair-rose•3y ago
This is interesting. I believe that this is connected to the DC proxy. However, you are right that this leak should be prevented in the crawlee. I will test it and process this change. Thanks a lot for bringing this up!
So the only way how to do it programmatically (http://kb.mozillazine.org/User.js_file). It is necessary to create a user.js file and add it to your docker build flow.
sunny-greenOP•3y ago
the only way how to do it programmatically (http://kb.mozillazine.org/User.js_file).Well, in the firefox I use for browsing every day - I have no WebRTC leak and I do not use additional User.js ! I just changed the
media.peerconnection.enabled and few other settings, thats it.
Is it possible to pass this and few other settings to Firefox used by Playwright/Crawlee?
Some command-line flags?
...to create a user.js file and add it to your docker build flow.I do not using Docker at all. I just installed Crawlee on a machine in datacener and thats it I see that Firefox is here: ~/.cache/ms-playwright/firefox-1357/ Is there any news on this (WebRTC detection/leak) ?
GitHub
feat: overwrite WebRTC APIs with a recursive ES6 proxy by barjin · ...
While this solution is a bit crude, it seems to work in 100% of all cases.
From what I found, it doesn't even trigger scripts inspecting properties of Web API objects (but it also might be that...
sunny-greenOP•3y ago
Great news! So, in the next release this will be availiable, right?
It is not merged yet, so not sure it would be part of next release, if you need it now, you can pull the got branch and build and use it locally.