Does the TOS allow scraping websites?

“crawls,” “scrapes,” or “spiders” any page, data, or portion of or relating to the Services or Content (through use of manual or automated means);

Does this part of the TOS apply to apps that scrape sites and/or people scraping railway's site?

Curious about scraping websites that don't have an API. Don't want to start working on something if it's not allowed by railway.

its more of a grey area, and railway would just prefer you didn't, so I would personally avoid it

Okay. What if a site allows scraping? Guessing that’s where they grey area is?

if it explicitly allows scrapping then yes that would be fine

what knid of website just says "pls scrape"

I know a fair few people who have personal sites all of which are fine scraping.

whats there to scrape 🤔

blog articles, etc. all sorts of different things.

mm ig

theres a difference between "fine to scrape" and "explicitly allows scrapping"

i think they meant that its like the website says "u can scrape if u want"

guessing railway also needs to keep within the laws of the country theyre hosting in and operating in. scraping isnt clearly in law in a lot of places.

hence why i asked

if its not in the law isnt it legal by default

you'd be surprised how different countries treat that.

fine im in the usa

I mean, even if the site says you can scrape them, I'd just avoid it anyway

what if it says "pls scrape or else"

For example BOM the aussie weather site, allowed scraping for years before their API came out.

that was the official way

and theyre even run by the government lol

web scraping just sounds like a bad idea regardless, unless you have some guarantee that the sites structure won't ever change

true, for something like BOM for example it never changed so simple as set it and forget it.

well if you want to scrape a site, and the site says somewhere that it fully allows scrapping, and you are sure the structure won't change, go for it

Also this was fun for learning to build a search engine in school http://www.scrapethissite.com/

Be a good web scraping citizen

the sites owner, is a fellow Brody

sounds like a good guy

any way we can get an official response on this from a staff member? there's a few features I'd like to add to a bot but they require me to scrape a page.

also would connecting to another service that DOES allow this to grab data would that also still be considered under this. 🤔

like railway -> middle server where it's allowed -> site

@Angelo - care to throw a quick answer at this?

you are good here

the DCMA/Fair Use is there to prevent some person just hosting a whole ass movie on the platform

figured that would be case but wanted to check

which people have attempted to do before

and get 🔨

sweet, thanks for the clarification

im also likely going to include abuse headers so if a site im scraping wants me to go away they have way to contact me.
same goes for using a clear user agent. marking my bot as a bot

should make everyone happy

appreciate it

🙂