3

I have created a web crawler from a tutorial and the website in the video seems to be down.

Would I be allowed to test it here at Ask Ubuntu, just solely for educational purposes?

If not, does anyone know a website that I could test it out on legally?

4
  • Thanks Sergiy Kolodyazhnyy
    – plutesci
    Commented Jul 9, 2018 at 4:36
  • There are the nmap sites made available to test nmap or any other port scanner, the machine has an http server so you can test there and nobody will sue you.. scanme.nmap.org
    – Yvain
    Commented Jul 9, 2018 at 4:50
  • @Yvain Port scanning is different from web crawling. Port scanning is figuring out what network services run on some computer out there. Web crawling is like clicking on all links on the website, except with a bot.
    – Sergiy Kolodyazhnyy
    Commented Jul 9, 2018 at 4:55
  • See meta.stackexchange.com/q/443/270345
    – muru
    Commented Jul 9, 2018 at 7:16

1 Answer 1

Reset to default
9

If you want actual permission from Stack Exchange Inc, use the contact form.

We can really only give you informal advice. Be polite. Don't hammer the server with hundreds of requests per second, don't leave it going for hours. Consider whether or not you need to index the whole thing or if you could limit it to something specific.

And if you're sharing the result of your scrape (eg you're setting up your own mirror), make sure you follow the copyright requirements for the content. The site designs are explicitly ⓒ Stack Overflow so copying them without permission is a no-no.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .