[mrki444]

Web crawlers are tools for scraping data from other websites. It is often used now since Big Data works with data and make new conclusions from it. Example, crawlers, are used on sites which compare products prices.

Crawlers are often in gray legal zone. Sometimes sites' owners allow crawlers scrap data (provide API key). In that case it is not a problem and everything is good with law.

Often crawlers collect data without site owner's permission and that can be illegal. Some say it is illegal since you didn't ask the owner for permission. Shame it is not because the data are public and somebody can manually copy paste it, but crawler just do that automatically.

Does anybody do anything with crawlers? Do you make it and web site owners do you block it?

[CyberFreak]

To be honest, I don't mind too much about them aslong as they are considerate. What I have seen alot of here is alot of Bots from different IP's requesting alot of pages between them or just one IP address scraping alot of pages per second. These are the problematic ones and the ones I outright block. Bots belonging to major search engines usually are respectful and don't cause issues. It is mainly just the more obscure search engines or SEO/Analytics type bots/companies that are problematic in my experience.

[mrki444]

True. search engines are biggest web crawlers. Often people don't think bot and crawler are same thing but they are.

I assume Forumcoin faucet had many problems with bots?

[GuestHu]

True. search engines are biggest web crawlers.

Well, there are search engines that are not web crawlers like duckduckgo. But, it's sad that the other search engines especially Google practice this illegal act and despite that, people still use that site. Maybe, they do not find it dangerous and creepy but rather convenient.

[mrki444]

Well, there are search engines that are not web crawlers like duckduckgo.

Than how DuckDuckgo search sites and rank it? It must somehow read your content and give it score, rank it, check keywords. All search engines are in essential bots or crawlers.

But, it's sad that the other search engines especially Google practice this illegal act and despite that, people still use that site.

If you think this is illegal practice, than deindex your sites from Google search engine and you will not get be ranked on Google or be crawled.