2024-04-02: Google has changed its operations since the original publication of this post. Now it does not show (and, possibly, does not steal) pages excluded in robots.txt. It continues stealing the same content when re-published on other sites. In any case, robots.txt does not grant any copyright permissions; it is merely a technical file, as explained below.
Google and other Big Tech companies were mostly honest, value-creating enterprises until around 2008. The main factor behind Big Tech’s wealth, and the collapse of honest journalism and civil society, was Google and Microsoft’s plundering of content from millions of websites with impunity. Here, I am focused on text-based content, like news, commentary, scholarly and scientific works—in other words, the works that contain or create human knowledge. Continue reading Google Plunders the Web →