Reddit is now blocking major search engines and AI bots – except those that pay

Reddit is stepping up its crackdown on web trackers. In recent weeks, Reddit has begun blocking search engines from displaying recent posts and comments unless the search engine pays, according to a report from 404 Media.

Right now, Google is the only mainstream search engine that shows recent results when you search for posts on Reddit using the “site:reddit.com” trick.404 Media reports. That leaves out Bing, DuckDuckGo, and other alternatives — and that’s likely because Google struck a $60 million deal that allows the company to train its AI models on Reddit content.

It’s a bold move for a huge site like Reddit to block some of the most popular search engines, but it’s not all that surprising. Over the past year, Reddit has become more protective of its data as it seeks to open up another revenue stream and please new investors. After making its API more expensive for some third-party developers, Reddit threatened to cut off Google’s access if it didn’t stop using the platform’s data to train AI for free.

On the edge reached out to Reddit with a request for comment but did not immediately receive a response.

Last month, to enforce its anti-scraping policy, Reddit updated the site’s robots.txt file, which tells web crawlers whether they can access a site. “It’s a signal to those who don’t have an agreement with us that they shouldn’t be accessing Reddit’s data,” Ben Lee, Reddit’s chief legal officer, told my colleague Alex Heath in Command Line.

With AI chatbots filling the internet with questionable content, finding things written by other humans has never been more important. I, like many others, have started appending “Reddit” to many of my searches just to get human responses, and it’s very frustrating to know that now I’ll only be able to do this on Google (or search engines that depend on it). ) – especially when I do a lot of my searches on Bing.