The Great GPT Firewall 📛

This collection is a curated list of websites that employ the robots.txt file to restrict access to AI Agents, AI crawlers and GPTs.

It will be updated monthly.

User agents & robots.txt

The robots.txt file allows website owners to control and limit the access of these user agents to certain areas of their website by specifying rules and directives.

# OpenAI’s web crawler: GPT3.5, GPT4, ChatGPT
# https://platform.openai.com/docs/bots
User-agent: GPTBot

# ChatGPT plugins
# https://platform.openai.com/docs/bots
User-agent: ChatGPT-User

# OpenAI Search bot
# https://platform.openai.com/docs/bots
User-agent: OAI-SearchBot

# Google's web crawler: Bard, VertexAI, Gemini
# https://blog.google/technology/ai/an-update-on-web-publisher-controls/
User-agent: Google-Extended

# Apple's web crawler, dedicated to GenAI projects
# https://support.apple.com/en-us/119829
User-agent: Applebot-Extended

# Claude
User-agent: anthropic-ai

# Claude Bot
User-agent: ClaudeBot

# Claude web
User-agent: Claude-Web

# Amazonbot
# https://developer.amazon.com/amazonbot
User-agent: Amazonbot

# Cohere
User-agent: Cohere-ai

# Perplexity
User-agent: PerplexityBot

# You
# https://about.you.com/fr/youbot/
User-agent: YouBot

# Common Crawl
# https://commoncrawl.org/ccbot
User-agent: CCBot

# Omglibot: webz.io
# https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/
User-agent: Omgilibot
User-agent: Omgili
User-agent: Webzio-Extended

# Facebook: Llama
# https://developers.facebook.com/docs/sharing/bot/
User-agent: FacebookBot

# Facebook
# https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
User-agent: Meta-ExternalAgent

# ByteDance: Duobao
User-agent: Bytespider

# Ai2
# https://allenai.org/crawler
User-agent: Ai2bot
User-agent: Ai2Bot-Dolma

# Diffbot
User-agent: Diffbot

# Huawei
# https://darkvisitors.com/agents/pangubot
User-agent: PanguBot

# Petal Search
# https://datadome.co/learning-center/how-to-block-petal-bot/
User-agent: PetalBot

Timpibot
# https://darkvisitors.com/agents/timpibot
User-agent: Timpibot

# Censorship area
Disallow: /

Disclaimer

Please note that this blocklist is intended for informational purposes only. Despite the provoking project name, it's fine to disallow web crawling and protect content ownership.

2026-07 update

Category: Press

• Scanned: 66
• ✅ Passing: 27 %
• 🔐 Blocked: 73 %
• ❓ Unknown: 0 %

Name	Country	Status
The Times	🇬🇧	🔐
BBC	🇬🇧	🔐
The Guardian	🇬🇧	🔐
The Economist	🇬🇧	🔐
Financial Times	🇬🇧	🔐
The Independent	🇬🇧	✅
The Telegraph	🇬🇧	🔐
Daily Mail	🇬🇧	🔐
The Sun	🇬🇧	🔐
Daily Mirror	🇬🇧	🔐
Daily Express	🇬🇧	🔐
Washington Post	🇺🇸	🔐
USA Today	🇺🇸	🔐
Fox News	🇺🇸	✅
ABC News	🇺🇸	🔐
NBC News	🇺🇸	🔐
CBS News	🇺🇸

Category: Video on demand

• Scanned: 9
• ✅ Passing: 33 %
• 🔐 Blocked: 67 %
• ❓ Unknown: 0 %

Name	Country	Status
Prime Video	🌍	🔐
Netflix	🌍	🔐
Disney+	🌍	🔐
Hulu	🇺🇸	🔐
HBO Max	🇺🇸	✅
Canal+	🇫🇷	✅
FranceTV	🇫🇷	🔐
TF1	🇫🇷	🔐
6Play	🇫🇷	✅

Category: Music

• Scanned: 6
• ✅ Passing: 67 %
• 🔐 Blocked: 33 %
• ❓ Unknown: 0 %

Name	Country	Status
Soundcloud	🌍	🔐
Youtube	🌍	✅
Apple Music	🌍	✅
Spotify	🌍	🔐
Deezer	🇫🇷	✅
LastFM	🇬🇧	✅

Category: Podcast

• Scanned: 8
• ✅ Passing: 75 %
• 🔐 Blocked: 25 %
• ❓ Unknown: 0 %

Name	Country	Status
Google Podcasts	🌍	✅
Apple Podcast	🌍	✅
Spotify Podcaster	🌍	✅
Buzzsprout	🌍	✅
Podbean	🌍	🔐
Acast	🇬🇧	✅
AudioMeans	🇫🇷	✅
Radio France	🇫🇷	🔐

Category: X

• Scanned: 6
• ✅ Passing: 100 %
• 🔐 Blocked: 0 %
• ❓ Unknown: 0 %

Name	Country	Status
PornHub	🌍	✅
YouPorn	🌍	✅
Xnxx	🌍	✅
Xvideos	🌍	✅
Xhamster	🌍	✅
OnlyFan	🌍	✅

Category: Religion

• Scanned: 5
• ✅ Passing: 100 %
• 🔐 Blocked: 0 %
• ❓ Unknown: 0 %

Name	Country	Status
Bible	🇺🇸	✅
Bible gateway	🇺🇸	✅
Jehovah's Witnesses	🇺🇸	✅
Vatican	🇻🇦	✅
Islamweb	🌍	✅

Category: Social media

• Scanned: 13
• ✅ Passing: 23 %
• 🔐 Blocked: 69 %
• ❓ Unknown: 8 %

Name	Country	Status
Facebook	🌍	🔐
Instagram	🌍	🔐
Reddit	🌍	✅
Hacker News	🌍	❓
Lobsters	🌍	✅
Pinterest	🌍	🔐
TikTok	🌍	🔐
Twitter	🌍	🔐
LinkedIn	🌍	🔐
Quora	🌍	🔐
VK	🇷🇺	✅
TripAdvisor	🌍	🔐
Yelp	🌍	🔐

Category: Artist

• Scanned: 42
• ✅ Passing: 67 %
• 🔐 Blocked: 26 %
• ❓ Unknown: 7 %

Name	Country	Status
Michael Jackson	🇺🇸	🔐
Madonna	🇺🇸	✅
Taylor Swift	🇺🇸	🔐
Rihanna	🇺🇸	✅
Bruno Mars	🇺🇸	✅
Justin Bieber	🇺🇸	🔐
Beyoncé	🇺🇸	✅
Katy Perry	🇺🇸	🔐
Lady Gaga	🇺🇸	✅
Hardwell	🇺🇸	✅
Dimitri Vegas Like Mike	🇺🇸	✅
Kanye West	🇺🇸	❓
Black Eyed Peas	🇺🇸	✅
Imagine Dragons	🇺🇸	🔐
Twenty One Pilots	🇺🇸	✅
Maroon 5	🇺🇸	🔐
Selena Gomez