Skip to content

Commit

Permalink
update aiblocker robots.txt
Browse files Browse the repository at this point in the history
  • Loading branch information
ezrizhu committed Dec 29, 2023
1 parent b818469 commit a6e4914
Showing 1 changed file with 51 additions and 0 deletions.
51 changes: 51 additions & 0 deletions assets/robots.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,56 @@
User-Agent: *
Sitemap: https://ezrizhu.com/sitemap.xml
Disallow: /health

# Dark Visitors robots.txt

# AI Assistant
# https://darkvisitors.com/agents/chatgpt-user

User-agent: ChatGPT-User
Disallow: /blog/

# AI Assistant
# https://darkvisitors.com/agents/cohere-ai

User-agent: cohere-ai
Disallow: /blog/

# AI Data Scraper
# https://darkvisitors.com/agents/anthropic-ai

User-agent: anthropic-ai
Disallow: /blog/

# AI Data Scraper
# https://darkvisitors.com/agents/ccbot

User-agent: CCBot
Disallow: /blog/

# AI Data Scraper
# https://darkvisitors.com/agents/facebookbot

User-agent: FacebookBot
Disallow: /blog/

# AI Data Scraper
# https://darkvisitors.com/agents/google-extended

User-agent: Google-Extended
Disallow: /blog/

# AI Data Scraper
# https://darkvisitors.com/agents/gptbot

User-agent: GPTBot
Disallow: /blog/

# AI Data Scraper
# https://darkvisitors.com/agents/omgilibot

User-agent: omgilibot
Disallow: /blog/

User-agent: omgili
Disallow: /blog/

0 comments on commit a6e4914

Please sign in to comment.