Post

Barry Schwartz
@rustybrick.com

🤖 How should SEOs handle robots.txt for AI crawlers, emojis and international URLs? Google’s Gary Illyes and Cherry Prommawin explain the rules, RFC 9309, and common pitfalls in this new discussion

seroundtable.comGoogle On robots.txt Rules For AI Crawlers, Emojis, International URLs & MoreGary Illyes and Chery Prommawin released their second video in the Non-Canonical Cabinet series; this one is named What can you actually put in a robots.txt? Here they discuss what is, and is not, supported by robots.txt files. And while Google may support features or directives, other search engines or AI crawlers might not.

1 repost4 likes

Barry Schwartz
@rustybrick.com

The SEL team did something similar to me. Took me months to realize, after several videos were posted on their channel, I finally noticed me in the intro of all of them...