What Is Robots.txt – Today’s Search Question

Robots.txt is probably something you’ve heard of, and you may even have heard that it’s important for search engine optimization. Today, we’ll explain what it is and when you should use robots.txt.

Robots.txt is a small file you can place on your website that tells search engines how you want them to handle your site. Primarily, it allows you to instruct them not to crawl your site at all, or not to crawl certain parts of it. This may sound very appealing, but there are pitfalls involved, which we address in the video. Don’t forget that you can subscribe to our YouTube channel if you like the clip, and if you want us to continue producing Today’s Search Question, be sure to click the thumbs up as well.