An SEO beginner’s guide to robots.txt

You’ve heard of this term before but don’t really get it? A robots.txt file is like the virtual bouncer guarding the entrance to your website. Its job is to inform search engine bots which areas they are allowed to visit and which ones they should avoid.