It may sound like something from “The Matrix Trilogy” but Google, Bing and other search engines use programs called web crawlers aka a spiders aka a robots or even agents to search and index the web.
Spiders, Web Crawlers and Agents! Oh My!
Don’t worry! These aren’t mechanical monstrosities or evil subroutines bent on destroying all of humanity or even using us as a convenient power source.
Web crawlers are the programs that search engines use to explore websites in order to list them and rank them in their directories. Except for malware, web crawlers are the good kind of robots.
When a web crawler explores a website, it looks for a robots.txt file. This is a simple text file with just a little bit of code that can instruct which web crawlers are allowed on the site and where they can and cannot go. Often it is used to close off sections of a website that might be a duplicate of another part of the site (such as a separate mobile version or a version in a duplicate language).
Anatomy of a Robots.txt File
The code of a robots.txt file generally looks like this:
User-agent: * Disallow:
The user-agent section specifies the web crawlers that are allowed or not allowed to index the site. The Disallow section is used to specify which files are off limits.
The statement above says that all web crawlers (an asterisk is a wildcard meaning “all”) are allowed to search the site with no restrictions (because there is nothing after the Disallow statement).
The next example says that only Google is allowed to search the site. All others are banned. The slash after the Disallow statement refers to all files.
User-agent: Google Disallow:
User-agent: * Disallow: /
For more information, visit http://www.robotstxt.org/robotstxt.html. If you are using WordPress, check out the article by Joost de Valk, one of the co-founders of WordPress, on how he configures his robots.txt files.
Need More Help?
If you liked this article, be sure to sign up for your FREE subscription to the Nickel's Worth of Free Advice newsletter where we send helpful articles every Tuesday and Friday. You can either fill out the form on the right side of this page or visit the Nickel's Worth of Free Advice signup page.
We're also happy to provide assistance directly. Just send an email to firstname.lastname@example.org.