When optimizing your web site most webmasters don't considerusing the robots.txt file. This is a very important file for your site.It let the spiders and crawlers know what they can and can not index.This is helpful in keeping them out of folders that you do not wantindex like the admin or stats folder or content that they can not index.Here is a list of variables that you can include in a robots.txt file and there meaning:
1)User-agent:In this field you can specify a specific robot to describe accesspolicy for or a "*" for all robots more explained in example.
2)Disallow: In the field you specify the files and folders not to include in the crawl.
3)# the number sign represents comments
Here are some examples of a robots.txt file for redball.com
User-agent: *
Disallow:
The above would let all spiders index all content.
Here another example
User-agent: *
Disallow: /cgi-bin/
The above would block all spiders from indexing the cgi-bin directory.
User-agent: googlebot
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /stats/
Inthe above example googlebot can index everything while all otherspiders can not index admin.php, cgi-bin, admin, and stats directory.Notice that you can block single files like admin.php.
Related Articles
- Chat With Search Engine SpidersWe are living in an age where robots and spiders are crawling all over your Web site. No, this isnt a tag line from an old 1950 horror movie, this is the way things are. Dont be frightened though. The fact that you have robots and spiders on your Web site is a good thing...
- Search Engine Spiders Lost Without Guidance - Post This Sign!Robots.txt Signpost Warns Trespassers From Private Property The robots.txt file is an exclusion standard required by all web crawlers/robots to tell them what files and directories that you want them to stay OUT of on your site. Not all crawlers/bots follow the exclusion standard and will continu...
- Software To Help You Create A Robots.Txt fileRobogen is a software for Windows 95, Windows 98,Windows XP and Windows NT based computers which can help you create robots.txt files without having to worry about the correct syntax of a robots.txt file...
- Robots.txt FileWe all know search engine optimization is a tricky business. Sometimes we rank well on one engine for a particular keyphrase and assume that all search engines will like our pages, and hence we will rank well for that keyphrase on a number of engines. Unfortunately this is rarely the case. All the m...
- Creating a Robots.txt fileSome people believe that they should create different pages for different search engines, each page optimized for one keyword and for one search engine. Now, if you do decide to create such pages, there is one issue that you need to be aware of...
- 5 Best Ways To Make Your Website Visible To Search EnginesEvery website owner needs to make sure their website is visible to search engines but its a task that is easier said than done. Here are 5 of the best ways to make sure your site is visible to the major search engines.
- Disabling Google And Other Search Engines From Crawling A SiteIf I disable Google to my Web site, its possible Google.com erase or drop down my Web site for his directory?
- Blocking Bad Agents from downloading your websiteIf you found this page searching Google or search engine, you must be looking for a solution how to bloc.....
- Generating SEO Report For Your WebsitePeople involved in online ventures are aware that SEO or Search Engine Optimization is essential for improving website-ranking with the Search Engines and also to attract more traffic to the site.
- 7 Top Ways to Avoid Link TheftIf you have a link directory on a website, how do you stop link theft by sites that dont link back, or trick you into thinking they do?...