Robogen is a software for Windows 95, Windows 98,Windows XP andWindows NT based computers which can help you create robots.txt fileswithout having to worry about the correct syntax of a robots.txt file.It comes in two editions: the free Limited edition and the Standardedition which costs $12.95. You can get Robogen fromRietta.com. In this article, I'll tell howyou can use Robogen to simplify the process of creating a robots.txtfile. This article assumes that you have read my article on what arobots.txt file is and how you can manually create it. If you missedthe article, it is available at1stSearchRanking.com
Iwould recommend that you download the trial version of the StandardEdition. The Limited edition does not support as many search engines asthe Standard Edition.
Once you have downloaded Robogen andinstalled it in your hard drive, start the program. Here are the stepsthat you should follow to create a robots.txt file:
Step 1:Click on the Logon FTP Server button. In the dialog box that appears,type in the address of the FTP server for your web site, make sure thatthe "This server requires me to log in" option is checked, and thentype in the username and the password which you use to access theserver in the appropriate boxes.
Now, depending on how your webhost has set up the server, the files for your web site may be presentin the root directory of the server itself. Or, it may be present in asub-directory of the root directory. In the latter case, click on the"Use specified root" option and type the relevant sub-directory in thetext box besides the option. If you don't know in which sub-directorythe files for your web site are present, ask your web host to give itto you.
Done? Now click on OK. After a while, Robogen willdisplay the directories and files which are present in your site in theright pane of the main Robogen program window. If you find that Robogenhas not displayed the files in your site, it means that you did notenter the correct sub-directory in which the files of your web site arepresent. In this case, ask you web host to give the correctsub-directory to you.
Step 2: In the left pane of the mainRobogen window, click on the small button present to the right of thedrop-down combo box at the top. In the menu that appears, select"Popular Search Engines".
Step 3: Now, click on the drop-downcombo box and choose the search engine that you are targeting (say,AltaVista). In the right pane, select a page which is NOT meant forAltaVista. Now click on the "<<<" button. You will find thatthe file which you had selected in the right pane is now present in theleft pane. In this way, click on each of the pages which are not meantfor AltaVista and press the "<<<" button.
Repeat step 3for each of the search engines that you are targeting. Just select thesearch engine that you are targeting from the drop-down combo box,select the pages which are not meant for the search engine one by onefrom the right pane, and click on the "<<<" button afterclicking on each age. Repeat this process for all the search enginesthat you are targeting.
Step 4: Upload the file to your serverusing the "Upload Robots.Txt" button. If you want, you can also use theSave menu item from the File menu to save a copy of the robots.txt filein your hard drive.
Related Articles
- Software To Help You Create A Robots.Txt fileRobogen is a software for Windows 95, Windows 98,Windows XP and Windows NT based computers which can help you create robots.txt files without having to worry about the correct syntax of a robots.txt file...
- Chat With Search Engine SpidersWe are living in an age where robots and spiders are crawling all over your Web site. No, this isnt a tag line from an old 1950 horror movie, this is the way things are. Dont be frightened though. The fact that you have robots and spiders on your Web site is a good thing...
- Creating a Robots.txt fileSome people believe that they should create different pages for different search engines, each page optimized for one keyword and for one search engine. Now, if you do decide to create such pages, there is one issue that you need to be aware of...
- Robots.txt FileWe all know search engine optimization is a tricky business. Sometimes we rank well on one engine for a particular keyphrase and assume that all search engines will like our pages, and hence we will rank well for that keyphrase on a number of engines. Unfortunately this is rarely the case. All the m...
- Blocking Bad Agents from downloading your websiteIf you found this page searching Google or search engine, you must be looking for a solution how to bloc.....
- Disabling Google And Other Search Engines From Crawling A SiteIf I disable Google to my Web site, its possible Google.com erase or drop down my Web site for his directory?
- Search Engine Spiders Lost Without Guidance - Post This Sign!Robots.txt Signpost Warns Trespassers From Private Property The robots.txt file is an exclusion standard required by all web crawlers/robots to tell them what files and directories that you want them to stay OUT of on your site. Not all crawlers/bots follow the exclusion standard and will continu...
- 7 Top Ways to Avoid Link TheftIf you have a link directory on a website, how do you stop link theft by sites that dont link back, or trick you into thinking they do?...
- How Link Exchange Partners Cheat YouIt is not just you, these things happen to everyone. Worrying only leads to more problems.
- Use This JavaScript To Hide Your EMail From SpambotsEver wander why so many webmasters refuse to insert their email address into the body of their webpages?