Answer: In search engine optimization (SEO), there are manyterms that are confusing. For example, many firms that specialize insearch engine advertising claim that they are search engine marketers,even though search engine marketing encompasses a wide variety ofskills (search engine optimization, directory paid inclusion, searchengine paid inclusion, vertical search, and search engine advertising).
In SEO, the terms spider/crawler, index, and rank have threecompletely different meanings. This article will address the differentmeanings, and why each term is important in the optimization process.
Search engine spiders
Search engine spiders are also known as crawlers. Search enginespiders find and fetch Web pages, and build a list of words found oneach Web page. So if a search engine "spiders" your site, the spider arrivesat your Web server by following a link to your site, requests that URLfrom your server, and then builds the list of words found on that URL.
If a search engine is unable to spider a page, then it cannotinclude the information about that page in the search engine resultspages (SERPs). Common reasons that search engines are unable to spidera site include:
· Problematic site navigation schemes
· Problematic URL structures
· Poor cross-linking
· Web server issues
Interestingly, your Web analytics software (WebTrends,ClickTracks, Omniture, Urchin, Hitbox, etc.) will tell you how oftenyour site is crawled by search engine spiders. If you change yourcontent frequently, search engine spiders will revisit your site morefrequently because search engine companies want to deliver the mostaccurate, current, and relevant search results.
However, a page will not rank higher just because a spidervisits your site more frequently. Spidering and ranking are quitedifferent processes.
The search engine index
After a search engine is able to spider a page and records thewords and phrases on a page, it places that data into an index.Whenever you perform a search on Google, for example, the data about aURL is stored in the search engine index. Some search engine marketerscall the index a "search engine database."
A page can be included in a search engine index but not rank.However, a page cannot rank unless it is in the search engine index.
Web site owners should monitor their sites' index count everymonth. An index count is the number of pages that are in the searchengine indices. Each search engine has its own way of determining theindex count. Below are current ways of determining the index count ofyour site:
· Google and Teoma
site:yourdomain.com yourdomain.com
· FAST Search
domain:www.yourdomain.com
· AltaVista
domain:www.yourdomain.com
· Yahoo
site:yourdomain.com or domain:yourdomain.com
· MSN Search
site:yourdomain.com
Take the time to perform these types of searches. Do not relyon link popularity checkers and other software that performs automatedqueries to the search engines. Unless the software owners havepermission from Google, Yahoo, MSN Search, Teoma and other searchengines to perform these queries (none of them do), the results mightnot be accurate.
If your pages are not included in a search engine index, thena site might have problems with server issues, site navigation, and URLstructure. In addition, a site can be spidered by a search engine butnot be included in the search engine index.
Search engine rankings
Okay everyone, repeat after me: No SEO knows what the searchengine algorithm is. Say it five times. Say it 10 times. Say it 100times or say it more until it sinks in.No search engine marketer knows what a search engine's algorithm is. Nice sales pitch, but one that is just not true.
No one knows exactly how a search engine ranks Web pages. Whatethical search engine marketers do is follow the terms and guidelinesset forth by the search engines. Unethical search engine marketers tryto exploit the loopholes in the terms and guidelines. It is often theunethical search engine marketers who claim to know Google's searchengine algorithm.
Search engine rankings tend to rely on three components:
1. Keyword rich text - A page must contain the words and/or phrases that people type into search queries.
2. Search engines must be given easy access to that text.
3. Other people must consider a page's content to bevaluable. If a person thinks content is valuable, that person tends tolink to the page that contains the content.
Of course, no search engine measures these components in thesame manner. For example, all search engines remove common words from apage, called stop words or filter words. Filter words are common words(a, an, of, or, and, the) that the search engines ignore during asearch. Search engines filter out these words because using these wordsin a search query can slow down the retrieval of search results withoutimproving accuracy.
Google's stop word list is not going to be the same as Yahoo'sstop word list. Additionally, no keyword density checker (from anysearch engine marketer) is going to have the same stop word list asGoogle. Likewise, each search engine measures link popularity in adifferent way.
Does that mean I advise people to create different pages fordifferent search engines? Of course not. A well-written, user-friendly,and spider-friendly site can get plenty of traffic from all of thesearch engines without resorting to search engine spam.
So remember, being ranked in a search engine does not have the same meaning as being indexed in a search engine.
 |  | |
Related Articles
- Paid URL InclusionIn this article you will get a concise presentation of all the various concerns involved with a paid URL inclusion to a Search Engine. The costs and benefits of this approach are weighed for different situations.
- The Truth About Web PromotionThe very basics of a web related business lie in the fact that web promotion is an ultimate word in the entire process of the system. Truly there are two types of search engines. Namely, Search Engine Directories and Search Engine Spiders...
- 10 Important Web Design Tips: SEO Friendly WebsiteThis article gives a very practical list od 10 Important Web Design Tips: SEO Friendly Website. The author explains the importance of each tip and then tells you how to achieve the needed design element in your webpage.
- Robots.txt FileWe all know search engine optimization is a tricky business. Sometimes we rank well on one engine for a particular keyphrase and assume that all search engines will like our pages, and hence we will rank well for that keyphrase on a number of engines. Unfortunately this is rarely the case. All the m...
- How The Yahoo! Directory Really Ranks Its Directory ListingsIn this article, I show you how the Yahoo! Directory ranks its listings, so you can use this knowledge to get higher rankings. First you must ...
- Grow Online With Reciprocal LinkingIf youve been marketing online, for even a short amount of time, you are probably aware of the many benefits of reciprocal links. This simple process of trading your link with other sites can generate visitors and increase your search engine popularity ranking. Keep in mind that, if your t....
- Search Engine Spider, Index, And RankingIn SEO, the terms spider/crawler, index, and rank have three completely different meanings. This article will address the different meanings.
- Page Cloaking and Some MoreWeb cloaking, a technique used to deliver different web pages to users and browsers, can seem very attractive. But is it really safe? Get an experts opinion.
- Protecting Your Search Engine RankingsYour websites ranking on search engines is a vital element of your overall marketing campaign, and there are ways to improve your link popularity through legitimate methods. Unfortunately, ...
- Protecting Your Search Engine RankingsYour websites ranking on search engines is a vital element of your overall marketing campaign, and there are ways to improve your link popularity through legitimate methods. Unfortunately, the Internet is populated by bands of dishonest webmasters seeking to improve their link popularity by faking....