jump to navigation

Beginner’s SEO Corner - I want to SEE! (Search Engine Eyes) March 25, 2006

(Visited 53579 times)

The search engines index your website by using their bots which are also called crawlers or spiders. And these bots do nothing but visits the URLs and check your links and visits those links too. The keywords on the pages are indexed and recording all keywords it can find* and including it in the indexed information.

When someone visits a website, the browser used to visit the website is sent to the website’s server which is known as the User Agent. There are several ways to know the User Agent either by using plain Javascript or using your favorite server-side programming language like: PHP, JSP, ASP, Perl, Cold Fusion, Miva, etc. And checking the user agent you can see if the visitor of your page is using Microsoft Internet Explorer, Mozilla Firefox, Opera, Netscape Navigator, Safari, AOL Browser, Konqueror or whatever browser.

If you use Firefox already (download Firefox if you are not using it yet) you can use the User Agent Swticher plug-in by Chris Pederick. And once you have that downloaded, you can change your User Agent easily. And if you want to see what search engines are seeing, you need to add the user agents of search engines.

Once you got the User Agent Switcher installed on your Firefox browser, you have the option to add more user agents and it will ask you for the: Description, User Agent, App Name, App Version, Platform, Vendor and Vendor Sub.

User Agent Switcher

Once you see these fields, you can add in the following for the big 3 search engines:

Google:

Yahoo:

MSN:

Now what reason why you would like to switch User Agents? Most websites will look the same anyway? Check out Greg Boser’s site with the different User Agents.

Greg Boser’s Website on any ordinary Browser:

Greg Boser's Website on Firefox

Greg Boser’s Website as viewed by Googlebot:

Greg Boser's Website on Google

So the reason for using there plug-in is for your own use when developing websites that take out unnecessary code that search engines do not need to see and just serve the necessary content, no images. As long as you do not show totally different content, you are safe and is not considered blackhat and is already content cloaking. Now how to do that is another story.
*Some research has it that search engines do not read the whole page and will read the page from top to bottom up to some limit. I can’t remember if this is by bytes or number of characters. Try doing some research and post a comment about it to inform everyone else.

Comments»

1. jonathan - June 25, 2006

This is useful if you want to create a summary of links in a page and present it to the bot. I think bots of this days are capable of excluding layouts (css or tables) when crawling pages.