How Do I Direct Spiders Around My Site?

hanging complete spider webs

That’s a good question as people new to site building aren’t aware that there are web spiders (bots), much less 3 types of site maps. Bot’s are essentially discovery programs used by the search engines. The bot’s roam around the web, finding new web sites and web pages by use of  common web links. Sitemaps are repositories of a web sites resources/pages, represented as links. Each kind of site map is similar but with it’s own specific purpose. With out sitemaps, bots would have difficulty finding all your site web pages and we all know disastrous that would be!

The three types of sitemaps used to direct the spider around your site are sitemap.html, sitemap.xml & sitemap.xml.gz. Way back during the ‘wildwest’ days of the internet (10 years ago), web surfers liked a concise way to scan the contents of the site they were viewing and, necessity being the mother of invention, the site map was born.  This type of site map was really only a normal ‘.html’ page containing short descriptions and links to the sites resources. Later on as the web matured (5 years ago), the site map was also used as a way for spiders to find all you sites pages. The only real reason for this was due to poor or broken site navigation.

Now that we have entered ‘Web 2.0′, the site map has been reincarnated as a search engine specific tool. This current not for human consumption incarnation of the sitemap is designed to let the search engine spiders find each page of our web site with out having to follow hyperlinks throughout the site. This type of sitemap uses the ‘.xml’ format file extension and is located at the root of the publicly accessible portion of the domain. The third type of sitemap is a Google specific archive ending in .xml.gz and is simply a compressed version of your .xml sitemap.

Just in case you are wondering, each file will contain the same destination pages, the only difference is in the format in which the information is delivered. Ideally speaking, every site really should have each type of site map present and a reference in the robots.txt.

There are many brands of software available to create sitemaps of each type. When using a Worpress blog, they are especially easy to create with simple plugins.

So there you have it!

Related SEM-Answers Posts

• Web Page Optimization Components 1 How To Rise In The SERPs and add zero's to your bank account In this series of articles, we will examine individually, each of the major components of web page optimization. Mastering and blending each of these components can be tricky to say the least and , it is likely......
• Web Page Optimization Components 3 How To Rise In The SERPs and add zero’s to your bank account Combination factors, both on and off page, are the most powerful of all the ranking factors. Get these right and you will have a much easier time not only getting the search engines to understand what it......

Tags: html sitemap, xml sitemap, xml.gz sitemap

Stumble It

Ping this!