How Do I Direct Spiders Around My Site?

hanging complete spider webs

That’s a good question as people new to site building aren’t aware that there are web spiders (bots), much less 3 types of site maps. Bot’s are essentially discovery programs used by the search engines. The bot’s roam around the web, finding new web sites and web pages by use of  common web links. Sitemaps are repositories of a web sites resources/pages, represented as links. Each kind of site map is similar but with it’s own specific purpose. With out sitemaps, bots would have difficulty finding all your site web pages and we all know disastrous that would be!

The three types of sitemaps used to direct the spider around your site are sitemap.html, sitemap.xml & sitemap.xml.gz. Way back during the ‘wildwest’ days of the internet (10 years ago), web surfers liked a concise way to scan the contents of the site they were viewing and, necessity being the mother of invention, the site map was born.  This type of site map was really only a normal ‘.html’ page containing short descriptions and links to the sites resources. Later on as the web matured (5 years ago), the site map was also used as a way for spiders to find all you sites pages. The only real reason for this was due to poor or broken site navigation.

Now that we have entered ‘Web 2.0′, the site map has been reincarnated as a search engine specific tool. This current not for human consumption incarnation of the sitemap is designed to let the search engine spiders find each page of our web site with out having to follow hyperlinks throughout the site. This type of sitemap uses the ‘.xml’ format file extension and is located at the root of the publicly accessible portion of the domain. The third type of sitemap is a Google specific archive ending in .xml.gz and is simply a compressed version of your .xml sitemap.

Just in case you are wondering, each file will contain the same destination pages, the only difference is in the format in which the information is delivered. Ideally speaking, every site really should have each type of site map present and a reference in the robots.txt.

There are many brands of software available to create sitemaps of each type. When using a Worpress blog, they are especially easy to create with simple plugins.

So there you have it!

Related SEM-Answers Posts

• Link Building And Parasite Hosting In these tough economic times parasite hosting can allow you to get one way back-links to you web site for free, reaping the SEO benefit, without spending a dime. This method can be very effective and the price is definitely right. On this page you will find out what......
• e-Commerce 2.0 Work From Home With Totally Free e-Commerce (more or less) A.K.A, open a web store without spending a dime Scenario A: So you have a the need to open a web store. You have your product all picked out and have found a drop shipper. What now? Scenario B: You......

Tags: html sitemap, xml sitemap, xml.gz sitemap

Stumble It

Ping this!