Multiple sitemaps question

Posted By: 123_123 ()
Posted On: 2005-May-02 18:08

I have a site with around 30K HTML pages in 14 directories.
I have made a sitemap for each directory, 14 sitemaps

Now then what I have in effect is a jumbo list of URL's all listing pages which are unique content wise, no duplicates.

The question is:

Could I submit those seperate URls to the engines ? Ok of course I could submit them but is there any implications in doing so ?

How would you setup the sitemaps for the spiders to go spidering ?


By that I mean, would you put a sitemap link on the index page, that sitemap listing two more sitemaps, which in turn list a few more ? Would such a structure work from a spidering point of view, or what would be a better solution ?
Thanks








Posted By: g1smd (Staff)
Posted On: 2005-May-02 18:24

Firstly, you don't submit sitemaps anywhere. The bots will find them if your site has incoming links from other sites and your sitemap is well linked within the site itself.

If your website has sections then you make one page of links for each section, and then have a main list which lists all of those. The main list should be linked from many important pages of the site.

Google recommends 100 links per page, and will not spider files larger than 100KB anyway. They did briefy spider pages up to, at least, 250KB but no longer do this.


Posted By: 123_123 ()
Posted On: 2005-May-02 18:38

The site has 14 sections, but each, "section", could have as many as 2000 pages within it

I have created a sitemap for each of the 14 sections, but as you suggest idealy each map should really be no more than 100 links. Following that I think you are suggesting I break up the maps into handlable chunks.

Again following that, if say one section has 2000 pages, then do we make 20 maps to cover that section, and run sitemap1,2,3,4,5 etc links at the top of the index page for each section, or do we do a site map of the site maps ?


Posted By: g1smd (Staff)
Posted On: 2005-May-02 19:02

If your website has sections then you make one page of links for each section, and then have a main list which lists all of those. The main list should be linked from many important pages of the site.


Posted By: linknz ()
Posted On: 2005-May-22 10:48

My search engine uses a web spider called "Linknzbot" and that would have no problems with your directory structure and the number of files.
On the very first indexing of a web site the spider is controlled to only index six levels down picking up nine links per level but on the second visit it will go down fifty levels and collect all links found, it will index up to 500K per page as well.
The way that you describe your links shouldn't be a problem to most web spiders.
All the best
Dave A


Posted By: ratan_angel ()
Posted On: 2005-Sep-19 10:44

On the very first indexing of a web site the spider is controlled to only index six levels down picking up nine links per level but on the second visit it will go down fifty levels and collect all links found, it will index up to 500K per page as well.


I was wondering if this is true for spiders of all Search Engines or does this apply to just a couple? If the latter is true, then which are the Search Engines to which this applies?

Apologies for the ignorance, but I reckon every bit of new information is an addition to one's knowledge.

Thanks guys!

Ratan


Posted By: yellowwing ()
Posted On: 2005-Sep-25 02:07

What about the new Google site map [feed?] feature? Does anyone have the lowdown on that?