mod_rewrite, keeping old pages indexed

Firstly hello everyone here, thought I would have a change of scene and visit somewhere new.

Bear with me with this, terrible at explaining this sort of thing.

I have an item search form, from which a user can search on 15+ parameters, which all get passed by the query string page to page which causes a problem with duplicated content. Also when they click on the item they get taken through to the item details page, the query string is also passed to this page for navigational purposes, again this causes more duplicated content.

So I have created a different search with 2 parameters which generates search friendly urls using mod_rewrite. The problem is that I have got over 1000 products indexed the old way and the preferable way for users to find the products is using the old search, but that way is producing to much duplicated content. I didn't want to add a rule to the robot file to disallow all the items on the old urls as it has taken time to build up the high rankings for some of these products. On the other hand I don't want any more new/additional products to be index from the old urls.

Anyone any suggestions how I can get all the old url's pointing to the new urls without using 301 redirect as this would affect the search result and also get the new urls indexed with out worrying about duplicated content and how to stop getting the old search indexed but still keep the results.

Or am I worrying to much about keeping the old items?

Hope this makes sense??

Thanks for your advice!

Shaun

[ Message was edited by: kirk112 07/10/2008 07:26 am ]

I feel your pain and sympothise but unfortunately I'm in this exact situation and I'm doing it the long way and it's working.

I have 15,000 listings, real estate, and the pages are not optimal for spidering, so we changed them and are in the process of removing the 15,000 old pages and indexing the new ones.

First I redirected them to the new pages using the 301 but that only made thousands of pages become unavailable because they were passed over.

Then I used htaccess / mod_rewrite to change URLS and send the spider to the new friendly pages. THen I removed the old sitemaps, created new one's, submitted them to google, and we're off to the races a few days later.

So far I have 5,000+ of the new pages indexed and returning the new, expected traffic. The older pages are being removed because we setup the robots file to take care of that.

We have seen a slight traffic decrease but that was expected. GOogle will only cache for so long on your site, if that means caching OUT old pages and marking them off that's what they do. THat can take a few days but as long as you can verify google is visiting your site multiple times per day you shouldn't have a problem getting rid of the old and in with the new.

Again, that all depends on where the majority of entry pages to your site are coming in on. If less than 50% are coming in on the "/" or home page then you may want to try and split it up into 4 sections, 1 per week or something. THat way you don't see the decline as much and it's over a longer period of time.

That was my mistake, I did it all at once but I HAD to. the old pages had almost 4,000 lines of code before the first human word and they were covered in Ajax.

On to better things, getting those new pages in order so they can produce even better online.

Good Luck!

One point, mod_rewrite doesn't generate any URLs at all.

What it does, is translate URL requests (and those requests come from people clicking the links on your pages - so it is your PHP script that generates the links), and it translates those URL requests into a server-internal path to fetch that content from, and the fetches it without revealing what the internal path actually was.

unless i'm missing something this is fairly simple. You've got your old product pages indexed, and indexed well. The search facility is for visitors use only, and not beneficial to search engines in any way whatsoever. So, all you do (as you say ) is block the wildcard used in the search pages (search?query etc etc)in robots.txt . Then you manually remove all duplicated pages from google also using the wildcard which is returned in all the search generated urls'.

Its basic ecommerce problems and solutions.. all variant product pages should be blocked except one, the original one. You shouldnt have any other variant of any single product page - new search, old search - relist by price, list by manufacturer etc etc should all be blocked without exception.