tleach
Joined: Oct 09, 2003
# Posts: 8
|
Posted: 2006-Sep-24 03:17
In March 2006 our eCommerce site was moved to a new host/server. Since then, our rankings in Google have fallen so far as to not even be relevant. Prior to the move we were ranked very high in our market. Oddly enough, our ranking in Yahoo is excellent while MSN is almost non-existent. We did not change a thing on our site, so it has been frustrating trying to figure out.
Is it possible when the site was moved to the new server and the host could have altered something that now causes google to rank us differently. I am confused by all the discussion between www.domain.com and domain.com settings - not sure if that could have been modified in any way.
We also use SiteDirector, a program that manage HTTP 404 Errors on our site. Catch 404 page not found errors (HTTP Error 404) and instantly redirect to a 404 error page we specified while it records the referring site and page requested. We have used this program for almost 2 years so I cannot understand how this caused any harm
We still seem to see about the same amount spidering on our site (yahoo spiders, approx 32k hits; msn spiders, approx 24k hits; googlebot, approx 18k hits) since we first started keeping score.
We are careful not to stuff our code with keywords or any other unethical acts. We are getting desperate and are all ears (eyes) for suggestions.
Thank you in advance.
|
 |
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10465
|
Posted: 2006-Sep-24 14:04
Can you run Xenu LinkSleuth over your site and see if it has any problems?
|
|
tleach
Joined: Oct 09, 2003
# Posts: 8
|
Posted: 2006-Sep-24 18:54
I donwloaded Xenu LinkSleuth and hit start, and within a couple of minutes was up to 120,000 links being analyzed...and growing. It appears it is creating new user sessions for every product link and placing them in the basket, then proceeding through checkout. The last time I attempted something like this I ended up with thousands of abandoned baskets worth millions of dollars. It seemed like an endless loop, so I abandonded the process. I have done the same with Xenu LinkSleuth since it appears this could be the never ending analysis.
Another thing I noticed last night was that when they moved my site I ended up with a robot.txt file (not robots.txt) that only had the following
User-agent: *
Disallow: /
I have replaced that with my old robots.txt file, although I can't understand how this could be part of the root cause of our problems.
I recently signed up for google webmaster tools which gave me access to an index summary page that indicated...
Pages from your site are included in Google's index. See Index stats.
Googlebot last successfully accessed your home page on Sep 19, 2006 .
We may not know about all the pages of your site. Submit a Sitemap to tell us more about your site.
You may want to take a closer look at:
Status Distribution
HTTP errors (0)
Not found (0)
URLs not followed (0)
URLs restricted by robots.txt (0)
URLs timed out (0)
Unreachable URLs (66)
I have not had time to absorb all that google webmaster tooks is telling me, but at first glance I took it as good news.
I appreciate very much the quick response. Afer having our site in the top 10, worst case top 20 for almost every search term we were interested in, resent results are driving us crazy.
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10465
|
Posted: 2006-Sep-24 19:30
>> User-agent: *
>> Disallow: /
Ahem! That says "index NOTHING from this site".
Fix that quick.
>> It appears it is creating new user sessions for every product link and placing them in the basket, then proceeding through checkout. <<
Bots should not be able to get to basket pages at all.
You need to fix the site design to stop that happening.
|
|
tleach
Joined: Oct 09, 2003
# Posts: 8
|
Posted: 2006-Sep-25 03:39
I figured out the meaning of the robot file, but hoped the fact that it was a robot.txt (no 's' on the end or robot) instead of robots.txt was causing the file to get overlooked by the spiders. Also, our site stats indicate the googlebot visits our site on a fairly regular basis.
I checked our site stats today and their are no abondoned basket because of the Xenu LinkSleuth program, however I am confused why the program is listed over 150,000 links being checked when we only have about 12,000 products. I let it run a bit more, but it adds new links at a rate of about 100:1 for each link verified. I am very green about the process being performed, but it seems like the program is in a never ending loop.
Is the number of links abnormal for a eCommerce site with 12,000 products or is there something else I should be looking at?
I truly appreciate the comments - I am so far out of my league here and don't know whether I have a SEO issue of programming issue.
Thank you
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10465
|
Posted: 2006-Sep-25 11:05
I missed that you said robot.txt; that one is probably ignored by search engines.
As for your large page count, this points to a duplicate content problem of some sort. If it was affecting Xenu LinkSleuth in that way, then I am sure that Google is probably hitting the same problem.
You need to analyse exactly what all of the URLs were and then restrict access to all but the canonical form for each page.
|
|
tleach
Joined: Oct 09, 2003
# Posts: 8
|
Posted: 2006-Sep-26 03:45
Thank you for your response. I looked at the log from Xenu LinkSleuth and it had 1885 links to a single category which only has one product in it. Each link in the log had a different userID and sessionID, but the www.domain.com/category.aspx was identical.
That makes me ask two questions - does google and other spider bots ignore the user and sessionIDs...and, if they do not, could google being seeing the same page on my site 1885 times? That could not be good.
[ Message was edited by: tleach 09/26/2006 03:01 am ]
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10465
|
Posted: 2006-Sep-26 20:01
They see it as 1885 duplicate copies of the same page... and you end up with a very big problem.
Bots must NOT see URLs with session or user IDs in them. That needs fixing immediately.
|
|
vanachte
Joined: Feb 10, 2004
# Posts: 405
|
Posted: 2006-Sep-28 20:20
Because they had no problems prior to the host move, it is possible that the new IP being used for the URL is causing problems?
Is it possible that the NEW IP was previously penalized for one reason or another, or that the new IP is in a bad neighborhood, sharing the server with other spammers?
Just a thought seeing as how the spidering of 100:1 was likely an issue prior to the move also.
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10465
|
Posted: 2006-Sep-28 20:28
Check the cache dates for those pages to see how long this has been going on.
|
|
tleach
Joined: Oct 09, 2003
# Posts: 8
|
Posted: 2006-Sep-29 00:23
Is it possible to find the list of pages visited by google so that I can see if google is showing duplicates? So far my only indication this could be happening is the use Xenu Sleauth.
I am asking similar questions on our shopping cart provider's forums and have been told when the shopping cart recognizes that a visitor is a spider that no userID or sessionID is generated. Others on the forum tried using Xenu Sleuth and found the same problem with duplicates that I am finding and yet the are getting good google results.
The IP address comment is interesting though since I am on a shared server and have no idea who my neighbor is. I will try to get additional info from the host to see if that is a possibility.
Thanks again
|
|
vanachte
Joined: Feb 10, 2004
# Posts: 405
|
Posted: 2006-Sep-29 00:42
A list of pages VISITED by Google will be in your sites raw log files.
You can view a list of pages INDEXED by Google by "site:www.domain.com"
This will only return 1000 results at most.
Not sure if the IP could be the issue, but I am sure it is possible if other sites are banned...
You can check what sites are at your IP.
First find your IP.
Easiest way to do this is open a command prompt and type
"ping www.domain.com"
Then take the domain and go to MSN and do a search for the IP
search: "IP:xxx.xxx.xxx.xxx"
This will give you a list of all sites MSN knows about hosted at that IP. May give you some insight.
|
|
neilt
Joined: Aug 29, 2005
# Posts: 169
|
Posted: 2006-Sep-29 09:40
I'm thinking of moving my site (not the one in my profile) from a linux server to a windows host. What should I look out for when moving the site. As it is with the same ISP I plan to keep the same IP address. Is this a good idea ?
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10465
|
Posted: 2006-Sep-29 22:10
You have a lot more control over your site if you use Apache.
I would recommend staying with Apache so you have control over redirects and canonical URLs for your content.
|
|
vanachte
Joined: Feb 10, 2004
# Posts: 405
|
Posted: 2006-Sep-29 23:58
I totally agree - stick with Apache. I have heard many complain about Windows when they suddenly have to do 301's or mod rewrites... what a pain. Apache makes it "easy" relatively.
|
|
neilt
Joined: Aug 29, 2005
# Posts: 169
|
Posted: 2006-Oct-02 10:47
The problem we have though is the downtime our site is experiencing with the current host (server related issues).
We are looking to move to our own dedicated Windows server with the same host, which we do not have any problems with.
|
|
vanachte
Joined: Feb 10, 2004
# Posts: 405
|
Posted: 2006-Oct-03 23:06
Downtime... that could kill your rankings in itself. If Google comes to visit your site and it is down, you're in trouble. If it happens once, no worry, but if it keeps happening look out!
|
|
tleach
Joined: Oct 09, 2003
# Posts: 8
|
Posted: 2006-Oct-08 02:50
I have learned a bit more about xenu sleath and found I could enable cookies, which eliminated the user and session id's that were driving me crazy when I first started. The new results have caused me more questions.
For every product on our site, we assign it to at least 2 categories and as many as 10 (or more). As an example, every product is assigned to a manufacturer category and a product type category. The software then assigns two different static url's to distinguish between the two categories, ie "Garcia Open Face Reel Model 2309-c2489" and "Garcia Open Face Reel Model 2309-c2573", with one located in a category called Garcia Reels and the other in a category labeled Garcia Reels for Freshwater Fishing".
The two page outputs are exactly the same, with the only difference being the static page url that is created by the eCommerce softare. Like I stated earlier, in some cases there may be 10 instances of the same product with only the url having a different category id on the end.
For the last 3 years of using this eCommerce package I have had products in two categories, which means I always had two identical product pages with two different url's on our site. Almost a year ago I tripled or quadrupled the number of categories which resulted in the duplication of identical pages being spidered. I have coorelated the timing of this change to the reduction of referrals from Google and MSN, although Yahoo doesn't seem to have been affected.
Now to the question...can I change all the links on my home page to "no follow" and only leave a link to our sitemap directory to prevent search engines from spidering the entire site and finding the duplicate pages? Our eCommerce package creates a sitemap that lists each category and product just once, which would prevent the spider from seeing duplicated content.
If this would work, this would be the easiest way to fix the problem for now. Any comments would be most appreciated.
Thanks in advance,
Tim
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10465
|
Posted: 2006-Oct-08 18:18
It would be easier to dynamically put a meta robots noindex tag on each page that should not be indexed. It will add a couple of lines of code to your scripting.
|
|
Inbox
