Printer Friendly Version
Email this thread to a friend
|
Featured Web Site Template |
|
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
Dinkar
Staff
Joined: Aug 12, 2001
# Posts: 4391
|
Posted: 2003-Jun-14 20:02
I agree that a good programmer can write an auto-rank checking software which an engine can't recognize today. But you can't guarranttee about tomorrow. Tomorrow, that engine can recognize it. It can easily done by renaming the search script. So next time, when you will run the auto-rank checking software, it will generate 404 error and that engine will know who is using the well-written piece of auto-rank checking software.
|
 |
PhilC
Joined: Eons Ago
# Posts: 75
|
Posted: 2003-Jun-14 22:54
I take it you mean that by renaming the search script, the search URL will be altered. If so, that needn't show up an auto-searcher. The program I use now doesn't do this, but the first one I wrote went to great lengths to emulate a person. The first thing it did was get the engine's front page. Then it parsed the page for the search form elements and constructed the search URL from them. The following searches parsed the results pages for the form elements, "next", and so on.
If the HTML changed, the search didn't/couldn't go ahead. Changes to the search script name or to the form elements were reflected in the URL it produced.
The point is that browsers construct the search URL from the form elements on the page and, if browsers can do it, then an auto-search program can do it.
[ Message was edited by: PhilC 06/14/2003 03:00 pm ]
|
|
OAC
Staff
Joined: Jan 25, 2001
# Posts: 6805
|
Posted: 2003-Jun-14 23:18
Have you factored the Google Toolbar into your thinking yet?
|
|
PhilC
Joined: Eons Ago
# Posts: 75
|
Posted: 2003-Jun-14 23:40
Nope - not into any of my programs, but it has crossed my mind. I do realise that those of us who have the toolbar installed are actually running a Google program on our machines, therefore, Google could underhandedly write the program to check what we do on and with our computers, but I really don't think they would do that. It really would be naughty.
The main thing with the toolbar is that people on fixed IPs can be seen to usually start searching via the toolbar (a pattern), and using an auto-searcher wouldn't do that. There is probably a difference between manual and auto in that repect but I havn't checked it out to be sure. However, people with the toolbar don't always begin their Google searching in it, I know I don't, so Google still couldn't say that those that are not started via the toolbar are auto programs.
Even so, I didn't say that an auto-searcher is undetectable under all possible uses; just that it can be written to be undectable in itself. I can set my program to search much faster than by hand, or to perform simultaneous searches on the same engine, in which case it's my settings that allow it to be detectable. That's my fault and not the fault of the program.
[ Message was edited by: PhilC 06/15/2003 03:32 am ]
|
|
Dinkar
Staff
Joined: Aug 12, 2001
# Posts: 4391
|
Posted: 2003-Jun-15 05:57
The first thing it did was get the engine's front page. Then it parsed the page for the search form elements and constructed the search URL from them. The following searches parsed the results pages for the form elements, "next", and so on.
Good  That should solve the renaming problem.
I have little programming knowledge and I would like to discuss this further.
So next problem is cookies. Are you considering cookies? SEs can track auto quries by using cookies.
|
|
PhilC
Joined: Eons Ago
# Posts: 75
|
Posted: 2003-Jun-15 11:28
Cookies can be turned off in normal browser useage, so searching without cookies turned on is a normal way of searching. Therefore, an auto-search program doesn't need to accommodate them.
It's up to the program (browser, auto-searcher, etc.) whether or not it sends a website's cookie to the website when it makes a file request. The fact that one isn't sent with the request is not an indication that it isn't a real person at the wheel.
I doubt that any auto-searcher goes to the lengths of supporting cookies because (a) it isn't necessary and (b) as you said, they are a means of tracking.
[ Message was edited by: PhilC 06/15/2003 03:33 am ]
|
|
OAC
Staff
Joined: Jan 25, 2001
# Posts: 6805
|
Posted: 2003-Jun-15 12:30
"I didn't say that an auto-searcher is undetectable under all possible uses"
You said "It isn't possible for an engine to recognise that *any* well-written piece of auto-rank checking software is being used." and " it is possible ... to write an auto-searching program that cannot be detected by any search engine, including Google."
If those statements don't indicate you meant under 100% of circumstances then I don't know what does. Now you changing your mind and saying there are some circumstances where auto searchers are detectable. At last we agree - your "undetectable" automated rank checker is detectable, under certain circumstances.
The remaining matter you now need to get a grip on is just how extensive those "certain circumstances" are. I can't believe that you don't really know what information you give away when using the full version of the Google toolbar, but if you are serious, and you don't, then I would suggest you read that Toolbar agreement, to which you agreed, when you asked Google to install it on your machine(s). Google doesn't have to be underhand - they tell you, in no uncertain terms, what information you are giving away. Your toolbar has a unique identifier. If you check any page of your site, in IE, at any time, they know.
There is a lot more, but I am not going into any more details - it's about time you worked it out for yourself. There is no such thing as an undetectable automated rank checker.
For the benefit of other members who are getting worried about the above because they use an automated rank checker, don't be - unless you are a heavy user they don't have the time, the resources or the inclination to go after you. You aren't the problem - it's those that make egregious use of such tools and burn up their bandwidth and searching resources, that cause them heartache. OK?
|
|
PhilC
Joined: Eons Ago
# Posts: 75
|
Posted: 2003-Jun-15 13:08
I think you're splitting hairs, OAC. The fact that my auto-searcher's settings can be set so that it is detectable is irrelevant. A programmer can leave out such options, so that nothing can be set that would make it detectable - under 100% of circumstances. So we don't agree after all.
Regarding the Toolbar and the passing of information to Google. Again, it isn't relevant to this discussion, because it has nothing to do with auto-search programs, or any other program except Internet Explorer. Auto-searching programs are completely seperate from, and have nothing whatsoever to do with, Internet Explorer, any other browser, or the Google Toolbar. The Toolbar has nothing whatsoever to do with any program except Internet Explorer. I think you are confusing something here.
|
|
OAC
Staff
Joined: Jan 25, 2001
# Posts: 6805
|
Posted: 2003-Jun-16 00:54
No, this gets back to pattern matching. I think you need to think about it some more.
However, I am glad you raised that point. In relation to pattern matching, you said: "I can also see how to program around it".
So, please let us know how you will/could program around it.
|
|
Dinkar
Staff
Joined: Aug 12, 2001
# Posts: 4391
|
Posted: 2003-Jun-16 05:20
IMO, you should handle cookies if you want to make 'toughest to trace' software.
Ok, my next question. I hope you won't mind
Are you handling referral URL and browser info (browser name, version)? Or you think that it is not important?
|
|
MJR
Joined: Dec 12, 2000
# Posts: 2163
|
Posted: 2003-Jun-16 07:54
hmmm....provocative reading
|
|
PhilC
Joined: Eons Ago
# Posts: 75
|
Posted: 2003-Jun-16 12:53
No, this gets back to pattern matching. I think you need to think about it some more.
I think the only way that an auto-searcher could be 'assumed' (not determined) is that an auto-searcher wouldn't request the PR for each returned page, whereas the Toolbar does. Whether that's what you meant or not, it's a very good point. My first thought is to get around it by using a Netscape User-Agent - pretending to be Netscape.
So, please let us know how you will/could program around it.
There are a number things that can be written into a program to prevent a pattern occuring. E.g. don't use the same order of search each time, don't stop the depth of searches at the same places for each search each time, throw in a random sprinkling of searches that aren't wanted but which break up any pattern, randomly go back to Google's front page, randomly click on various links that Google returns (cache, directory, similar pages, whatever), I previously mentioned putting in random delays between searches, throw in the occassional "link" search for randomly selected URLs from the serps, etc, etc, etc. There are all sorts of things that a program can do to prevent a pattern being established. Don't do all of them every time the program runs, but randomly select a few each time.
|
|
PhilC
Joined: Eons Ago
# Posts: 75
|
Posted: 2003-Jun-16 13:09
I don't mind any questions Dinkar
I think I should mention that I'm only trying to make the point that auto-searching programs can be written to be wholly undetectable; I'm not saying that I've written or use one. In fact, I said earlier that the program I use now doesn't implement all the ideas that I'm throwing in, and also that there may be no publically available auto-searcher around that has been written well enough to be undetectable.
Are you handling referral URL and browser info (browser name, version)? Or you think that it is not important?
I think that, for any auto-searcher to be undetectable, it really should look like a typical, existing browser. That includes passing referral and user-agent information.
I don't believe that using (by hand) a self-written or rare browser that is not recognised, and that may or may not pass certain information with the request, could cause anyone to be penalised for using auto-software, when they obviously aren't. So an auto-program that doesn't behave exactly like a known browser would almost certainly be fine. However, because it would have to camouflage itself in other ways, it's probably best not to draw attention to itself by not behaving like a standard browser.
|
|
OAC
Staff
Joined: Jan 25, 2001
# Posts: 6805
|
Posted: 2003-Jun-16 13:29
Those will result in the cost of detection being greater but won't make the checker undetectable.
The not requesting PR would only a minor possible confirmation as neither Opera, Mozilla (well, until recently) and Netscrape don't have toolbars.
No, that is not the direction of my thinking. Let's get realistic about this issue. The fact is, Google isn't concerned by relatively low volume use of automated rank checkers, so using one without any of the protection you propose, is going to have no effect for low volume use.
You can't write an automated rank checker which is, in practice, undetectable, at high volumes, especially when the high volumes include large numbers of repetitions. All those features you mentioned just make it a bit less easy to detect. Undetectable, no.
And the users they want to stop are the high volume users.
Your claim was "It isn't possible for an engine to recognise [the use of] a well-written piece of auto-rank checking software."
I'm saying, they don't care when the volumes are low but they can detect high volume use of an automated rank checker, if they throw enough of their resources at it. Unlike their competitors, Google actually posseses the resources necessary to accomplish this task. Because such a task chews up resources, they will, not do this very often. however, they have traced and tracked such usage and then applied penalties against sites associated with high volumes of queries.
I am not arguing that you can't make it difficult - just against your claim that an undetectable automated ranking tool can be written (for the people who need one - i.e high volume users). And if it can be detected under even very limited circumstances of high volume use, then it is not, and never was, undetectable. Difficult to detect maybe, but undetectable, no.
|
|
OAC
Staff
Joined: Jan 25, 2001
# Posts: 6805
|
Posted: 2003-Jun-16 13:41
PS. The pattern matching defeating mechanisms you propose, only cover part of the problem. You need to think more about pattern matching, high volume use and the toolbar.
You could add other features to make it even more difficult to trace, like munging your IP, but in the real world, your application is going to have to be reasonably simple to configure and operate. If it is not, you can bet the bank that it won't be long before someone screws up in your organization and doesn't follow the "security" procedures. And then it is all over, red rover.
|
|
PhilC
Joined: Eons Ago
# Posts: 75
|
Posted: 2003-Jun-16 14:04
Now it's my turn to split hairs
Your claim was "It isn't possible for an engine to recognise [the use of] a well-written piece of auto-rank checking software."
I didn't actually use the words "the use of" as you correctly indicated. I merely stated the above quote, without the addition, and I stick by it.
I agree that heavy use of an auto-searcher would be assumed to be an auto-searcher, but I maintain that it isn't the program that would be 'caught', but the excessive use of it. The program itelf can remain hidden from detection. The best that Google could do is 'assume' it to be a program at the other end, and they would make that assumption because it is highly unlikely that a person would sit at a computer and spend hours upon hours performing searches on Google. It's a matter of probability, and not a matter of certainty. It isn't the checker that's detectable, it's excessive useage that gives rise to a high probability - not a certainty.
It looks like we're going to have to agree to disagree, OAC.
|
|
Ron Carnell
Joined: May 15, 2001
# Posts: 206
|
Posted: 2003-Jun-16 17:06
Aren't semantics fun?
It is not terribly difficult to write software that is capable of fooling other software. If Program G is following an algorithm (as ALL software must), then its responses will always be 100 percent predictable, and Program X need only anticipate each of those responses. That can be a little boring to program, but it certainly isn't difficult. Everyone here probably already has an auto-rank checking program that does this pretty well. We just happen to call them browsers. And if you think your browser isn't an auto-rank checking program, try doing it without a browser some time and you'll find it automates the procedure a lot. Program X need be nothing more than a slightly different kind of browser to completely fool Program G (one that can easily be loaded without a Google Toolbar, if push comes to shove).
Semantically, a "well-written piece of auto-rank checking software" would, by MY definition, eliminate the possibility of excessive usage. If it didn't spread its queries across time spans similar to those of a normal browser, it would no long qualify as well-written. We don’t need fast, because typical results don't change frequently. We just need automated. (If you are a professional SEO and have tons of results to check every month, you can just buy more copies of Program X and run them simultaneously on more machines. It'll help SEO's get used to the idea that larger business have more overhead than smaller businesses, something too many startups fail to consider.)
Does that mean that Program X can necessarily fool Google 100 percent of the time? Not by Alan Turing's definition, because there isn't a program written yet than can consistently fool people.
As easy as it would be to write Program X, it would be even easier to modify Program G to thwart X's efforts. Computational pattern matching (as opposed to human pattern matching) is still in its infancy, with far more failures than successes, and requires enormous resources to be even marginally useful. It would be a bit like using a 357 Magnum to kill flies (and probably about as effective, too). But Google need not even resort to such "shock and awe" tactics to thwart all attempts at automated rank checking software. Submitted an URL to Alta Vista lately? Had to type in their graphical Submission Code to do so? This is only one example of many different ways Google could completely shut down auto-checking programs.
But the question isn't what Google (the people) can do, but rather what they are likely to do.
I doubt we'll see them seriously tracking IP patterns on searchers right away, at least not until the price of a Cray drops a bit more. I doubt we'll see them adding graphical Submission Codes to their screens, because they care far too much about the user experience. But Google's current penalization policies should be taken as a warning, because I also doubt they'll forever ignore the problem should it truly become a problem. Writing Program X, to completely fool Program G, would be easy. Multiply X by tens of thousands of users, however, and that problem will both surface and -- ultimately -- be solved.
So, in a nutshell, I think both points of view are right. We're just tripping over the semantics.
|
|
alex
Joined: Nov 21, 2003
# Posts: 15
|
Posted: 2003-Jun-18 04:22
Google says it won't penalize you for bad neighbours incoming links because they are 'beyond your control'
So if I use software to monitor se rankings, why should it be different? If my IP address is different from my website's address... what can google do? What if I run the program to track my competitors ranking twice a day? Is google going to penalize them? I don't think so!
can you help me with this doubt?
thanks for this useful forum
|
|
PhilC
Joined: Eons Ago
# Posts: 75
|
Posted: 2003-Jun-18 11:26
It's not the website that's at risk, it's your ability to search on Google. They can block your IP from searching.
|
|
igneus
Joined: Eons Ago
# Posts: 2
|
Posted: 2003-Jul-26 01:09
I have written a batch Google Page Rank processor which is virtually undetectable by Google. It's currently able to do 10 checks/second, or 600 domain names/urls per minute.
Please PM for details
[ Message was edited by: unreviewed 07/26/2003 05:26 am ]
[ Message was edited by: igneus 07/27/2003 10:58 am ]
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
