Search Engine Programming?


Active Topics	New Topics	My Topics	Search	Profile	Inbox	Register	Rules/TOS

Add to hotlist

Mark This Forum Read

Printer Friendly Version

Email this thread to a friend

Sure Fire Search Engine Optimization (In: General Search Engine Optimization)

Search engine loophole? (In: General Search Engine Optimization)

Featured Web Site Template

Hundreds More at Free Site Templates.com!

Web Site Partners

Sponsored Links

Jet City Software

Whos Here ?

There are 0 guests and 1 members in the forums right now.

hunterz2007

Reflects user activity within the last 5 minutes

Moderator(s): Prowler, jcokos

Forum Index · Search Engine Forums · Building, Hosting and Maintaining Your Website · Coding & Databases - PHP, ASP, Perl, etc. · Search Engine Programming?

Previous Topic Next Topic

Member	Message
colt45 Joined: Mar 25, 2004 # Posts: 6	Posted: 2004-Jun-10 18:03 Hi All, I was needing help with a problem I am having and was hoping for some help? How do I get my linking program which is written in php to spider an entire site starting from the root to find my link and stop once it does or must stop after some time, if the link is not found? This is blowing my mind, any help will be greatly appreciated.. Thanks, Colt45 (Copy Shortcut)

Sinoed Joined: Dec 11, 2000 # Posts: 5266	Posted: 2004-Jul-04 13:49 Well what you're talking about is a script that would take a little bit of logic & time on your part to create. Essentially what you need to do is tell the spider to start searching for URL's. It would have to take in the input (aka. raw HTML if you're spidering the web) and find what it needs. In order to do that your spider would have to ignore everything but <a href=""> tags. Once it does that you need to tell it what to do if it finds a link - either it does something like displays a message or it continues on. If you were to create something along the lines of the following you could start by spidering one page. To spider multiple pages you'd have to save the URL's it found in a file or array. After it finishes each URL it would read a URL from the list or array delete it from the list and start the searching process again. I don't know whether PHP is capable of spawning multiple processes, I usually think of Java as a better choice for something like that. Code: [copy] // use some kind of form to allow a user to // enter in the name of the site you want to search <form name=spider action=spider.php method=post> Enter in URL: <input type=text name=site> <input type=submit value='Find URL'> </form> // while it hasn't found the url & 2 minutes haven't passed, search for $url $url = "http://www.someurl.com"; if ($submit) { -connect to site -use a function to strip out all of the extra characters; while (!$url $minsDiff != '2' ) { -read in text and look for $url; if ($url) } echo "Found the URL"; exit; } } } Here is function from www.php.net which calculates the elapsed time: Code: [copy] function calcElapsedTime($time) { // calculate elapsed time (in seconds!) $diff = time()-$time; $daysDiff = 0; $hrsDiff = 0; $minsDiff = 0; $secsDiff = 0; $sec_in_a_day = 606024; while($diff >= $sec_in_a_day){$daysDiff++; $diff -= $sec_in_a_day;} $sec_in_an_hour = 60*60; while($diff >= $sec_in_an_hour){$hrsDiff++; $diff -= $sec_in_an_hour;} $sec_in_a_min = 60; while($diff >= $sec_in_a_min){$minsDiff++; $diff -= $sec_in_a_min;} $secsDiff = $diff; return ('(elapsed time '.$daysDiff.'d '.$hrsDiff.'h '.$minsDiff.'m '.$secsDiff.'s)'); } It would help you to read a little bit about how a spider is written. There is a good article on it from Developer.com. Anyways, hope that helps get your mind around it a little. :) (Copy Shortcut)

mincklerstraat Joined: Eons Ago # Posts: 10	Posted: 2004-Sep-16 18:45 You could also try to hack an existing spider to do this work for you - phpdig comes to mind. However, this is a common enough sort of need for finding recips - maybe hotscripts would already have something, see if they have a 'link checker' or 'reciprocal link checker' section. I'd hate to start writing something like this from the start. What about all those DHTML and javascript links? What about redirects and all that? Yuk! (Copy Shortcut)

Previous Topic Next Topic

Forum Index · Search Engine Forums · Building, Hosting and Maintaining Your Website · Coding & Databases - PHP, ASP, Perl, etc. · Search Engine Programming?

You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:

You have not yet logged in, or registered properly as a member
You are a member, but no longer have posting rights.
This is a private forum, for which you do not have permissions.

If you are a recent member, it's possible that you simply have not yet confirmed your account. Please check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions contained within.

If you cannot find this message, click here to Re-Send it.

If you are still experiencing problem, please read the Login Assistance Article for some advice on what may be causing your login not to work properly.

Switch to Advanced Editor and ... Create a New Topic or Reply to this Thread

New posts

Forum is locked