More Virtual Promote ... Gazette · Webmaster & SEO Tools · Scumware.com · Free Website Templates

SEF

Search Engine Forums
Helping to make the Web - Since 1998
Virtual Promote Member Spotlight
A Forum Software Solution

A flash based forum for your website. (barb478)

Login Password Forget your password?    Trouble Logging In?
.
Forums Index Active Topics New Topics My Topics Search My Profile Register Inbox   Rules & TOS
.
 
Forum Index · Search Engine Forums · SEF Community & Networking · Members Lounge · Pet Spider - where can I get one?
 
Add to hotlist
Reply to this thread Create a New Topic in this forum
Mark This Forum Read
 Printer Friendly Version Print this thread
Email this thread to a friend eMail this thread to a friend  
Moderator(s): excell, SportsGuy, g1smd
Previous Topic Next Topic
Member Message

SportsGuy
Moderator
Joined: Aug 30, 2002
# Posts: 3496

View the profile for SportsGuy Send SportsGuy a private message

Posted: 04/28/2008 10:00 am
Edit Message Delete Message Reply to this message

So, say a person's looking for code to build a crawler.

Nothing fancy, just a basic crawler that you can:

1 - feed a select list of links into to tell it to start crawling from (think the main page of different sites)
2 - tell the spider to watch for select keywords (give it a list to look for)
3 - index only those with the keyword/phrase mentioned
4 - spec whether the mention happens in the metas, the body, the domain, the URL, etc.

Assume the desire is for a crawler that will live on a server and can be given it's own IP if absolutely needed - MySQL database friendly and, thus, likely php-based, or at least Linux box friendly.

...and it's care-taker is not a programmer, so this pet spider will need to come with instructions. wink



Quadrille
Insider
Joined: Nov 15, 2000
# Posts: 785

View the profile for Quadrille Send Quadrille a private message

Posted: 04/28/2008 11:27 am
Edit Message Delete Message Reply to this message

I really don't know ... but there must be an open source answer?



formerskeptic
Joined: Oct 05, 2001
# Posts: 265

View the profile for formerskeptic Send formerskeptic a private message

Posted: 04/28/2008 10:26 pm
Edit Message Delete Message Reply to this message

Pretty tall order you got there smile Try Risearch.
The script has most if not all the features you're looking for wink


 
Forum Index · Search Engine Forums · SEF Community & Networking · Members Lounge · Pet Spider - where can I get one?
Who's Online?
There are 272 guests and 1 members in the forums right now.
Reflects user activity within the last 5 minutes
Previous Topic Next Topic
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
  1. You have not yet logged in, or registered properly as a member
  2. You are a member, but no longer have posting rights.
  3. This is a private forum, for which you do not have permissions.

If you are a recent member, it's possible that you simply have not yet confirmed your account. Please check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions contained within.

If you cannot find this message, click here to Re-Send it.

If you are still experiencing problem, please read the Login Assistance Article for some advice on what may be causing your login not to work properly.

Switch to Advanced Editor and ... Create a New Topic or Reply to this Thread



Related Forum Topics
  1. Yahoo releases a new web page spider (In: Yahoo!)


© 1995 - 2006  ·  iWeb, Inc  ·  DBA JimWorld Productions