Printer Friendly Version
Email this thread to a friend
|
Featured Web Site Template |
|
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
foxy1013
Joined: Sep 15, 2008
# Posts: 1
|
Posted: 2008-Sep-16 07:58
I am new here.
Sorry about the little offtopic but I have no one else to ask.
I am doing a research about spiders.
And i need to know if a company offers access of spidered data of the pages in the world. If so give me some links.
Or what would be needed to start your own crawling server? What software, machines, hard drive? What database to expect?
Thank you for your answers in advance.
|
 |
Prowler
Staff
Joined: Aug 14, 2000
# Posts: 1788
|
Posted: 2008-Sep-16 09:35
>>And i need to know if a company offers access of spidered data of the pages in the world. If so give me some links.
Google, MSN and Yahoo.
>> what would be needed to start your own crawling server? What software, machines, hard drive? What database to expect?
It all depends on your objectives. If you want to crawl a few sites, you could start with a Linux server with a spider written in Perl using Mysql as the database. If you consider crawling all the pages in the world, you won't be asking this question here.
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10418
|
Posted: 2008-Sep-16 13:16
Cuil spent some 50 million, or something, and the result of that is poor to terrible.
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
