A well written perl program, running under apache with mod_perl can achieve results in less than a second (ie: www.hitsgalore.ws)

Poortly written perl programs can be hellaciously slow. Same goes for C... the general idea is that C is faster, but it really comes down to HOW the program was written, not necessarily the language.

I can make perl outperform C 3:1, and can also do likewise the opposite way....

I would love one of those, I don't know what I would do with it though, But it helps to explain why they are so fast.

My goal is to ensure that a moderatly configured system can handle a gigantic database, and a pretty good load, and still have an average return of < 1 second on any given search.

How do you achieve this ?

It comes down to database design, number one, and indexing algorithms. You start with a database that's designed to hold a lot of data, but still be indexed and indexable for searching. A well designed database can return 200 results from a 30 million keyword database in the same time it takes to return 1 result from a 2 record database. It all comes down to indexing, IMO.

Starting with a solid DB Plan, you can rely on the SQL Engine to do it's job efficiently, and your Perl or C++ merely has to make the Query and spit out the results.

John

I can push it well beyond 100,000 if I want to, even simulate 100 or more simultaneous searches per second.

Drop me an email, and i'll send you a copy.

John

mySQL blows Oracle away. Period. I have a customer that runs both, and his mySQL version of our program (Jackhammer) can handle almost 3x the load. Searches are faster, connections are more stable. It's just a better system. The downside to mysql is that it doesn't support all of the newer SQL things like procedures and triggers, but pound for pound, it kicks serious butt.

We've got a few Jackhammer users that are serving upwards of 120 million searches per month, without a problem. One is using mod_perl, the other SpeedyCGI. The SpeedyCGI seems to be much better in terms of the load and memory usage.

Hope that helps.

Could you be so kind enough to elaborate on turbo linux and especially zeus, These are 2 are new to me as I was used to sun solaris/red hat linux. D these 2 work in combination or just on their own, also what makes zeus the 'rolls royce' of server OS's?

thanks

The thing about turbolinux that I love is that it's very tuned for speed, and very stripped down. Meaning, it's a TINY kernel, with no puff, and nothing you don't need to run a website (like X, etc). Zeus is an apache replacement that outperforms apache by a 3:1 margin. It's costly, but if your serious, very worth it.

John

Hyperseek is fastCGI ready (and mod_perl & CGISpeedy ready), out of the box. Not sure about LinksSQL. I know that Alex loves mod_perl and LinksSQL runs under mod_perl quite well, but I don't know about the other speedup solutions, you'd have to ask him directly about them.

Perl is interpreted, C is compiled... and you can't beat compiled software.

As for Google's speed, no doubt every piece of code was written by Google staff from the ground up, including their database software. I can't imagine Google using commercial (or open source) software there, especially with their unique use of hardware - namely an entire index stored in DRAM (rather than hard drive) distributed amongst 10-20 thousands machines.

There's a lot of information out there on Google's set up thanks to interviews with Google's technical staff... it's worth a look.

quote:

---

Perl is interpreted, C is compiled... and you can't beat compiled software.

---

Incorrect. Granted, perl is interpreted, but when you write perl programs, for apache, with mod_perl/fastCGI/CGISpeedy in mind, you can achieve 3:1 performance gains over C.

Simply put, all of these technologies precompile the perl program into memory, where they stay, persistantly. (Each one does it a different way). The benifet is that when someone from the browser hits one of these "sped up" programs, unlike normal perl or c, apache doesn't need to load & compile the perl, and unlike C, it doesn't even need to run the program at all.

All of the startup overhead is gone, so your application is better than instant.....

Trust me, a simple "Hello World" written in perl vs the same thing written in C runs 2x faster under mod_perl. If you make the application more complex, and throw in database connections, big queries, latency, etc, the gap widens,and perl really kills C... Straight perl vs c is no contest, c wins that hands down.

I've done LOTS AND LOTS of benchmarking on this stuff....

As for google, I think it was written in C, by a group of college kids as a class project, and then one thing led to another...

Yes, Hyperseek runs perfectly with Turbolinux and Zeus (That's what I run on my development machines).

Zeus claims a huge speed improvement over apache for normal .html and images too. Not sure how they do it, or how to measure it at low levels, but it sure feels fast

John

A few months ago I read that Google was predominantly powered by software written in C++ ... ?