Open source, distributed computing, webcrawling search engine.. A DU team effort?

[The Vinyl Ripper]

If you have heard of other distributed computing projects like seti@home or folding@home then you will grasp what Grub's Distributed Web Crawling Project is about.

http://www.grub.org/

Grubby Lives!

Search is part of the fundamental infrastructure of the Internet. And, it is currently broken.

Why is it broken? It is broken for the same reason that proprietary software is always broken: lack of freedom, lack of community, lack of accountability, lack of transparency. Here, we will start to change all that.

Grub started back in 2000 with a simple concept of distributing part of the search process pipeline: crawling. In a way, we were a bit ahead of our time, but our intention then was what it is now. We want to help fix search.

Now, with the help of Wikia, community members, contributers, and Open Source developers our time has come again. Come be part of something greater. Come help us change the World.


Now I use Google a lot and I like it quite a bit. There is a problem with Google though, their algorithm is a closely held secret. By keeping their algorithm secret, Google can manipulate search queries into giving whatever results they wish to provide. Is Google honest and providing the best search results possible? Who knows, it's impossible to tell without knowing the algorithm they use.

Grub is an attempt to open up that secret algorithm and provide transparency to the search engine at last. The distributed computing part is where DU comes in, like seti@home, you download a software client that runs in the background on your computer. That client crawls the web from your continuously web connected computer, processes the data and sends the results back to the Grub server for compilation and analysis.

This is a project in the very early stages of getting started and I think that it has real potential to be a positive force in the online world of which we are all now a part.

I read some time back that seti@home is the most powerful supercomputer on the planet, it uses the spare computing cycles of tens of thousands of computers spread all over the world. If you have a multi gigahertz processor 99% plus of those billions of clock cycles are wasted while the computer waits for you to tell it what to do. With seti@home and now with Grub, those wasted clock cycles can be put to productive use.

I've been running the Grub client for a few weeks now since I first heard of it. Like any infant, Grub isn't much use for anything right now but with sufficient clients running it has the potential of becoming an extremely powerful and liberating tool on the internet.

I have started a Democratic Underground team on the Grub site I linked to above. I would like to invite DU members who have an interest to download the client and join the team. From what I've seen it wouldn't take too many DU'ers to race ahead of all the other teams.

If you decide that the distributed webcrawling project is not for you, all you have to do is uninstall the client program..

Let me know what you think..

I'm about to toddle off to bed, I'll check in later this morning.

[Greyhound]

1. Bookmarking for future participation, sounds like a worthy cause.

Don't have a working rig right now but will jump in when its built.

[The Vinyl Ripper]

2. Thanks for looking..

By the way, here is the URL to use the results from Grub so far.

http://www.grub.org/html/tools.php?op=wisenut

[133724]

3. GNU GRUB

GNU GRUB is a Multiboot boot loader. It was derived from GRUB,

GRand Unified Bootloader, which was originally designed and implemented by Erich Stefan Boleyn.

Briefly, boot loader is the first software program that runs when a computer starts. It is responsible for loading and transferring control to the operating system kernel software (such as the Hurd or the Linux). The kernel, in turn, initializes the rest of the operating system (e.g. GNU).
GRUB 2 and GRUB Legacy

Currently under development, GRUB 2, has replaced what was formerly known as GRUB (i.e. version 0.9x), which has, in turn, become GRUB Legacy.

GRUB 2 aims at merging sources from PUPA in order to create the next generation of GNU GRUB. A mailing list and a wiki have been setup for discussing the development of GRUB 2.


http://www.gnu.org/software/grub/

[The Vinyl Ripper]

4. Yes, I'm familiar with the boot loader grub..

I've torn my hair out a couple of times over that one. :D

[papau]

5. Sounds interesting - I'll check it out :-) n/t

n/t