For me, I think a consortium would work very effectively. Similar to how Compaq and company totally crushed the PS/2 databus in the late 80s by having a consortium that worked together to produce a competitive product.
If all of these search engines worked together to build a shared scraper they would have considerably more financial resources for it, and then they could also develop the API and back end infrastructure for it so that they can each provide their own algorithms and techniques to remain competitive to eachother. Do you think it's possible for DuckDuckGo et al to sit down together and actually try to do this?
I strongly agree that these walled gardens need to be handled somehow, but I think government intervention will lead to a government controlled walled garden which doesn't seem much better.
DuckDuckGo has gone on the record I believe and said that they don’t have the resources to do this and so that’s why they use Bing as a backend. I think it is possible for them and others to try a consortium but I don’t it’s likely that they will succeed in dethroning Google.
I think that’s the question here: who do you trust more? The government(s) or Google? I think that the walled garden Google has built around the open web is based on a natural monopoly on web crawling, which means that it’s economically more efficient for there to only be one entity engaged in that activity.
So, who would you rather have in control of web crawling? I’ve seen what Google has done to the open web with this power and I don’t like it and so I am willing to fight to give the government the chance at wielding that power to protect and develop the open web.
I won’t go so far as to say I wholeheartedly trust the government in all things, but rather that for this particular aspect of the economy, based on my research and classical economic thinking, that I would trust giving the government this power more than I would with letting Google keep it.
If all of these search engines worked together to build a shared scraper they would have considerably more financial resources for it, and then they could also develop the API and back end infrastructure for it so that they can each provide their own algorithms and techniques to remain competitive to eachother. Do you think it's possible for DuckDuckGo et al to sit down together and actually try to do this?
I strongly agree that these walled gardens need to be handled somehow, but I think government intervention will lead to a government controlled walled garden which doesn't seem much better.