This is an interesting idea and brings up a really good point about crawling being something that can lead to a fundamental monopoly when successful. However it misses that the storage, algorithms and indexing required for searching the crawled sites is fundamentally part of the development of the crawlers and the end search engine. If this was a forced monopoly there would suddenly be a situation where creative or new methods for crawling or differences in how the crawled sites are searched would be limited by the bureaucracy of the government monopoly.

Now what if a consortium of search engines worked together to create a direct competitor to Google. Could that work? And it would avoid the total centralization of the monopoly to a government agency?

Hello, author here. I can’t recall if anybody has tried the consortium idea but I would welcome it, the same way I would any new competitor to Google that leapfrogs them in someway. I’ve been doing this researching and raising the alarm about Google’s web crawling advantage because it’s something every search operator but Google (even Bing) complains about and offers up as part of the explanation for why Google does so much better. I am touch with many different people trying many different strategies to break Google’s hold on the internet, and I view my role as advancing one particular strategy that nobody else has really tried yet (government regulation of web crawling).

There’s only so much space in an op-ed column so I couldn’t get into the interplay between crawling and indexing but I’ve published much more about this at https://knuckleheads.club/introduction/ . Crawling doesn’t stand alone and the development of new methods of crawling goes hand in hand with new indexing and search features. But right now only Google gets to decide what those new indexing and search features are going to be because they are the ones with the dominant position in the market as well as with far and away the most privilege when it comes to website access.

What I would like the government to do is get Google to open up their web index so that innovation from outside Google can come in and bring competition back to the search market by pursuing new indexing and search strategies that Google hasn’t or won’t pursue.

For me, I think a consortium would work very effectively. Similar to how Compaq and company totally crushed the PS/2 databus in the late 80s by having a consortium that worked together to produce a competitive product.

If all of these search engines worked together to build a shared scraper they would have considerably more financial resources for it, and then they could also develop the API and back end infrastructure for it so that they can each provide their own algorithms and techniques to remain competitive to eachother. Do you think it's possible for DuckDuckGo et al to sit down together and actually try to do this?

I strongly agree that these walled gardens need to be handled somehow, but I think government intervention will lead to a government controlled walled garden which doesn't seem much better.

DuckDuckGo has gone on the record I believe and said that they don’t have the resources to do this and so that’s why they use Bing as a backend. I think it is possible for them and others to try a consortium but I don’t it’s likely that they will succeed in dethroning Google.

I think that’s the question here: who do you trust more? The government(s) or Google? I think that the walled garden Google has built around the open web is based on a natural monopoly on web crawling, which means that it’s economically more efficient for there to only be one entity engaged in that activity.

So, who would you rather have in control of web crawling? I’ve seen what Google has done to the open web with this power and I don’t like it and so I am willing to fight to give the government the chance at wielding that power to protect and develop the open web.

I won’t go so far as to say I wholeheartedly trust the government in all things, but rather that for this particular aspect of the economy, based on my research and classical economic thinking, that I would trust giving the government this power more than I would with letting Google keep it.