Whatever the next version of information retrieval at internet scale is, I don't think it will involve indexing the web. I think the web's moment has passed.
Maybe the future is something like publishers supplying datasets to train models on? Or like how search was built on top of the web, it will be something more organic and ad hoc.
The internet is shrinking. The value of indexing the wider internet has less and less value to the average person. So much sits in walled gardens now and for many people that is perfectly fine. If it isnt in tiktok or instagram it doesnt exist.
Isn't this just a return to pre-90s internet? Back then the internet was only used by a few educated people. The masses watched TV etc to fill their time. Now it's tik tok etc. The early 2000s was an unusual time when more people actually used the internet, but it's going back to normal now.
The future of content is social media. Instagram Reels and TikTok. Snapchat and YT Shorts.
People don't want to read articles, they want to watch short form videos about what's interesting to them.
An interesting side effect is that I think this will lead to more direct-to-consumer sales. Customers will be led straight to your website instead of an Amazon order page.
It still needs to be indexed to be searched. Doesn't matter if it's articles, books, videos, posts, long or short.
> An interesting side effect is that I think this will lead to more direct-to-consumer sales. Customers will be led straight to your website instead of an Amazon order page.
That's an excellent outcome if it happens. Businesses should produce quality information and content related to their product category, instead of purchasing ads on other websites. There's a lot of good examples of this.
Maybe the future is something like publishers supplying datasets to train models on? Or like how search was built on top of the web, it will be something more organic and ad hoc.