YouTube just happened to show me Gamers Nexus Lossless Scaling & Lossless Frame Generation video just now. [1]
Which, I can't help but wonder. If this ran on the Frame... To aid in streaming from your desktop... In conjunction with Foveated Streaming...
What's even more bonkers to me is the idea that this could really be baked in to the streaming, if you think about it.
Server generates 100 beautiful frames in one second.
The streaming hardware reminds the server that with the current network conditions, that it can only handle X bits per second.
The Frame says, "Hey, here's where the eyes were looking, here's where they are looking, here's where they're projected to look."
The Server knows how many actual frames it can send in a second.
But the Server can run the same Lossless Scaling & Lossless Frame Generation that the client will run.
The Server can then compare the predicted frames against the actual frames, and it can send the delta needed to correct the predictions. Especially in areas close to where the eyes are looking.
The Client consumes the stream, predicts frames, applies the delta, and ends up with better framerate and quality...
Can you do this all low latency, too? I suspect so.
Add it all up, and this is pretty bonkers. I mean, it's kind of obvious that that's how streaming "should work," but I think I hadn't really put it all together yet in my mind. My mental model was too "off the shelf." It didn't include how much effort the client could do... And that the server could then also do, to help the client... It's really a beautiful pairing, when you think about it.
What's really bonkers is that Steam could learn what final images are supposed to look like, for a given game, and train AI specifically for that game. Update the server and the client, and bam, everyone gets better quality. With zero developer interaction.
Which, I can't help but wonder. If this ran on the Frame... To aid in streaming from your desktop... In conjunction with Foveated Streaming...
What's even more bonkers to me is the idea that this could really be baked in to the streaming, if you think about it.
Server generates 100 beautiful frames in one second.
The streaming hardware reminds the server that with the current network conditions, that it can only handle X bits per second.
The Frame says, "Hey, here's where the eyes were looking, here's where they are looking, here's where they're projected to look."
The Server knows how many actual frames it can send in a second.
But the Server can run the same Lossless Scaling & Lossless Frame Generation that the client will run.
The Server can then compare the predicted frames against the actual frames, and it can send the delta needed to correct the predictions. Especially in areas close to where the eyes are looking.
The Client consumes the stream, predicts frames, applies the delta, and ends up with better framerate and quality...
Can you do this all low latency, too? I suspect so.
Add it all up, and this is pretty bonkers. I mean, it's kind of obvious that that's how streaming "should work," but I think I hadn't really put it all together yet in my mind. My mental model was too "off the shelf." It didn't include how much effort the client could do... And that the server could then also do, to help the client... It's really a beautiful pairing, when you think about it.
What's really bonkers is that Steam could learn what final images are supposed to look like, for a given game, and train AI specifically for that game. Update the server and the client, and bam, everyone gets better quality. With zero developer interaction.
[1]: https://www.youtube.com/watch?v=8w_Hqngky0s