Is it though? I read your comment first and agreed with what you wrote, but then I saw the GIF at the top of the article and instantly got what they're doing. They're solving what I call the problem of "taking a photo of what you see".

(I don't know what the professional vocabulary is for this, so please excuse the long-winded explanation.)

Here's the scenario: when I sit on the chair and look straight forward without moving, I can see a certain area of the room in a certain way. If I take out my phone, place it where my eyes are and take a photo, the resulting image looks nothing like what I just saw from that position. The biggest difference is, it captures much less of the field of view, but also the angles look wrong.

I also repeated this "experiment" virtually. The other day, I was doing some interior design in Blender, and created a room and a corridor using real dimensions. I tried, out of curiosity, to render a view from the perspective of myself (using my eye height). I explored camera configurations for a good hour, and couldn't find anything that would render the image corresponding to how I would perceive that space in real life. Again, the main problem was, either the field of view was too small, or angles were too badly distorted.

Of course a flat image of a scene, viewed from distance, is not the same as seeing the scene itself. Both my phone and Blender are correct. Perhaps they reproduce exactly what I'd see if my phone and screen were windows to the scene. But that's not what I'm after, that's not how my brain tries to interpret these images[0]! Instead, my mind tries to comprehend the image as if my eyes were the camera. It does some weird and hard to describe mapping[1], trying to put itself in the place of the camera, and immediately notices that the image doesn't look like what eyes would perceive.

When I saw that GIF showing a room rendered with hFOV=160°, the FOVO variant immediately clicked. My brain took that image, did its weird mapping, and concluded that this is exactly how the room would look like to me, if I was standing where the camera is!

So in my books, whatever they're doing, it's a big step in the right direction. These corrected renders feel like what I'd see if I were the camera. For games, this could improve immersion when playing on a flat screen. Outside of games, such transform on a photo would help people communicate their perception of the things they photograph.

--

[0] - In this case. I have no problem seeing a picture as a picture, but I can also try to see it as recording of perception. It's a choice I can make. I assume that's pretty standard and everyone can do this if the image looks close enough to first-person view.

[1] - The best description I can come up with - and mind you, I'm going off my feelings and introspection - is as follows. Assume a simplified model of sight as made of 3 layers. In layer 1, you have raw input signals from the eyes, which give you two tiny images. Layer 2 expands these images into an unified, much larger image, correcting for saccades and filling in blanks via memory and inference. Then layer 3 tries to understand the image.

What my brain seems to do, when I'm trying to comprehend a first person perspective render or a photo, is identifying that artificial picture, isolating it and spawning a subprocess running just layer 3, feeding that artificial picture as its input, as if it came from the layer 2.

Are we looking at the same picture? The first render in the article? The picture is a flat projection about 10cm wide on my screen, so it's displayed on a tiny FoV. Looking at a picture of 160deg render, stretched over 10(?) degrees of my field of view just looks like an extreme fish eye? The article suggests it's "natural" and it may well be if I'd see in it VR or at least a huge display... but maybe I'm not getting it?

And what are these "occlusion" path the article talks about? How does the edge of the building become visible above the head of the robot in any projection, unless the eye moves? Light either a) reaches the aperture from the edge of the roof, or b) it doesn't reach the aperture because it's occluded by the robot. Anything else would imply some non-euclidean light transport here like those black hole renderings...

The robot gif looks like it does something really strange wrt occlusion. But what? The caption mentions occlusion paths and nonlinearity, but neither of those terms are found in the rest of the article!

Right-click on the GIF and open it in a new tab, and zoom it up a bit if it helps. Then try to look at it as if you were playing a video game, or imagining yourself being in the room.

The linear projection image, the "one that looks like the space is being sucked into a vortex" is what you'd get in a traditional game if you set it to render the same visual field human eyes see, instead of 90° that's typical for first-person shooters. The other one, the one that looks slightly bendy around the edges but otherwise OK, is much closer to what you'd see if you were standing where the camera is.

> The picture is a flat projection about 10cm wide on my screen, so it's displayed on a tiny FoV.

That's not the perspective I'm talking about. I'm talking about the other one, in which you look at the picture on the screen, but see it as if the picture was your entire field of view. I'm not sure how to explain it better than I tried in my previous comment, so let me put it this way:

Look away from the screen, down the length of the room you're currently in. Absorb the space you see, without moving your head. Try to do a mental "screenshot". Imagine downloading it from your brain and viewing it on your computer screen. It would look completely different from a real photo taken from where you're sitting. That FOVO render seems to be close to what the "mental screenshot" would look like on a computer screen, at least in my case. And it's the first time I saw anyone doing it.

That makes more sense (although I'd probably never get used to that in a flat projection (screen) type game, no matter how hard I tried to immerse in it. The extreme distortions of things round details being very oval etc seems like they would break immersion very quickly, but might work in VR.

I still don't understand wth is going on though with the occlusion.

https://www.fovotec.com/architectural

Indeed, we're remodeling the house, and our architect is going to give us some 3D renderings, and I know it's going to look nothing like when we're actually there. Maybe should propose this to him...