sciencetech-blog

comments

Classic science fiction film The Matrix depicts a future in which humans experience the world in a simulated reality.

But if Facebook has its way, this fiction could become fact. 

Michael Abrash, chief scientist from Facebook-owned virtual reality (VR) experts Oculus said The Matrix provides the best sense of what virtual reality could someday be like.

And he used optical tricks to prove that we are merely 'inference machines' and the world we see now is already an illusion. 

Scroll down for video 

Michael Abrash, chief scientist from Facebook-owned virtual reality firm Oculus said 1999 film The Matrix provides the best sense of what virtual reality could someday be like. At the firm's F8 conference in California he used optical tricks to prove that we are 'inference machines' and the world is already an illusion

The comments were made at the social network's annual F8 conference in San Francisco.

'While science fiction novels gave me the conceptual framework for thinking about VR, it was The Matrix that made me believe in it,' Mr Abrash said.

'Even though it was based on technology that won't exist for decades, if ever, The Matrix gave me a deep sense of what VR could someday be like.

'Not only how real it could be, but also how exciting it would be to bend and stretch that reality.'

In particular, he quoted a speech made by character Morpheus. 

Summing up what's truly unique about VR, the character, played by Laurence Fishburne, says: 'What is real? How do you define real?

'If you are talking about what you can feel, what you can smell, what you can taste and see, then 'real' is simply electrical signals interpreted by your brain.'

'Unlike Morpheus, I'm not offering you a choice,' said Mr Abrash. 'No matter what choice you pick, we're all headed down the rabbit hole together.' 

'While science fiction novels gave me the conceptual framework for thinking about VR, it was The Matrix that made me believe in it,' Mr Abrash said. He quoted Morpheus (played by Laurence Fishburne centre) who said in the franchise: 'What is real? 'Real' is simply electrical signals interpreted by your brain'

'Unlike Morpheus, I'm not offering you a choice. No matter what choice you pick, we're all headed down the rabbit hole together,' Mr Abrash said. This made references to pills offered to Neo in The Matrix. He used the illusion (pictured) to show that our brain already infers what's real based on the limited data it receives

The red and blue colours in the previous image are simply what a viewer's brains perceive, based on the rest of the information around them. The pills are actually grey (pictured). And Mr Abrash said that even when a person knows the pills are grey, they still see them as red or blue

This refers to the pills offered to Neo in The Matrix. If he takes the blue pill the story ends and he is told he will wake up in his bed and 'believe whatever he wants to believe'. 

Taking the red pill means he stays in The Matrix.

Mr Abrash continued that most people focus on the 'virtual' in virtual reality, but we should be focusing on the latter.

'Morpheus made two critical points with his sentence: our conscious minds never actually interact with the real world and that we interact with sensors on our eyes, ears and tongue, and throughout our body. This is just a very small subset of the real world.'

He gave the example of vision. Humans only have three colour sensors, we can't see infrared or ultraviolet and we have a blind spot in each eye. 

'Our visual data is actually astonishingly sparse and even if we were able to accurately record and process every photon that reaches our eyes, we'd still have too little data to be able to reconstruct the world accurately,' he said.

He used the recent black and blue/white and gold dress as an example.

This illusion was repeated with a Rubik's Cube in which the blue tiles on a yellow background were highlighted alongside the yellow tiles on the blue background (pictured)

The true colours of the tiles is shown. 'Your visual system isn't interested in whether the photons coming from a tile on a random image are red or blue or grey.'Knowing that didn't keep anyone from being eaten by lions on the Savannah. What it is interested in is identifying potentially relevant features, in the real world'

'Our visual system takes its best guess and sends that to the conscious mind,' he continued. 

'The way that the brain compensates for the limited data it receives is by maintaining a model of the real world that it constantly updates as new data comes in.

'And it is that model, not the real world, that you experience and trust implicitly. We are inference machines, not objective observers.

'The fact that reality is whatever your mind infers from the nerve impulses sent to sensors, based on the model of the world, is at the heart of what makes VR different and more powerful than anything that has ever come before.' 

He then gave three examples that demonstrate how this inference models breaks down.

The first showed a red and blue pill on hands that were shown on a yellow background.  

He used this to reveal that the colours of the pills are the same shade of grey, and the red and blue colours that people see are simply what their brains perceive, based on the rest of the information around them.

And even when a person knows that the pills are grey, they still see them as red or blue.

This was repeated with a Rubik's Cube in which the blue tiles on a yellow background were highlighted alongside the yellow tiles on the blue background. 

'Your visual system isn't interested in whether the photons coming from a tile on a random image are red or blue or grey.

'Knowing that didn't keep anyone from being eaten by lions on the Savannah. 

'What it is interested in is identifying potentially relevant features, in the real world, under a variety of conditions.

'Your visual system constantly corrects for the colours in the scene. It is reverse engineering reality rather than just recording it. The colours seen are your brain's "best guess."'

'Welcome to the rabbit hole'.

A second example used a checkerboard with and without spots. The first looks flat and the second looks like it's bulging. 

Mr Abash explained this is because our visual system deploys its cells to detect high contrast edges in the horizontal and the vertical.

A second example used a checkerboard. The left image looks flat and the right looks like it's bulging. This is because our visual system is designed to detect high contrast edges in the horizontal and the vertical. When spots are added, this system also detects high contrast on 'slightly slanted axes' as well, which skews it

In a third example, two tables were shown that had the same sized top, but appeared to be different shapes. 'We live in a 3D world and 3D objects implied by the 2D shapes of the tables are quite different from each other. Your brain does this automatically for you. It's wrong but allows us to function in a 3D world'

When the spots are added the visual system detects high contrast not only in the horizontal and vertical, but also on 'slightly slanted axes' as well. As a result the perceived orientation is skewed.

In a third example, two tables were shown that had the same sized top, but appeared to be different shapes. 

'We live in a 3D world and 3D objects implied by the 2D shapes of the tables are quite different from each other,' Mr Abrash said.

'Your brain does this calculation automatically for you, making the assumption that you're looking at 3D world. That's wrong but allows us to function in a 3D world.'

Onto movement, Mr Abash said 'movement is the area in which the inferential nature of reality really shines through.'

Onto movement, Mr Abash said 'movement is the area in which the inferential nature of reality really shines through.' For this example he used an illusion (pictured) that causes what's known as a motion after-effect (MAE). It is caused when visual neurons react to moving objects or stimulus 

It also works with audio. Mr Abash played a video in which a woman said 'bar' repeatedly, but if she altered the shape of her mouth, she appeared to be saying 'far.' This is called the McGurk Effect (screen grab shown)

For this example he used an illusion that causes what's known as a motion after-effect (MAE). 

It occurs when visual neurons react to moving objects or stimulus.

Cells that respond to motion in a clockwise direction become tired and when the spinning stops, the cells sensitive to motion in the opposite direction become active. This makes stationary objects to look like they're moving in that direction.

'Motion detection takes its cues from the highest frequency, which is the highest detail within a scene,' Mr Abash said. 

'And when the dots are focused near the fovea, the high-resolution part of your retina, the sharp circular edges are the highest frequency components so the movement of the dots drives motion detection.'

Mr Abash concluded: 'I think it's fair to say our experience of the world is an illusion. One that evolution has honed to be highly functional in terms of survival and reproduction. VR is about experiencing a virtual world as real [and] experiences are nothing more or less than whatever your mind infers'

Away from the fovea there's not enough resolution to pick up those edges clearly. So motion detection is based on the movement of the lowest frequency patterns within each dot.

It also works with audio. Mr Abras hplayed a video in which a woman said 'bar' repeatedly, but if she altered the shape of her mouth, she appeared to be saying 'far.' This is called the McGurk Effect.  

'The McGurk Effect is an emphatic answer to "what is real?"' What you hear is based on what you're looking at, not just the sound hitting your earphones,' continued Mr Abash. 

'You are not a microphone, you're an inference machine. In short, reality is what our brain reconstructs it to be based on our model of the world.

He concluded: 'I think it's fair to say our experience of the world is an illusion. One that evolution has honed to be highly functional in terms of survival and reproduction.

'Since the primary objective of the perceptual system is to stay alive, it operates rapidly and automatically.

'VR is about experiencing a virtual world as real [and] experiences are nothing more or less than whatever your mind infers.

'What that means is that virtual reality, done right, truly is reality as far as the observer is concerned.' 

Quoting Cipher in Matrix, Mr Abash concluded: 'Buckle your seat belt, Dorothy, because Kansas is going bye-bye.'