The Emerging Language of XR and the Opportunity for Schools and Students
Updated: Mar 6, 2021
10 years ago I was embarking on the process of making my first IMAX film. We’d written our treatment, raised the finance, and had one last step of getting a distributor on board. We had a great topic, amazing locations, and the money to make the film, and yet every distributor was wary of us.
We later discovered the distributors were wary because they felt we didn’t understand the language of the medium and they weren’t sure if we were prepared to put in the work to learn it. We absolutely did put the work in to understand the format, and it was that work that has driven our approach to XR.
Language vs Communication: They’re Not the Same Thing
The simple act of communication is an incredibly complex yet fascinating subject. The goal of communication is to transfer information; the purpose of language is to provide a set of rules to allow you to achieve communication between two complete strangers.
Language is a ‘construct’ of communication. Without language, communication becomes very difficult, in fact charades is a great example of what happens when you remove the language construct. The wild flailing about that results is a desperate attempt to find a body movement, a symbol, anything that two people associate with the secret word.
You can see language rules in many different systems. Computers use internet protocols to set the rules of communication between to ‘stranger’ computers, you can tap out Morse code, or wave flags in semaphore. There are lots of different language types, and all with their own strengths and weaknesses.
The IMAX format has glaring differences to TV and traditional film, but also complex nuances and subtleties. Closeups were out (nobody looks good on a 2-story screen!) wide shots were definitely in, while on-camera sound was definitely out because the “Solido” - IMAX’s 3D camera - sounded like a jet engine as it spun up the film.
The mini fridge sized IMAX "Solido" camera
The main difference was IMAX's key driving principle: the experience is paramount.
In experiential formats, the experience overrides the information. You learn through the experiences you have and the emotions you feel, rather than the information you are told during the experience. When you make an IMAX film you must give the audience an experience that they can’t get anywhere else.
This is also true for AR and VR, they are experiential formats.
This of course raises the question – how do you make a good experience?
Defining the Language of AR and VR
History tells us that when a new creative format appears, the first content attempts usually try use the language of an old medium and apply it to the new one. When radio was first invented, radio adverts involved a person reading out the words from the newspaper advert. Early films were set to look like a theatre performance and right now most AR experiences feel a lot like repurposed 30 second TV adverts.
Much of this is to do with the fact that the rules are very hard to work out, if you’ve ever tried learning a second language you will know the frustration of going back to childlike communication (and still failing at that!). Even if you’ve been told the rules of the new language, you still get things wrong.
In her book, "How Emotions are Made", neuroscientist Lisa Feldman Barrett describes a phenomena called "experiential blindness"
Our brains are constantly receiving all this sensory information (data) which is informative, but also ambiguous and it has to be interpreted. So the brain uses concepts to quickly interpret the data...and sometimes it is an incorrect interpretation. This is known as “experiential blindness” – the inability to perceive what you don’t already have a concept for.
In other words, the brain is constantly trying to make sense of the data it is receiving. One of the easiest ways for it to do that is to use past experience as a guide.
Right now users have so little history with AR and VR, there is no standard set of expectations, or symbolism, so when they encounter something new it can take time for it to be assimilated into (or rejected from) the language.
When the first VR content was created, an accepted truth was the audience couldn't handle hard cuts, so every scene needed to crossfade. Now the audience is more sophisticated, so its seen as far less of a consideration.
There is a melting pot of experimentation in XR content creation right now, and the possibilities seem endless. In the VR game Half Life: Alyx get you the ability to write on windows. It was a fun little idea that I'm sure the developers wouldn't have expected to be used by teacher to teach his students math.
All the work and experimentation means the audience and the creators are starting to communicate properly - a language is slowly emerging but if you've never created for XR it can be difficult to get it right.
Take this emerging fact of VR communication:
Virtual reality can impart a “perspective” on a situation. With the correct point of view, you can help someone develop empathy towards someone else’s experience.
This is a profoundly awesome (and scary) idea that Stanford research has revealed, and they’ve shown same effect is not present in 2D.
With that in mind, attempt this thought exercise:
Imagine you want to give a viewer the experience of having a job interview. You choose to film in 360 VR and now you need to set the room up with props, people, and camera. How do you set this all up - where is the best place to put the camera, who are you filming and what is in the room? How do you impart perspective?
In 2D you might have a wide shot to establish two characters, the interviewer, and the interviewee. You might have some over the shoulder shots of each character, or maybe you shoot the interviewer with a lower camera on the table and the interviewee with a higher, to establish power dynamics. The point being in 2D you have dozens of possible camera positions and cuts.
For VR, one “correct” solution for the exercise is to place the camera in the position of the person being interviewed, and have the interviewer direct their questions at the camera. This means when you put the headset on, you are the interviewee, and the interviewer is directly asking you the questions.
This is a great exercise because it highlights how different the rules of VR filmmaking are to 2D. In 2D filmmaking, talking directly to camera usually feels very confrontational while in VR it feels natural.
AR and VR and Schools
XR is one of the first languages to have emerged in a globalised environment and it will affect every part of our lives. It will be everywhere, a huge industry with countless job opportunities. Will there be regional dialects for AR and VR usage? Will European VR be different to Chinese VR and the USA’s VR styles? Or will the content and usage of XR standardise globally?
The answers to these questions will lie in who creates the content. At Vortals, we want to empower the students to do the creating, we want them to build the new digital language and we want to evolve with them as they push these boundaries. Those that are making will change the world.
For schools, XR is an amazing opportunity for their students to get in on the ground floor, to drive the language forward and be the creators of the language who push it into new frontiers.