Image Image Image Image Image Image Image Image Image Image

Bartlett School of Architecture, UCL

Scroll to top

Top

No Comments

(Im)mobile Montage: Crafting a Virtual Reality Film and Achieving a Notion of Presence

(Im)mobile Montage: Crafting a Virtual Reality Film and Achieving a Notion of Presence
A traditional film and a virtual reality film (VRF) are two autonomous and independent forms of art, and this research will demonstrate where the similarities and differences between them are to be found. The process that sees a traditional film go from concept to screen differs widely from that involved when making a film intended to be shown on a head mounted display (HMD).  In a traditional film, for example, the director frames a series of juxtaposed, sequential images on a fitted screen; when the screen is fixed to the body, however, as it is when using a HMD, the way in which a director works in terms of framing the gaze of the viewer is considerably changed.  Most of the techniques involved in traditional filmmaking, therefore, such as the viewer’s position with regard to camera movements, are no longer applicable to VR filmmaking.  In a traditional film, for example, the camera moves in various ways: dolly and zoom shots, crane shots, pan and tracking shots; the camera in virtual reality, however, is the observer him- or herself: this physical entity, which moves and embodies a scene as it is making journeys in space (Bruno, 2002).  Furthermore, movements of cameras in a virtual reality film are, for the most part, not recommended, unless the observer has a certain level of control, since such movements may give rise to ‘cybersickness’.  This leaves the director of a virtual reality film without the predetermined camera configurations, positions, and angles that we see in traditional filmmaking.  As a final consideration, existing film production methods must be carefully evaluated, and adapted, in order to establish a notion of presence in a VRF.   
Introduction

‘Everything that can be an object of our internal perception is virtual, like the image produced in a telescope by the passage of light rays.  But we are justified in assuming the existence of the systems (which are not in any way psychical entires themselves and can never be accessible to our psychical perception) like the lenses of the telescope, which cast the image.’ Sigmund Freud, The Interpretation of Dreams

The medium of virtual reality (VR), referring to the use of a head mounted display (HMD) that enables users to experience, and interact with, a computer-generated environment, arguably began with Ivan Sutherland’s research, nearly half a century ago.  Sutherland envisioned a ‘Utopian Ultimate Display’, which would free the user from the constraints imposed by VR data glasses, since the virtual environment would extend across an entire room, and a computer would assume control over the materials displayed. The entire virtual environment would, therefore, be comprehensible to, and impact upon, all five senses (Sterling, 2018).  Aiming at the construction of a synthetic environment, with the virtual objects appearing to be actually, physically there, Sutherland’s approach of a screenless computer mediated telepresence (Leopoldseder, Stocker, and Schöpf, 2015) presents itself as the ultimate level of interactivity, which surpasses that interactivity offered by the media of a virtual reality film (VRF), and a traditional film.  In this sense, the screen, this residual element of a long line of technological inventions based on the model of vision (Lippit, 1994), becomes a limit, and a marker of difference between the user/subject, and the virtual world.  Only two decades after Sutherland’s ultimate VR display discourse, Baudrillard went further, in proposing a progression to a simulacrum, in which ‘virtual screen-based spaces’ appeared to become the primary sites for mediating between real world environments.  For Baudrillard, the question is no longer one of imitation; rather, it is one of substituting signs of the real for the real itself – the central tenet of the current phase of VR development.  In this way, VR’s haste to move away from the cinematic frame, and to create a screenless virtual environment, can be perceived as the desire for a world without alterity, as it merges into the world, and, rather than being considered a technology, becomes instead a state of being, a condition (Baudrillard, 1983).  

Screens themselves, however, are ambivalent objects – illusionist windows, and physical, material entities at the same time (Mondloch, 2010).  Henri Bergson, a pioneer in the field of the phenomenology of the moving image, argued that the object we see before us, according to the speculations of philosophy, exists only in our mind or, more specifically, exists only for the mind; the object itself is entirely different from that which is perceived.  The colors ascribed to the object by the eye, and the resistance found in it by the hand, are states of our mind (Bergson, 1991).   External images are glanced at through our eyes; we identify ourselves with them, we are drawn to them, and they transmit their movements to the image we have of our own body.  Bergson further explains that the body responds and influences the external images, by giving movement back to them.  By means of this, a type of screen-mediated viewing of art existed well before the invention of still or moving photographic media, and the HMD.  This viewing of art goes back, in fact, as far as Leon Battista Alberti’s fifteenth-century formulation of the canvas/screen as a window that opens onto a space “beyond the frame”; it also harks back to the baroque theatre as a spectacle, the ‘speculum princeps’; it even traces as far back as the Paleolithic painter’s drawings from 30,000 years ago. In France’s Chavet Cave drawings were found of a bison with eight legs, suggesting movement: almost an early form of proto-cinema.  

In this sense, camera obscura images, shadow shows, magic lantern projections, panoramas, dioramas, and a variety of contemporary attractions, like the sensorama, the Cave Automatic Virtual Environment (CAVE), and the HMD, which position the observer in front of screens, are merely the latest chapter in a long-standing practice of art production and reception (Lippit, 1994).  Certainly then, in terms of the presence of images displayed via the screen, and the fact that our senses are open to them, these images represent the world of ideas, the imaginaire; in making them visible, the screen distinguishes itself from the materiality of light, and from the universe that surrounds it (Bergson, 1991). As a consequence, what matters in the context of this research is not what we see, as a spectacle, but what we think of all the motives and consequences provided by the screen, and the significance of these (Lippit, 1994).  In relation to this, the recently constructed ‘transportation theory’ attempts to describe the age-old phenomenon whereby individuals, imagining scenarios in which they could venture off into these images, become absorbed in a story, or transported into the narrative world to such a degree, that the effects of the story may begin to show on their real-world beliefs (Green, and Brock, 2000). This is an important phenomenon to bear in mind, when considering the applicability of techniques to achieve ‘transportation’ in film and VRF; this ‘transportation’ is commonly defined as ‘suspension of disbelief’ in terms of film, and as ‘presence’ in VR. ‘Suspension of disbelief’ has long been used as a term to denote viewer engagement in film; ‘presence’ is the more recent term developed to assess the level of transportation within VR (Mateer, 2017)  In both media, transportation is the primary responsibility of the director; it is important to bear in mind, though,  that the expectations of a viewer seeing a film at the cinema  are very different to his or her expectations (and reactions) when in a VRF.  

It is important at this juncture to identify two distinct types of subjects, one type for each type of media being discussed here: a ‘spectator’ in film, and a ‘user’ in VRF.  The distinction between ‘user’ and ‘spectator’ lies in their two fundamentally different roles, and accompanying set of actions. The ability of the user to interact with a virtual environment is regarded as the central feature of VR; its possibility for interaction distinguishes VR from cinema (from the Ancient Greek ‘κίνημα’ ‘kínēma’, meaning ‘movement’),  where there is reduced participation and involvement for the ‘spectator’ in what he or she sees. Thus, a characteristic of VR is that the role of the subject (i.e the user) is ‘active’ as opposed to the ‘passive’ role of the subject (i.e. the spectator) in cinema (Aylett, and Louchart, 2003).  The ‘user’, therefore, is defined as a person that actively participates in the building-up of the resulting virtual reality experience, maybe by altering a predicted outcome, or by triggering a certain sequence, based on his or her movements; a ‘spectator’, however, contemplates or watches a narrative display. It is worth emphasizing, as does Kate Mondloch in ‘Screen: Viewing Media Installation Art’, that film theorists make a distinction between the “subject” (the position assigned to the observer by the film and various cinematic codes) and the “viewer” (the actual person who watches the film and his or her complex viewing responses) (Mondloch, 2010).  Therefore, in this paper we will subsequently employ the term ‘spectator’ as the passive viewer that does not alter the progressive outcome of the plot.  

An exploration of the particular characteristics of each subject, regarding film and VRF, and a comparative analysis as to how a director must gather information for a user or spectator, harks back to the early days of cinema and film theory, and the montage sequences of Sergei Eisenstein. Eisenstein’s discussions of the immobile and mobile spectator in relation to film are well-known, and are most notably articulated in the renowned essay ‘Montage and Architecture’. Here, Eisenstein takes the Acropolis of Athens as a perfect example of one of the most ancient of ‘films’: its montage sequence for an architectural assemblage is subtly composed, and can be apprehended by the individual, shot by shot, as he or she walks amongst the buildings of the Acropolis.  However, if the spectator cannot move, the director must forge an imaginary path, unsuitable to a single gaze, where elements that are dispersed in reality converge on one unique point. (Eisenstein, 1989) 

(Figure 1) Montage and Architecture. Sergei Eisenstein. Mobile and Immobile Montage.

This being said, a formal definition of a VRF is still being developed, given how relatively new the phenomenon is. Simultaneously, various other related terms continue to emerge (along with differing definitions and nuances), such as ‘cinematic virtual realities’ (CVR) (Mateer 2017). In this paper, we will take the opportunity to define what constitutes a VRF, and where this stands in relation to VR experiences and traditional film. First of all, a VRF uses a combination of real-time CG and pre-rendered sound elements, pictures and animations, that is, footage that is not rendered in real-time by the hardware in use; users do have a certain level of influence over the progressive outcome within the VR environments.  In a VRF, triggers, buttons and virtual objects are scattered throughout every scene, allowing the movements of the user to set in motion a film that potentially plays out in a non-linear manner. Unlike VRF, CVR does restrict the level of control users have to an ability to choose an angle within the environment from which to view the scene; the inability of users to actually interact with elements contained within the virtual world is the primary difference between VRF and CVR.  While both are immersive, CVR experiences are effectively linear presentations with the duration of each experience dictated by the length of the media assets employed (Mateer, 2017).  

 

(Figure 2) ‘Graphic analysis of a subjects’ role in virtual environments and cinematic experiences.’

Even though CVR represents a new type of filmmaking, our research has elected not to engage with this method, since its emotional involvement is not an automatic product of filtering a traditionally blocked scene from 2D through a 360 degree camera, nor of creating an animation from another platform, and inserting this into a VR environment (Mateer, 2017).  Conceptually, it is appealing as a theoretical approach; The Daily 360 coverage of The New York Times was an attempt to take this approach, amid much fanfare and hype about a new medium; this a project foundered, and the program is no longer running.  Virtual Reality Films themselves relate back to the formative days of cinema, where explorations of the ‘kino-eye’ led directors to explore fixing the camera in innovative positions, and to determine montage rhythms, tempos, and scene transitions. It is worth stating at this juncture that the essence of this research, beyond establishing the best product in terms of a VRF, is to explore and decipher the essential steps needed to create a truly immersive and genuine virtual reality film experience, where every user leaves the film having an intimate sense of presence.  

(Figure 3) ‘A brief history of Virtual Reality: from the Baroque theatre to the Augmented Reality Glasses.’ [1. The Baroque Stage, (1618); 2. The Panorama (1787); 3. The Stereoscope (1838); 4. Lenticular Print- ing (1898); 5. The View-master (1939); 6. Sensorama (1957); 7. During the 60’s, 70’s, 80’s, VR took a leap and a series of inventions developed: ‘Telesphere mask’ (1960), ‘Headsight’ (1961), ‘Teleyeglasses’ (1963); ‘TV-Helm’ (1967); ‘Sword of domiciles’ (1968); ‘World on a Wire’ (1973); Data- gloves (1976); 8. Smartphones and Augmented Reality (2008); 9. Google Carboard, alongside the Google Glasses, and Oculus Rift (2012-2016); 10. HoloLens(2018).} (Leopoldseder, Stocker, and Schöpf, (2015)).

Framing the Sphere

‘When I say there is no Form in ‘Rhythm 21’ (1921), I mean that by taking the whole movie screen, pressing it together and opening it up, top, bottom, sides, right, left,  until you don’t perceive form anymore, you perceive movement.’ (Hans Richter by Hans Richter, Hans Richter)

From experimental film and video to virtual reality

From this point on, our research will concern itself with virtual reality film (VRF), and its interactivity and other particular characteristics, as a form of cinema that has been evolving ever since the development of film: a material construct and a time-based medium.  Throughout its development, VRF’s emergence as a medium has enabled users to jump out of a non-lucid, dreamlike state (in which a spectator exists before the frame of a conventional movie), and to make conscious, lucid, analyzed decisions within the virtual world.

In a retrospective analysis of how VRF has evolved as a medium, alongside other narrative art forms, one essential concept is the vision of cinema identified by Peter Greenaway. Greenaway envisioned cinema as the total sum of all technologies, which work towards articulating the moving image, as a continuum of elements that are not perceptibly different from each other (Rees, 2011). Cinema embraced the big-screen movie as well as the computer screen, along with the digital image and the handmade film; it was not only material art, but was also the most phantasmagoric of art forms in its final product and effect. Film’s impression of movement is an illusion in that the image does not move; it consists of a series of static frames on celluloid. Video, including computer-generated environments such as VRF and games, follows the same principle, using electronically coded signals.  For the most part, the source of the image is, in both cases, strictly invisible to the observer.  A.L. Rees, in ‘A History of Experimental Film and Video’, develops this line of thought further by pointing out that in video and digital media the image-in-motion is coded as a scanned electronic signal (Figure 4).  Film, video, and electronic media – including virtual reality, with its parallax effect under stereoscopic lenses – are cinematic equations which diverge in their own directions even though they would seem to converge into the one discourse of the moving image. They all advance in different ways to achieve their grand illusions, and each experiences ruptures from the reality which it attempts to denote (Rees, 2011). From these polarised tendencies, each medium obtains its own material expressions and codes.

Figure 4. ‘Representation of digital media as a series of pixelated codes.’ The illusory images-in-motion (A,B,C) are pixelated signals that change-displace in each frame.

For most of the early history of cinema, films which stood apart from the commercial sector (and even the ‘art house’ sector), such as the avant-garde, with its rapid camera movements and long takes, film grain, and hand painting, attacked the normative vision and the comfort of the spectator (Rees, 2011).  Independent artists and directors of avant-garde films at the turn of the twentieth century were in a similar position to the artists and directors of VRF a century later, in their exploration of the innovative rhythmic patterns of a scene, though the latter of course involved a head mounted display (HMD).  There is an almost direct link between the explorations of the early avant-garde films, which date back to the celebrated year of 1895 when the Lumière Brothers showed their machines in Paris and London, and the recent pioneers of virtual reality filmmaking – such as Penrose Studio (whose output includes ‘The Rose and I’), Baobab Studios (‘Invasion!’, ‘Asteroids’) and Oculus Story Studio (‘Dear Angelica’, ‘Lost’, ‘Henry’). Both of these genres are directly connected by their approach to film and video, which emphasizes vision over text and dialogue.  

As A. L. Rees suggests, these early days of experimental film and video alike came at periods where concepts from modern and post-modern art were intertwined with those of cinema.  There are a number of examples, as to how contemporary discourses of the arts would shape and influence cinema. For instance, the concept of the ‘assault on the eye’, which has been traced back to the study of optics, recommended to painters by Cézanne at the dawn of modernism(Rees, 2011). Picasso’s elegy to Norman McLaren’s drawing over film strips provides another example, as does cubism’s influence of framing: instead of a single viewpoint suspended in time, (which the photograph had by then perfected), this prompted exploring changing angles and viewpoints on its subjects (Rees, 2011). The ‘visual certainty’ of cinema was, in this way, already being questioned, and this produced two kinds of film-making that would later influence other movements in experimental cinema, up to and including virtual reality filmmaking. There were, for example, the short, oblique films in the tradition following Man Ray, and also the abstract German films, which broadly set up different spaces for viewing from narrative drama, and interrupted the stable perceptions of the audience, aiming at non-identification of subject and image (Reese, 2011).  It was in this social environment that the German abstract filmmakers Viking Eggeling and Hans Richter began to study the art of movement, which eventually led them to film.  At first, their abstract films were simply the best means by which to articulate the unfolding rhythmic patterns which Richter and Eggeling drew out on long scrolls of paper.  For Richter, the orchestration of time was the aesthetic basis of this new art form.  In ‘Rhythm 21’, Richter articulated movements and rhythms, rather than form, to construct his elemental concept of articulated time (Richter, 1971). These artistic endeavours, then, served as a foundation to build upon, opening, as it were, a door to perception and telepresence, which could be delved into by means of the stereoscopic lens.  These early avant-garde practices and their explorations of the movie screen, with its ever-growing discoveries of optical illusions with regard to the representational elements, led us to a reproduction of the art of Virtual Reality Films. 

Form & Rhythm

Throughout the entire history of experimental film and video there is one factor that determines the level at which the spectator engages with the film: the rhythm. Rhythm represents a problem of significant importance in terms of the conception of the moving image, due to its expressive structure (Wahlberg, 2008). In this section, rhythm, which is a product of a combination of framing, editing and ‘settling in and setting the scene’ (a term coined Oculus Story Studio, which was in operation from 2014-2017), will be shown to be an element in films, and VRF, that both appeals to the viewer, and, importantly, provides “an unfixed location of meaning”(Unseld, 2017). 

A key concept here is the manipulated space-time of cinema; this has been expressed in a number of ways: the ‘monistic ensemble’ of Sergei Eisenstein, for example, or the ‘intervals’ between scenes of Dziga Vertov, or the science and avant-garde films of J. C. Mol. All of these reflect the filmmaker’s incessant experimentation with time measurement and the visualization of rhythm.  This was perhaps most clearly evinced during the 1930s, with Mol’s technique of filming the growing and wilting of plants. To film this, a clock was set up, which controlled the coordination of camera flashes, and one frame was exposed every quarter-hour. Subsequently, during a day cycle, only four to six seconds of film were shot (Wahlberg, 2008). Our apprehension of time within the camera eye is subject to the fragmented structure of the frame, which informs the experience of the spectator.  In one of Mol’s conceptual approaches, during an interlude in ‘De Stijd en de Film’ ( ‘Time and Film’, The Netherlands, 1928), he offers a lecture in rhythm and tempo as pivotal elements in the realization of cinematic time:

‘A film consists of a series of thousands of little pictures.  If when projected the same number of pictures is shown on the screen each second, as the number of pictures which was taken in a second, we see the natural movement on the screen.  If we take the pictures more quickly, however, the movements on the screen are retarded; it looks as if time is going more slowly.’

(Figure 5) ‘A virtual reality film interpretation of Hans Richter’s Rhythm 21.’ By studying Richter’s compositional montage and drawing over film strips, a 3D spatial construct of the avant-garde film is presented.

Mol goes on to say that “in the same way, time can also be accelerated,” which illustrates his concept of the malleability of space-time, in terms of moving images – in this particular case, the study of plants and blossoming flowers. Where film and virtual reality, and subsequently VRF, principally differ is in the director’s handling of continuity (Mateer, 2017).  In the case of film, continuity is handled in manifold forms: continuity of viewpoint, continuity of motion, continuity of setting, continuity of sound. Because of the differences discussed above, this idea of ‘shots in sequence’,such as in Mol’s experiments, is not directly transferable to VRF.  What can be transferred from traditional film techniques to VRF, however, is the time-length of a scene, and at what pace to show the scene, without loss of presence in a virtual environment. 

Regardless of the narrative structure each medium adopts, all cinematic forms are subject to tempo.  Could there be a two hour long VRF? Could we implement the rhythm of Stan Brakhage’s film ‘Stellar’ within a virtual environment, or follow Henri Chommete’s cine-poem, ‘Jeux des Réflets et de la Vitsesse’ (‘Games of Reflection and Speed’), when using an HMD without succumbing to nausea?  Such questions form the basis of the case studies, which are presented at the end of this chapter (‘The Protagonist Walks’), and which undertook to assess the viability of applying filmmaking techniques from the feature films of selected directors, such as Andrei Tarkovsky’s ‘Sacrifice’ (1986), Alexander Sokurov’s ‘The Russian Ark’ (2002), and Ingmar Bergman’s ‘Persona’ (1966), to virtual reality filmmaking.  Since this research is to embark upon a graphic analysis of the techniques mentioned above, it would be useful at this juncture to establish the best practices and guidelines to use as a reference throughout this exercise.  First and foremost, it should be noted that the most prominently positioned figures, in terms of obtaining an insight into designing for the infinite canvas of virtual reality, are practitioners from mainstream commercial companies such as Facebook, Google, and Unity.  This research will draw guidelines from these, in combination with guidelines drawn from the creators of the cinematic virtual reality experiences ‘Lost’ and ‘Dear Angelica’. For the experienced VR user, both films stand as two of the best examples seen so far in this relatively new medium; the creators were the team from Oculus Story Studio, a division of Oculus VR, which was dissolved in May 2017. One member of that team, Saschka Unseld, encapsulated the five lessons learned whilst making ‘Lost’ in a blog article, and via Oculus’s facebook presence:

  • Don’t rush the pacing

‘[…] rather than thinking about the story as a series of “actions” a character takes, we ended up thinking about the story as a series of “moments”. As an example: 1. We are alone in a dark forest 2. A massive robot hand searches the area 3. The hand sits down alone disappointed 4. The hand waits for something that approaches.’

  •   Respect the Ritual of Settling In and Setting the Scene

‘[…] The first part of this was figuring out what would the very first thing the audience would see. This “first thing” is something called “The In”.  For ‘Lost’, we wanted to find something that takes the audience by the hand and step by step, lead (sic.) them into the world rather than of just dropping them directly and risking to overwhelm them.’

  • Let Go of Forcing the Viewer To Look Somewhere

‘[…] we have to let go of our almighty control control of what the audience sees. Instead of instantly pushing the story onto the viewer, take a step back for a while and let the viewer take part in discovering the story themselves.  By not forcing the viewer to look somewhere and making the surroundings interesting in all directions, the story incites the viewer’s curiosity in the world. And through this curiosity, have them take a more active role in experiencing the story.’

  • Be Aware of Spatial Story Density

‘In VR there should never be just one interesting story related thing to look at.  Stories and storytelling should be as three dimensional as the space around us. At any given moment we need to make sure that there is a certain amount of density of story elements that fills that space.’

  • Simplify Scope

‘Telling stories in real-time CG is hard, Not just creatively but also technically […] rendering two high resolution frames (one per eye) at 90 frames per second.  A complex environment like a forest might be achievable in animated films, but in VR it was a challenge […] we had to restrict massively in order to make the experience smooth and still look good.’

When comparing Unseld’s guidelines for the creation of an animated film aimed at virtual reality storytelling with the research articles of Mckenzie (1994), Akira (1998), Slater & Wilbur (1997), Bouchard, St-Jaques, Aylett & Louchart (2003), Robillard & Renaud (2008) and Mateer (2017), a surprising number of contradictions became apparent.  All of these latter articles focus on the diegetic cues within the virtual environment, the importance of guiding the viewpoint (a stark contrast to Unseld’s ‘Let Go of Forcing the Viewer to Look Somewhere’), and the technological performance and the psychological response of the users. None of them mention Unseld’s “letting go” as a way of taking the cinematic experience in VR environments, and moving it one step forward.  

The Oculus Story Studio article is a salient reminder that directors and artists alike continue to experiment with the new medium of displaying the moving image on an HMD, in the same way that early directors and artists experimented with the first cinematographic cameras.  As a brief example of the latter, from the silent film era, the cinematographic montage of Aleksandr Dovzhenko’s ‘Earth’ has the camera (given its technical limits) remain largely still, with the only things in motion being the characters; the sole occasion in the film on which the camera moves is the sequence where it is positioned on top of a tractor, adopting a downwards angle onto the plowed fields of Ukraine. This scene resembles ‘Pearl’, by Google Spotlight Stories, one of the very few CVRs where the set moves but manages to successfully avoid “simulator sickness” (Google); this is because there is no mismatch between physical and visual motion cues, and the spectator is not physically moving when he or she is projected to be inside a moving car.

Sense of Presence 

In an attempt to advance the cinematic experience of VRF, this section will undertake an in-depth exploration of the role of ‘presence’. In VR, ‘presence’ is defined as the psychological state whereby virtual experiences feel authentic, and is a useful gauge for assessing the level of ‘transportation’ within VR.  Since virtual reality filmmaking is a relatively new art form, methodologies for this particular medium from within the film industry are scarce. To that effect, throughout this research process, articles from other academic fields have been explored, in order to sum up the various components of Unseld’s concept of ‘letting go’. The fields drawn from, along with their accompanying articles, include computer science (‘A Sense of Self: The Role of Presence in Virtual Environments’ and ‘A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments’); technology (‘Real and Illusory Interactions Enhance Presence in Virtual Environments’ and ‘Being There: The Subjective Experience of Presence’); psycho-education (‘Anxiety Increases the Feeling of Presence in Virtual Reality’); neuroscience (‘New Frontiers in the Rubber Hand Experiment: When a Robotic Hand Becomes One’s Own’); and media (‘Directing for Cinematic Virtual Reality: How the Traditional Film Director’s Craft Applies to Immersive Environments and Notions of Presence’).

Within VR there were initially three dimensions of presence: social, personal, and environmental, (Heeter, 1992); a fourth dimension, that of spatial presence (Regenbrecht and Schubert, 2002) was later added, and this was developed by analogy with the ‘embodied models’ of other researchers (Witmer and Singer, 1998). Presence, it is important to note, is not the same as immersion (Salter and Wilbur, 1997). Immersion, for instance, is the objective feature of the hardware and software utilised, and has to do with how the computer’s display delivers an ‘inclusive’, ‘extensive’, ‘surrounding’, and ‘vivid’ illusion of reality to the participant (Salter and Wilbur, 1997); its main features require matching feedback from bodily movements.  Presence, on the other hand, is both subjective and objective, and relates to the sense of ‘being there’; it involves the participant behaving in the same way in the virtual environment as they would in analogous circumstances in everyday reality (Salter and Wilbur, 1997).  Not all real or illusory involvement, including the use of virtual bodies (VB) in virtual environments, causes a sense of presence.  Further to this, Rogenbrecht and Schuber emphasise that the notion of presence develops from the mental representation of possible bodily interaction, rather than from the objective possibility of triggering scenes, or talking to characters. The essence of this is that, no matter what ‘plot’ line is involved, the more that this represents an alternate, self-contained world, the greater the chance of achieving presence. 

In summation, the four states of presence, as summarised here in part by Heeter, in his article ‘Being There: The Subjective Experience of Presence’ and by Regenbrecht and Schubert in ‘Real and Illusory Interactions Enhance Presence in Virtual Environments’ are as follows:

  •   Personal Presence

‘[…] a measure of the extent to which, and the reasons why, you feel like you are in a virtual world. […] for example, although the rules of this world are different than the laws of physics in the real world, there seems to be a consistent pattern that I can learn to recognize.’

  • Social Presence

‘[…] refers to the extent to which other beings (living or synthetic) also exist in the world and appear to react to you. […]  Someone or something else that seems to believe that you are there may help convince that you are (in fact) there.’

  • Environmental Presence 

‘[…] the extent to which the environment itself appears to know that you are there and reacts to you. Perhaps lights turn on when you enter a room, or portals to other worlds flash into existence when you draw near.’

  • Spatial Presence

‘[…] the mental representation of movement of one’s own body (or body parts) as a possible action in the virtual environment, or from the meshing of bodily actions with objects or agents in the virtual environment.  The more possibilities there are of interacting, the more cognitive meetings are possible […]’

Even though data confirms that a relationship between the user and his or her avatar (or virtual body, VB) influences behavior within the virtual spaces, the user may or may not be able to establish a meaningful connection to the virtual world.  In some cases, participants gain a sense of agency and ownership, as in the ‘rubber hand experiment’, whereby participants actively engage with higher realism and authenticity.  In other cases, whereby, for example, the bodily avatar produces negative psychological effects, such as in a mismatch between the physical and virtual avatar, the results of spatial presence are not successful.  What concerns VRF, here, are the components of the personal, social, and environmental presence; it is not concerned with the bodily representation in a virtual world. The key consideration of VRF is to engage the user with illusory interactions, within a self-contained world, and a possibly non-linear outcome of a plot, where this is triggered by the user.  This is in contrast to the case of cinematic virtual realities (CVR),  where the spectator, according to John Mateer, is predominantly concerned with the consideration of ‘personal presence’, since it is based on simulating pre-rendered real-world simulations, sounds and images.

Taking into consideration these notions of presence in a VRF, in combination with Unseld’s five ‘lessons learned’ in the making of ‘Lost’, it is now possible to dismiss the fact that it is not necessary, as Mateer suggests, ‘to predict and control the user’s viewpoint within the virtual scene’. It may even  be possible to dismiss ‘virtual salience’ (the perceptual properties that make visual objects stand out from their surroundings), which has previously been used to direct attention (Nielsen, Moller, Hartmeyer, Ljung, Nilsson, Nordahl & Serafin, 2016). Therefore, with reference to the words of Unseld, by not forcing the viewer to look at specific objects, or in specific directions, we incite curiosity in the world, and once that curiosity has been picked, then the storytelling can be followed.  In the following section, ‘The Protagonist Walks’, the research embarks upon a graphic analysis of the techniques mentioned above.

The Protagonist Walks

The single most important force in Tarkovsky’s construction of space is the motion of the camera, as seen throughout the opening and closing shots of ‘Ivan’s Childhood’ and ‘Sacrifice’ (Bird, 2008).  The coherent internal logic of the camera’s movement, with reference to traditional film, with its framing of the gaze in time and space, must be suspended when in virtual reality film.  Drawing on the guidelines outlined above, the filmmaking techniques employed in the opening scene of ‘Sacrifice’ are graphically analysed, and compared to those of VFR (Figure 6), and what can be concluded is that to achieve a notion of presence in a VRF version of ‘Sacrifice’, a standing user must be positioned much closer to the action of the characters and the pace should run faster than originally shown.  Secondly, if an ‘environmental presence’ is one of the objectives, the users must let themselves be drawn by the attractions of the terrain and the encounters they find there -a concept taken from Guy Debord’s Dérive (Andreotti, Costa 1997).  Furthermore, the rhythm-tempo of a Tarkovsky movie disengages the VRF viewer-user unless interactive triggers and element are positioned within the scene.  

In the case of The Russian Ark (Figure 7), as the user cannot follow the main character through The Hermitage, as does the narrator of the film, a sitting observer-user must be considered for this exercise as a standing user cannot follow the distance covered by the camera in the film due to technical limits of sensors and position tracking mechanisms.  Two options arise, a guided motion provided by the direction of the head (A) and, on the other end, the user B is given a device a for teleporting from one spot of the environment to the other (B).  After implanting the option A in previous projects, it happens to be that the user gets a feeling of ‘cybersickness’ quickly.  Subsequently, option B appear to relate to gaming mechanisms that engages the user-player through an environment and the user does construct his own story line.

Finally in the case of ‘Persona’, this seems to be a study case that appeals the researcher of this paper to a great extent: it conjures most of the elemental conditions needed to succesfully recreate a VFR, anxiety, settling in, unpredictable outcomes, simplicity. For instance, the dreamscapes created by Ingmar Bergman in ‘Persona’ is an example that can be interpreted to a further extent in continued research.

 

Virtual Reality Film: A Window of the Soul

‘How does a project mature? 

It is obviously a most mysterious, 

imperceptible  process. 

It carries on independently of ourselves,

 in the subconscious,

crystallizing on the walls of the soul, 

It is the form of the soul 

that makes it unique, 

indeed only the soul decides 

the hidden ‘gestation period’ of the image 

which cannot be perceived

by the conscious gaze.’

  Tarkovsky,  Instant Light

 

Virtual Reality Film as a dream metaphor

Film has been a persistent metaphor for dreams, ever since the birth of cinema. Cinema, according to Laura Rascolli, embodies an unusual dichotomy: on the one hand, it is a ‘perfect’ system for the reproduction of reality; on the other hand, it is a medium of magic and dreams (Rascarolli).  Filmic images were, in one sense, a material-based medium, resulting from the capture of light on celluloid film strips, later shown on the screen; in another sense, these filmic images also offered themselves to the spectator as a medium detached from physical support, and independent of the real.  Rascolli goes on to state that the filmic image relates to the oneiric images of dreamers, ‘as it replaces the real’. And yet, spectators, unlike dreamers, have the opportunity to withhold participation or involvement with what they see. 

The comparison between spectator and dreamer is part of the ongoing analogy of film and dreams.  The Surrealists, for example, considered the spectator to be someone that stands between a daydream and dream; spectators are aware that the film is not a dream simply because they are conscious of not dreaming.  Dreamers, on the other hand, according to Theo Botz-Bornstein’s reflections on films and dreams, which follow the Freudian mould, are spectators of their own dreams, since reality and non-reality appear indistinguishable. Botz-Bornsteinstates that ‘the main characteristic of the dream, and therefore dreamers, is thus not to be “strange” but to be a reality distant through its eminent lack of self-awareness’. Countless examples of directors creating dreamscapes serve as evidence as to how the filmic image ‘introduces unconscious optics, as does psychoanalysis to unconscious impulses.’ (Rascarolli) Naturally, therefore, the analogy of films and dreams informs the holistic nature of virtual realities. Jaron Lanier states, for example, in ‘Mondo 2000’, that virtual realities are a structure of the unconscious.

The Screen is a Portal

As a closure into crafting a virtual reality film and achieving a notion of presence in a ocular centric world, the work of Juhani Pallasma comes to mind:

In our time, light has turned into a mere quantitative matter and the window has lost its significance as a mediator between two worlds, between enclosed and open, interiority and exteriority, private and public, shadow and light.

The screen, the frame, are windows that allow the viewer, the spectator, the user to enter a remembered or imagined place; they both form part of the cinematic experience and are part of rituals of transportation into the human condition.  Virtual reality films provide moving images that permit its users to actually feel they have erased the boundary between the two worlds divided by the window, and achieve sense of ‘being there’.  Even though Sutherlands vision was to erase the screen, and to an effect it is happening at the turn of the twenty first century with the use of augmented reality glasses, the polarised angle of virtual reality filmmaking has the possibility to reflect on the importance of content in virtual realities environment leaving every user with an intimate sense of presence.

Bibliography 

Andreotti, L. and Costa, X. (1997). Theory of the dérive and other situationist writings on the city. 1st ed. Barcelona: ACTAR, pp.10-17.

Aylett, R. and Louchart, S. (2003). Towards a narrative theory of virtual reality. Virtual Reality, 7(1), pp.2-9.

Baudrillard, J. (1983). Simulations. 1st ed. Cambridge, Massachussets,: Semiotext[e], pp.1-5; 81-85.

Barker, J. (2009). The tactile eye : touch and the cinematic experience. 1st ed. Berkeley : University of California Press. 

Bergson, H. (1988). Matter and Memory. 1st ed. New York: Zone Books, pp.1-30.

Bird, R. (2008). Andrei Tarkovsky, Elements of Cinema. 1st ed. London, United Kingdom: Reaction Books Ltd. 

Booker, C. (2004). The seven basic plots of literature. 1st ed. New York: Continuum, pp.87-106.

Botz-Bornstein, T. (2018). Films and Dreams, Tarkovsky, Bergman, Sokurov, Kubrick, and Wong Kar-Wai. 1st ed. Plymouth: Lexington Books, pp.31-54.

Bouchard, S., St-Jaques, J., Robillard, G. and Renaud, P. (2008). Anxiety Increases the Feeling of Presence in Virtual Reality. Presence, 17(4), pp.377-382.

Bruno, Giuliana. (2002) Atlas of Emotions. 1st ed. London: Verso.Ltd.

Caspar, E., De Beir, A., Magalhaes De Saldanha Da Gama, P., Yernaux, F., Cleeremans, A. and Vanderborght, B. (2014). New frontiers in the rubber hand experiment: when a robotic hand becomes one’s own. Behavior Research Methods, 47(3), pp.744-755.

Chiaramonte, G. and Tarkovsky, A. (2004). Instant Light, Tarkovsky Polaroids. 1st ed. London: Thames and Hudson, pp.6-61.

Eisenstein, Sergei M. (1989) Montage and Architecture. Journal Assemblage no. 10, 1989 with an introduction by Yve-Alain Bois, ‘The text was originally to have been inserted in a book-length work’. MA: MIT Press.

Freud, S. (1976). The Interpretation of Dreams. 1st ed. Middlesex, United Kingdom: Penguin Books Ltd. 

Green, M. and Brock, T. (2000). The role of transportation in the persuasiveness of public narratives. Journal of Personality and Social Psychology, 79(5), pp.701-721.

Katz, S. (1991). Film Directing Shot by shot, Visualizing From Concept to Screen. 1st ed. California,: Michael Wiese, pp. 3-159.

Heeter, C. (1992). Being There: The Subjective Experience of Presence. Presence: Teleoperators and Virtual Environments, 1(2), pp.262-271.

Leopoldseder, H., Stocker, G. and Schöpf, C. (2015). Ars Electronica 2015. Festival für Kunst, Technologie und Gesellschaft. Post city. Habitat for the 21st century. Ostfildern: Hatje/Cantz. 

Mateer, J. (2017). Directing for Cinematic Virtual Reality: how the traditional film director’s craft applies to immersive environments and notions of presence. Journal of Media Practice, 18(1), pp.14-25.

McKenzie, J. (1994). Virtual Reality: Performance, Immersion, and the Thaw. TDR (1988-), 38(4), p.83.

McCreery, M., Schrader, P., Krach, S. and Boone, R. (2013). A sense of self: The role of presence in virtual environments. Computers in Human Behavior, 29(4), pp.1635-1640.

Mizuta Lippit, A. (2018). Virtual Annihilation: Optics, VR, and the Discourse of Subjectivity. Criticism, 34(4), pp.595-610.

Mondloch, K. (2010). Screens: Viewing Media Installation. 1st ed. Minneapolis: University of Minnesota Press, pp.1-39; 77-92.

Pallasmaa, J. (2012). The Eyes Of the Skin, Architecture and the Senses. 3rd ed. Chichester, West Sussex, United Kingdom: John Wiley & Sons Ltd. 

Rees, A. L. (2011). A History of Experimental Film and Video. 2nd ed. London: Palgrave Mcmillan, pp.1-52.

Rascolli, L. (2015). Oneiric Metaphor In Film Theory.  Kinema, a journal for film and audiovisual media, [online] ‘Fall 2015’. Available at: http://www.kinema.uwaterloo.ca/about.php [Accessed 14 Aug. 2018].

Regenbrecht, H. and Schubert, T. (2002). Real and Illusory Interactions Enhance Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments, 11(4), pp.425-434.

Richter, H. (1971). Hans Richter by Hans Richter. 1st ed. Canada: Holt, Reinhart and Winston Inc, pp.110-128.

Slater, M. and Wilbur, S. (1997). A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments.  Presence: Teleoperators and Virtual Environments, 6(6), pp.603-616.

Sterling, B. (2018). Augmented Reality: “The Ultimate Display” by Ivan Sutherland, 1965. [online] WIRED. Available at: https://www.wired.com/2009/09/augmented-reality-the-ultimate-display-by-ivan-sutherland-1965/#start-of-content [Accessed 24 Sep. 2018].

T. Nielsen, Lasse & Brandt Møller, Matias & D. Hartmeyer, Sune & C. M. Ljung, Troels & Nilsson, Niels & Nordahl, Rolf & Serafin, Stefania. (2016). Missing the point: an exploration of how to guide users’ attention during cinematic virtual reality. ‘Paper presented at VRST 2016, at the 22nd ACM conference on Virtual Reality Software, Munich, 2-4 November’. pp. 229-232. 

Tarkovsky, A. (1986). Sculpting In Time. 1st ed. Austin: University of Texas Press, pp.82-103.

Unseld, S. (2018). 5 Lessons Learned While Making Lost. [online] Oculus.com. Available at: https://www.oculus.com/story-studio/blog/5-lessons-learned-while-making-lost/ [Accessed 16 Aug. 2018]. 

Wahlberg, M. (2008). Documentary Time, Film and Phenomenology. 1st ed. Minnesota: University of Minnesota Press, pp.63-78.

Designguidelines.withgoogle.com. (2018). A new dimension – Designing for Google Cardboard. [online] Available at: https://designguidelines.withgoogle.com/cardboard/designing-for-google-cardboard/a-new-dimension.html# [Accessed 24 Sep. 2018].

 

Submit a Comment