Category: Interaction S20

How does response act as a medium?

Myron Krueger’s text is key in illustrating how response can act as a medium, particularly by focusing on what he calls responsive environments, which “perceive human behavior and respond with intelligent auditory and visual feedback” (423). By detailing the motivations, technicalities, and deliberate decisions behind installations/responsive environments such as METAPLAY, PSYCHIC SPACE, and other of his famous pieces, Krueger points out how response is the medium. In an environment where the interaction between humans and the environment is the most important component, visual and auditory aesthetics are of secondary importance. Instead, crafting an experience that successfully responds to users actions (or lack of it) and making the response evident to them is key, and is the main factor that establishes response as a medium.

Reading this in 2020, and being fully aware that Krueger’s text was published in the 70s makes me wonder how much this notion has changed, particularly when areas like interactive media arts, integrated digital media, and creative technology are more consolidated than before. Now, there is no question that response is the medium, but the aesthetics and quality of that same response (the visuals, audio, animations, etc.) have now arguably become almost as important as the interaction itself. Now that we have surpassed the age where people get instantly awed and amazed at the existence of technology like VR, projection mapping, etc. it feels like new, relevant yet modern forms of output need to continuously be developed.

In Krueger’s Responsive Environments paper, he argued that in order for an interactive medium to respond intelligently, “it must know as much as possible about what the participant is doing”. It is important for the computing machine to obtain as much multi-sensory information about users inputs as possible to use its algorithmic process to produce a corresponding response. The way in which the medium responds also reflects its intelligence, be it to the users’ position, velocity, or a change of shape.

Expanding that concept into VR environments, I believe that collecting users inputs in great resolution and accuracy is imperative to intelligent responses. Users inputs here can be headset position, rotation, velocity, and acceleration. They can also be from the controllers (all the data mentioned above, plus click detection, drag detection…). For more premium VR headset that employs spatial tracking, users’ position inside and outside of the environment can be utilized.

One potential possibility that promise to be a game changer is the ability to track users’ eyes, which opens infinitely many doors for novel interaction and responses as this mimics how we visually perceive in real life. One example for an intelligent response with regards to eyes tracking is foveated rendering. Foveated rendering uses algorithmic processing to render areas where the user is looking at at a higher resolution than the periphery (which will be blurrier), which produces a more realistic VR environment while saving bandwidth, thus achieving a faster response time. As users move their eyes around, the focal area changes accordingly in a timely manner, thus an intelligent response.

Response acts as a medium to create a predictable and consistent relationship between humans and computers. Krueger suggests that this interaction is, in itself, a new medium of art, comprised of input data received through sensors, cameras, etc., rules designed by the artist which process this information, and an output reflecting these rules and the input. The many steps required to design and implement this relationship are what Krueger views as the medium, as artists in the field are forced to think about how people will engage with the piece. For Krueger, response as a medium is not focused on the output, or visual and auditory responses from the computer, he even argues that this might distract from the relationship between human and computer. He argues that it is rather this relationship which is the primary component of this medium.

The reason why this is a medium in itself is because of its ability to make viewers participating actors in pieces of art. Furthermore, the artist maintains some distance from the piece, providing the rules, systems and creative vision for its operation, but potentially “relinquishing total control” as the piece is experienced and used by people. This contrasts with previous mediums of art, as viewers become users, using their body to influence the piece, and artists are unable to assert their creative vision as strongly as they once did. 

To fully immerse in a VR environment, any small discrepancy disrupts the entire experience. Fundamentally, any type of response in VR should make sense and correspond to the performed action. For example, in a VR environment that includes a battle system, swinging an arm should not result in your leg kick out. Additionally, to respond intelligently, a response in VR must be able to take into account various different behaviors and to distinguish them. There should be consistency in determining behavior and the response.

VR presents an opportunity to explore beyond what is possible in this world. As an avid gamer, an intelligent response in VR in the context of games should be able to tell which action you are choosing and correctly affect the game. Take the example of Beat Saber: when the user slices from up to down, the response in VR should be a slash from up to down as well.

Even though VR headsets are becoming cheaper and more advanced, till the point that some are cheaper than an average phone, personally, I think the reason why VR is pretty much a niche market is because most of its contents still rely on existing technologies added with a few tactics that have not offered the average user a major breakthrough in the way they experience the medium. As Murray quoted McLuhan, “the content of any new medium is an older medium”, from my experience with VR, I personally think that VR is on its way to become expressive. But for now, most experiences in VR, be it 360 videos, immersive video games, or virtual social network is somewhat more on the side of additiveness than expressiveness.

For example, Murray talked about how filmmakers have exploited the properties of film to cut scenes, change focus, create dramatic effects… and that resulted in a transformation of “photo-play” from a recording technology into an expressive medium. While there exists some 360 videos that use spatial sounds and unique perspectives that engage the users in a constant state of self-location and scene-navigation, the majority of them (and also happen to be the most accessible through traditional video streaming platforms like YouTube) are still shot with the same approach used in traditional film-making. The burden of having to move around the immersive world, in this case, is higher than the added benefit of a wider field of view that can be traditionally replicated by using multiple fish-eyes lenses.

I think that from Murray’s four principal properties, the one that VR represents the most is the spatial affordance, however, it is also largely based on the remaining three. To me, spatiality is the definition of VR, as almost any environment can be created in VR. VR is unique because by physically remaining in the same location, the user can “travel” to any space, whether it is a different country or the middle of a patient’s stomach during a surgery. VR is different from other media because it creates the effect of feeling physically present in a certain space, thus being more realistic than imagining a space by reading a book or seeing it on a computer screen.

*

Compared to Murray’s description of the interactive fiction computer game Zork, VR is a huge breakthrough in terms of visualizing and experiencing a space. Whereas Zork provided the user simply with text for imagining spatial situations and making certain choices in them, VR combines giving the user the option of an action with visual and sensory add-ons. A user can not only see these options in front of their eyes in the headset but also hold certain objects or sometimes even move around to experience the environment as vividly as mimicking a real-life situation.

The phrase “virtual reality” just a few years ago meant something that seemed like a toy or just pleasant entertainment. Today, however, VR blends entirely with our everyday lives. It allows us to plunge ourselves into video games as if we were one of the characters. That being said VR creates a type of “fake world”, so when people put on a VR headset, their brain starts to believe that they have entered into a new world where they can actually move around and interact with virtual objects. Nevertheless, VR not only provides an immersive experience but also the sensation of being one with the character. It is very engaging to relate emotionally to the whole atmosphere while personally experiencing everything in detail. I believe in the near future VR will be the main focus. As you have seen, the world of virtual reality is constantly evolving. In fact, it could actually bring hundreds of benefits to any area such as education, health and entertainment.  As an example, from Murray’s reading, Eliza was an imitation of a therapist which was created at MIT. Where users would sit at a terminal and send messages to Eliza, who would respond that would initially involve some processing. Most of her answers were switching out pronouns to make the sentence and answer after some time. As Murray mentions in her readings, there was uncertainty and speculation whether or not it was possible to write a program that could be so persuasive. People actually thought they were talking with an actual therapist, for instance, in the story with the vice president where Eliza was not only persuasive but also very successful as a therapist in her role as well. 

Murray asserts that digital environments are spatial, but her definition of that property more appropriately fits to an assertion that digital environments simulate spatiality. In her own words, these “environments are characterized by their power to represent navigable space” (96). That is to say that the “computer screen is displaying a story that is also a place,” which leads to the “challenge” of “invent[ing] an increasingly graceful choreography of navigation to lure the interactor through ever more expressive narrative landscapes” (100). In the sense that Murray describes, then, VR can represent this property at its most extreme. As graphics and motion tracking and field of view improve over time, there will be a thinner veil between physical reality and a virtual experience. However, to appreciate this potential fully, Murray’s definition has to be expanded. 

Digital environments do not only simulate spatiality; they exist spatially. Computers and the devices in which they are embedded take up space. Digital environments  crisscrosses the world in cables and satellites. These environments draw us into overly relaxed postures, shifting the way we take up space as well. Of course, digital environments do simulate spatiality, but this is trick, one we are acutely aware of. We can willingly suspend our disbelief and pretend digital environments extends beyond our screens. We can overlay useful digital environments onto our real world, the way we do with Google maps. We can also leave these simulations. In VR, these choices still exist, except there is much less, if any, need to suspend disbelief because the user is in the digital environment. Hence, VR takes the spatial principle from being a representation or a display to a truth.

Murray defines additive formulations, or “multimedia” in contemporary terms, a medium which use and application is highly dependent on the nature of the derived mediums. For instance, she gives the example of the “photo-play”, what we call “film” now. During the initial stages of development, film was merely seen as a combination of photo and play, limiting the creative scope of film. Murray describes, the only way in which a medium can progress from an additive form to that of an expressive one is through bold exploration and exploitation from the creator’s part, respecting the medium as one of its own. As such, whether a technology is additive or expressive depends partly on how the technology is applied and the number of bold attempts made by creators since the introduction of the technology. Using Murray’s definitions of the two terms, it is possible to attempt to deduce whether Virtual Reality, one of the many new emerging technologies defining the Fourth Industrial Revolution, is at a stage of being an additive or expressive form. VR combines extends the visual experience one can encounter through a flat screen. Its primary focus is to give users a sense of being in another “reality”. The current application of VR is wide, ranging from military use, entertainment, education, etc. However, it is not yet a complete immersive experience as there still exists physical gadgets – mostly those that require the user to have a heavy head set or carry a remote etc. There has been cave systems (uses projections on walls for immersive experience) to eliminate the need for headsets, but it is still limiting due to the two dimensional nature of the projection. As such, despite the advancements that have been made, there have yet to be a definite technology created specially dedicated to VR. Hence, I personally believe VR is still at a stage of being an additive form – heavily reliant on simulated false reality through pre-existing forms of media.

I think that it is most interesting to think about how VR will utilize the encyclopedic properties of digital environments to create engaging experiences. All of the knowledge present on the internet and networked devices can be explored in a setting that allows complete immersion for users, but how can virtual reality present this information through near full-sensory immersion? I believe that apart from adhering to physical and spatial properties of the real world, virtual reality experiences will have the difficulty of enhancing our experience of movement within the experience. I believe the challenge will be: how can the most engaging aspects of viewing and navigating the world around us be combined with innovative ways of movement and sensory stimulation in virtual reality to enhance the user’s experience? I believe VR will force developers and designers to alter the current state of “encyclopedic knowledge,” which I believe is at the present limited to video, photos and text on the internet, and how new and old knowledge will be transformed into appealing experiences in VR. 

I believe that Murray’s description of Sid Meier’s Civilization as a game that can inhibit our ability to interpret the underpinnings of alternate realities through the experience’s seeming encyclopedic knowledge will remain more or less true in virtual reality. In fact, I personally believe that in its present state, virtual reality may inhibit users’ ability to “ask why things work the way they do” because of the physical constraints that VR poses with its wonky headsets and controllers. The mere fact that I can explore the whole database of the rules and backstory of Civilization in mere seconds and with moving my mouse by a few inches is extremely efficient compared to the required physical, full-body movements in virtual reality. Therefore, I believe VR designers will need to consider how experiences can allow full, conscious participation of users while remaining engaging and requiring minimal mental and physical exertion to question and understand the environment around them.