Shannon Novak and ‘Tonnetz’

Shannon Novak and ‘Tonnetz’
Professor Martin Rieser

“Shannon Novak’s exhibition, Tonnetz, is based primarily on research in the field of harmonic science that suggests there is an unseen biological lattice that underlies visible cellular structures, a lattice that generates and responds to sound” (The McKinney Avenue Contemporary website).

In musical tuning and harmony, the Tonnetz  (German for “tone-network”) is a conceptual lattice diagram representing tonal space, first described by the mathematician Leonhard Euler in 1739. In Novak’s work, the assumed living harmonic relationship between sound, colour, and form is of even more ancient origin. There exists a long history of the notion of the ‘music of the spheres’, in which the movements of the celestial bodies create their own distinct musical signature. The sonorous resonances of the celestial bodies were said to be inaudible to most human ears, with notable exceptions, such as Pythagoras himself, who was rumoured to have detected these musical emanations. Plato linked the major second and perfect fifth to yellow and the perfect fourth to red, in an extension of the Pythagorean harmony of the spheres to encompass planets, tones and colours. Aristotle suggested a parallel between the harmony of colours and the harmony of musical intervals. Newton when investigating the spectrum of light, linked the intervals tone, minor third, fourth, fifth, major sixth, minor seventh and octave to the colours red, orange, yellow, green, blue, indigo and violet.

That there are neurological patterns in the brain that account for any perceived relationship between chromatic values and sound vibration may be suggested by colour synesthesia, or chromesthesia, which affects a small number of people who see sound in terms of colour. The involuntary ability to hear colour, see music as shapes or even taste words results from an accidental cross-wiring in the brain that is found in one in 2,000 people, and in many more women than men. Shannon too is a synesthete, and describes his Augmented Reality interventions at The McKinney Avenue Contemporary as synesthetic responses to space and objects “For example, the drinking fountain when not in use is supposedly quiet, yet to me it is loud – I synesthetically experience the object as a series of sounds, colours, and forms as revealed through the Augmented Reality intervention”.

However synesthetes show the same trends as non-synesthetes do. For example, both groups say that loud tones are brighter than soft tones, and that lower tones are darker than higher tones. Synesthetes nevertheless choose more precise colours than non-synesthetes and are more consistent in their choice of colours given a set of sounds of varying pitch, timbre and composition. Recently, a group of neuroscientists have been able to prove that synesthetes do indeed “see” sound. A series of brain scans showed that, despite being blindfolded, synesthetes showed “visual activity” in the brain when listening to sounds (Nunn et al., 2002).

Russian-born artist Wassily Kandinsky was a synesthete and is widely credited with making the world’s first truly abstract paintings, but his artistic ambition went even further. He wanted to evoke sound through sight and create the painterly equivalent of a symphony that would stimulate not just the eyes but the ears as well. He not only removed all recognisable subjects and objects from Western art around 1911, but also achieved a new pictorial form of music.

Olivier Messiaen, the French composer and organist claimed that his complex chords and rhythms came to him in “coloured dreams” in which he saw blue, red, and green spirals moving and turning with the sounds. “When I hear music, I see in the mind’s eye colours which move with the music. This is not imagination, nor is it a psychic phenomenon. It is an inward reality.” He composed many synesthetic works such as Chronochromie-Strophe I (1960).

The use of technology to explore the relationship between colour and musical harmony is not a new one. In the early years of the 20th Century, an English Professor Alexander Wallace Rimington determined to show the possibilities of the colour-sound relationship; he designed and largely made the electric Colour Organ apparatus. He believed that musical sound and colour were both due to vibrations that stimulated the optic and aural nerve endings. So if one could perform music whilst creating lighting effects that, metaphorically speaking, were on the same wavelength, it would be possible to capture more realistically the composer’s intentions. The Colour Organ was some ten feet high, with a five octave keyboard which was similar to that of a church organ, being controlled by stops. A line of “colour keys” was situated above the conventional (sound) keyboard, and connected to a lens-and-filters system, so that coloured light projections could be “played”. Stage drama might be enhanced, too, and in this sense there was perhaps some influence from Bayreuth, for Richard Wagner was certainly interested in colour, as Professor Rimington knew. Experimenting with electrically driven gadgetry, Professor Rimington devoted much of his London residence to his Colour Music installation.

But there is a further context in which Novak’s work needs to be understood, and that is in the development of mobile Augmented Reality. Augmented Reality (AR) is employed here as a new artistic medium, adding virtual content to a given space that is experienced in real-time and in semantic context with the real-world environment. Canonical sites can be appropriated and augmented. These virtual installations are reviewable with a smartphone application. The layering of information over 3D space produces new ways to experience content that is fuelling the broader migration of computing from the desktop to the mobile device, bringing with it opportunities for broader user dynamic engagement with space and social media. 

Artists and other users are being encouraged to view their smartphones, iPods, and tablet computers as tools for production and display.  Augmented Reality tools can be used to explore concepts in ways that are ‘user led’ and increasingly participatory. In 2010, an Augmented Reality art show  “Augmented Reality: Rogue Art Exhibition” made its debut at The Museum of Modern Art (New York). The physical show was visible to regular visitors, but those who were using a mobile phone application called Layar on their smartphones could see additional works on each of the floors, merging form and content in a non-didactic way.

As computing leaves the desktop and spills out onto the pavements, streets and public spaces of the city, we increasingly find information-processing capacity embedded within, and distributed throughout the material fabric of everyday urban space. Ubiquitous computing evangelists have heralded a coming age of an urban infrastructure, capable of sensing and responding to the events and activities transpiring around them. Imbued with the capacity to remember, correlate and anticipate, this near-future “sentient” city is envisioned as being capable of reflexively monitoring its environment and our behaviours within it – becoming an active agent in the organization of everyday life in urban public space. However beyond such techno-utopianism, the use of mobile technologies can give new agency to the public, and create a distinctive meld of embedded history and imagination.

This extension of interactive technology from fixed installation to real urban geographies is radically altering the modes of audience participation and reception. When the physical space overlaps the space of imagination, the emergent space for art and performance appears to open new perceptions of space and place in the audience.  We need a redefinition of the concept of physical space (including hybrid environments), since through such technologies a new form of urban space seems to be emerging, which is not primarily visual, but in essence, conceptual. In his book The Practice of Everyday Life, Michel de Certeau defines urban space according to the patterns of those who use it. He suggests, “…space is composed of intersections of mobile elements. It is in a sense actuated by the ensemble of movements deployed within it . . . In short, space is a practiced place” (De Certeau, 1984, p. 117).

Increasing use of 3D displays has come at the same time as increasing use of gestural input using technologies such as Microsoft’s Kinect and the advent of portable and immersive wearable computing such as Google Glass. The combination of these technologies offers potential for much richer and more natural user interaction and immersive experiences. The advent of mobile technologies has placed powerful computers in the pockets of more people than have ever possessed a desktop PC. It has created new affordances for creatives out there in real space, dissolving the traditional gallery and museum walls and has allowed new audiences to relate to the spaces of their urban worlds by turning them both into places and spaces of possibility, where inner and outer spaces, histories and narratives can be interlocked and explored. It has allowed the user the privilege of co-authorship via social media and other two-way interventions.  In a sense we are looking at the very beginning of a new communication form, one that can exist in both the hybrid world incorporating the new “Hertzian” spaces and in the imagination of the new audiences.

Augmented Reality, a “field … in which 3-D virtual objects are integrated into a 3-D real environment in real time” (Azuma, 1997), is on the verge of becoming a significant cultural and aesthetic phenomenon. Due to the rapid technological evolution of smartphones, video projectors, video game consoles, and other electronics, it is no longer a “slightly futurological domain” (Ryan, 2004). Despite this cultural significance, Augmented Reality is still a widely uncharted territory for the arts. Implementations of Augmented Reality merge material and virtual places to create hybrid environments. Due to the mobility of smartphones and head-mounted displays, the whole world is a potential setting for interactive Augmented Reality environments. Events in these environments may be determined not only by general topographical features and architectonic details, but also by historical, social, and cultural connotations of a specific place.

As opposed to the tradition of landscape painting, which situates the viewer in a frontal relationship to a carefully enframed array of visual information, Novak’s work affirms the materiality of the invisible world. It shows environment to be a multi-layered, multi-sensory, and dynamic interplay of forces that cannot be encapsulated and circumscribed by the frame. The coup d’oeil cannot sufficiently capture the fact that environment is a complex ecological system formed by the intersection of both visible and invisible phenomena. What, after all, signifies the presence of “nature” more than the call and response of birdsong or the rise and fall of chirping cicadas? Sound provides a sensory framework that imparts depth and texture to the listener’s surroundings. It does not simply enrich the visual experience of landscape, according primacy to vision, but actively shapes the interaction between listener and environment.

Novak’s simple elegance of visual annotation overlays the architectures of the city with precise visualisations of music, using a coda every bit as accurate as Laban’s dance notation or as the synesthetic interpretations of music that inspired Kandinsky. We stand on the threshold of an emergent art form embodied here in the novel visualisations of this experimental techno-artist.

© Professor Martin Rieser 2013

Professor Rieser has worked in the field of interactive arts for many years. He is a research Professor at the Institute of Creative Technologies in the Faculty of Art Design and Humanities at De Montfort University, Leicester, United Kingdom. His art practice in Internet art and interactive narrative installations has been seen around the world including Cannes, Holland, Paris, Vienna, Thessaloniki, London, Germany, Milan, and Australia. He has published numerous essays and books on digital art including New Screen Media: Cinema/Art/Narrative (BFI/ZKM, 2002), and has recently edited The Mobile Audience, a book on locative technology and art from Rodopi.

References

Azuma, R. T. (1997). A survey of augmented reality. Presence-Teleoperators and Virtual Environments, 6(4), 355-385.

De Certeau, M. (1984). The Practice of Everyday Life. Berkeley: University of California Press.

Nunn, J. A., Gregory, L. J., Brammer, M., Williams, S. C. R., Parslow, D. M., Morgan, M. J., & Gray, J. A. (2002). Functional magnetic resonance imaging of synesthesia: activation of V 4/V 8 by spoken words. Nature Neuroscience, 5(4), 371-375.

Ryan, M. L. (2004). Cyberspace, cybertexts, cybermaps. Dichtung-Digital-Journal für Digitale Ästhetik, 6(31).