Processing:
Processing IDE:
Class activities: exploring examples of what processing can do, connecting Arduino to Processing and exploring image processing.
Category: Designing Narrative Experiences
Notes:
Projection Mapping diagram:
Projection mapping examples:
Claude Monet: The Immersive Experience: projection mapping in gallery space
Van Gogh London Exhibit: The Immersive Experience: projection mapping in gallery space
’Borderless’ by teamLab: projection mapping in gallery space
‘Gallery Invasion’ by Skullmapping: interactive gallery experience
‘Le Petit Chef’ by Skullmapping: Interactive dining experience
‘Box’ by Bot and Dolly: Interactive robotic performances using projection mapping to simulate graphics
’Photosynthesis’ by Sinoca Studio: Projection Mapping used in Natural Contexts
Class activities:
MadMapper introduction: connect the computer to a projector and download the MadMapper demo version. This allows you to project images and videos directly onto the wall. We first started by using simple animation and graphics. Then we tried fitting these within a designated space, the picture frame.
Design challenge- Project mapping on 3D surface:
Objective: to transform a simple 3D surface through projection mapping. The focus will be on using media to reinterpret the it in a creative, narrative-driven way. Think of it as not just as a surface but as an object with potential meanings. What story could you tell through this shape?Use MadMapper to fit the media onto the surface, ensuring that the visuals align with the object’s physical form. The projection should go beyond aesthetics. What is the narrative or message? Consider how light, colour, and movement interact with the object to build the narrative.
Projection mapping on face by Wildbytes for Ponds Cold Cream: inspiration for practical work.
MapMapper guides on Blackboard
Flowers and People – Dark by Teamlab:
This interactive artwork is not a prerecorded image played back, it’s generated in real time. The continuous interaction between individuals and the installation results in a constantly evolving, generative artwork. The flowers undergo a cycle of budding, growth, and blossoming before their petals eventually wither and fade. This process of growth and decay continuously repeats itself. The proximity of the viewer to the artwork determines whether the flowers shed their petals simultaneously, wither and die, or rejuvenate and blossom again.
Flutter of Butterflies Beyond Borders, Ephemeral Life- Teamlab:
When no one is in the space, it is completely dark and nothing exists. When people enter the space and stand still, butterflies are born from their feet where they stand, or from their hands where they touch the wall. When people touch the butterflies themselves, the butterflies die. The butterflies fly beyond borders and through other artworks, entering monitors or leaving rooms, dissolving the boundaries between artworks and flying through other works. By flying seamlessly through the other pieces, the artwork is freed from the concept of “frames” and blurs the boundaries between works. The artwork is not a pre-recorded image that is played back: it is created by a computer program that continuously renders the work in real time. The interaction between people and the installation causes continuous change in the artwork.
Walk, Walk, Walk- Teamlab:
The figures cross the boundaries of other works and continue to walk around, influencing the artworks that they enter into. Sometimes they enter other spaces and continue as a new artwork. The group of figures keep walking, meaning that the artwork is constantly moving. When people touch the figures, they react, sometimes stop, sometimes change directions and choose a different path. “Hoho kore dojo” is a Zen expression that conveys how every step is a place to learn. The figures in the artwork walk endlessly as they confront various situations.
Glenn Kaino’s magical and immersive forest installation:
Flutter Wall- Dominic Harris:
In Flutter Wall the viewer is presented with colourful butterflies and have the opportunity to touch the creatures without causing harm. The contact brings the insects to life. The work can quickly transform into a swarming kaleidoscope of butterflies, their frenzied flight dominating the screen before once again they settle to create a stunning array of colour.
The Treachery of Sanctuary- Chris Milk:
This is a large-scale interactive piece which tells the story of birth, death, and transfiguration that uses projections of the participants’ own bodies to unlock a new artistic language. The work consists of three 30-foot high white panel frames suspended from the ceiling on which digitally captured shadows are reprojected. 3D models of birds interact with the shadows captured by three hidden Kinects.
Awake- Sofia Aronov:
Awake is an exploration of the future of illustration: new technologies allow objects to respond to the environment and to the observer, paintings become alive and viewers are not passive anymore.
Interactive Projection Mapping Installation:
https://www.bareconductive.com/blogs/resources/create-an-interactive-projection-mapping-installation
Interactive Projected Light with Kinect V2 Sensor- Johnny Devine:
https://www.devinejohnny.com/projectdetails/interactive-project-light-with-kinect-v2-sensor
Pre-reading:
‘Design beyond vision’ by Ellen Lupton:
Class notes:
Buzzer:
NeoPixel LED:
Servomotor:
Pre-Reading:
‘Cultural Probes and the Value of Uncertainty’: Cultural Probes are collections of playful, open-ended tasks or artifacts designed to provoke inspirational responses from people, rather than generate precise or comprehensive data or solve a problem. This emphasises the importance of playfulness in the design process. Cultural probes encourage a design approach that is more about discovery, empathy, and subjective interpretation than solving practical problems. Some of the key takeaways are embracing uncertainty, subjective interpretation and play as a design tool.
Extracts from ‘Lupton and Lipps’ (2018):
Notes:
Examples of Multi-Sensory Design:
Light Theremin by Kevin Powell Studio: There are two potentiometers that vary the pitch and the oscillation speed of the LED in the finger. That LED is used to get the theremin “wobbly” effect through the light dependent resistor on the bot’s temple. The 3.5mm jack in the mouth is the output. There’s another 3.5mm jack on the back of the head for easily removing the hand connection.
‘In Real Life’ by Olafur Eliasson’s at Tate Modern:
‘Bouquet’ by Niklas Roy: a synaesthetic olfactory device which allows the user to perceive color through fragrances
3D Printed Interactive Wearable Designs by Anouk Wipprecht:
Intro to Sensor Kits: These are made up of several modules, each module being able to either input or output different types of data, therefore being able to support different interactive installations and experiences.
The Modules:
Arduino Sensor Kit:
Analog Inputs:
Button controlling LED:
Potentiometer Upload the code and see a live variable change in real-time as you turn the knob.
Light Sensor: Stream data from the photoresistor (a sensor that measures the amount of light ) into the Arduino software. As you cover it with your hand and then remove it the value change depending on how much light is being received.
Ultrasonic Distance Sensor (HC-SR04): This means you need to record an analog input of the distance between two objects, measures proximity.
Arduino sensor kits- Analog inputs:
More information about sensor kits:
https://sensorkit.arduino.cc/sensorkit/module/getting-started/lesson/00-getting-started
Design challenge: Chromesthesia (sound-to-colour) synesthesia:
Examples of Interactive Experiences:
‘@><#!!! the life of an overtaxed surface’ by Vera Hausmann, Till Maria Jürgens and Vitus Schuhwerk: A thin metal coated foil with invisible string is connected to servo-motors and an arduino board. This senses if somebody comes close. Through squeezing and releasing, the flat motion turns into a spatial transformation and expands towards the user. It reacts by proximity and physical interaction. The intensity of expression is determined by the user’s interaction. The foil reacts more intensely to more than one interaction. An arduino board controls the interaction.
‘Laser Microphone’ by Kai Lab: Laser Microphone acts like a microscope for sound, amplifying the physical vibrations of tiny objects that are otherwise too quiet to hear. This installation has been used to hear things such as the movement of insect wings, water spray and the movement of bubbles. A laser is used to illuminate a vibrating object. When the material of the object oscillates, it disrupts the beam of the laser and casts a shadow on a light sensor placed opposite the laser. The varying light intensity is translated into an electrical signal, which is amplified by an electrical circuit. This signal can then be fed into an audio playback device. The sound detected by the device is real-time and unprocessed.
‘Twin Objects – Devices for Long Distance Relationships’ by Elise Migraine: Twin Objects is a collection of devices designed to act as a ‘hotline’ in attempt to nurture intimacy that long-distance relationships can lack. It is a playful solution to enhance the experience of instant messaging. It consists of two buttons which react to each other’s positions. When one button is pressed down, it causes the other twin button to move up creating a physical, interactive experience.
‘till you stop’ cake decoration machine by mischer traxler studio: A machine decorates a cake with lines, similar to a Spirograph, until the customer/visitor decides to stop the decoration process. Then in a second process sugar pearls are dropped onto the glazing. The decor is continuously changing and the costumer/visitor has to decide whether they prefer a simple design or a more complex one. Once the decoration machine is stopped it can not be started again.
‘equilumen’ by mischer traxler studio: It consists of a long rod with a glass sphere on each side. When the lamp is balanced, both spheres give off the same amount of light by sharing the lamp’s power equally. Once somebody pulls down one end of the lamp, this sphere will light up to its maximum while the other sphere gets dimmed down. It is a constant game of ‘give and take as soon as someone interacts with the lamp – symbolising that the right balance is delicate and hard to achieve.
Arduino Introduction:
Arduino:
Labelled Arduino:
Labelled circuit:
LED on:
Download Arduino IDE:
Basics of Arduino coding:
Results of the coding:
Bare minimum code:
Blink:
Digital Read Serial:
Fading an LED:
Learn more about Arduino coding:
Pre-Reading:
Ursula K. Le Guin and her ‘Carrier Bag Theory of Fiction’:
’The Arduino Comic’ by Jody Culkin: this comic is a brief introduction into Ardunio, an open source electronics prototyping platform.
Week 1: Expanded Narratives:
Examples of narrative experiences: below are some examples of expanded narrative experiences that utilise digital technologies and audience engagement.
‘Kitty AI: Artificial Intelligence for Governance’ by Pinar Yoldos (2016): This is a fun piece with the narrative that in the future (2039) political governance is replaced by a speaking kitten AI.. This speculative future scenario is inspired by a Black Mirror episode. Audiences interact by talking to the AI. It’s a light hearted way of commenting on the current political governance and I like this way of approaching a serious topic.
‘Present Shock II’ by United Visual Artists (2023): This installation shows the viewer loads of statistical clocks and live news feeds representing real-time information from all around the world highlighting how the speed and volume of data in the Information Age. The continuous installation successfully creates the feeling of “information overload” with a tense, stressful and ominous atmosphere.
‘Please Feed the Lions’ by Es Devlin [Interactive Poetry, Sculpture, Projection] (2018): This interactive installation in Trafalgar Square combines projection-mapped sculptures, fuse light, music & tech. The lion roared poetry, with the words being chosen by the public. The public were asked to “feed the lion” words. By daylight, the constantly changing collective poem was shown by LEDs in the mouth of the lion. By night, the poem was projection-mapped over the lion.
‘G80 (un)Equitable Variables’ by Fragmentin (Interactive Installation] (2023): in this installation, the audience interact with lots of sliders, each corresponding to a variable of one of 80 human conditions. When a single slider is moved, other sliders react and change, showing the how all variables are connected.
The story itself:
Joseph Campbell’s ‘The Hero’s Journey’: describes the common heroic storytelling narrative in which a heroic protagonist sets out, has transformative adventures, and returns home. This storytelling format is being challenged by more audience centric narratives utilising new interactive technologies.
Coraline:
Coraline narrative experience:
2 versions of the same room: I would replicate these 2 rooms by using different lighting, and props. Creating a drastic contrast would be key to making this installation engaging. The layout of the rooms would be the same to create the mirroring effect. There would be no actors involved to play the characters, just the audience and the sets/props. I think this would force all the other elements to be stronger in order to create an immersive experience that stimulates all the senses.
The ‘real world’:
The ‘other world’:
The tunnel: Coraline crawls through this to get from the real world to the ‘other world. I would add this to the narrative experience so the audience would get the tactile experience of crawling through an enclosure space, to build the mystery and tension.
I would light this tunnel similar to this:
Augmented Reality (AR): I want to integrate the idea of this green amulet from the movie into the experience. Using AR technology the audience hold their phone up around the ‘real world’ and it will show them all the beautiful possibilities of the ‘other world’. The striking contrast bewteen what they’d see in person and through their phone would tempt them into the ‘Other world. This plays on the idea of deception and illusion.
AR: this is an example of how this technology would work. They would see the dull, boring room around them and the beautiful, colourful version on their phones.
Button eyes: the buttons are such an iconic part of Coraline so I felt it was important to include them in this narrative experience. There would be a mirror in the ‘real world’ and using AR technology the audience would see themselves with button eyes if they looked through their phone. This foreshadows the danger and deceit of the ‘other world’.
Everything in the ‘other world’ would be fake, the food would be plastic, the smell would be artificial, the decorations would just be flat cutouts or painted. The only thing that is real and audiences could touch and interact with would be this button box in the middle of the table.
Reflection/ prep for next week:
Introduction:
About the module:
Learning outcomes:
Assessments:
Submissions:
Brief:
Timeline of weekly sessions:
Samuel Green Illustration 