In 2013-14, Bill and I were invited to do an artists-in-residence at the University of North Carolina, Colleges of Computing and Informatics and Art and Architecture. While there, we collaborated with faculty and students throughout the campus in diverse areas such as software, architecture, engineering, machine vision, psychology and dance on a project titled the willful marionette. Our goal was to build, from the scanned image of a human figure, a stringed marionette that responds in real time to spontaneous and interactive human movement, engaging with a viewer by reading their movements and expressions.

 

Our work has focused on the tenuous nature of existence through the digital remapping and reconfiguration of the human body and our practice has employed, among other technologies, computer graphics, three-dimensional scanning and motion-capture to explore issues related to this. In prior works we animated the three-dimensional scans of dancers by attaching motion-captured choreography to them, creating virtual marionettes in video animations. This is how we first became interested in marionettes. the willful marionette strings are manipulated by motors and software. There are two depth sensors that read and analyze the behaviors and gestures of participants that, in turn, cause the marionette to react. The puppet's subsequent actions are designed to elicit further responses, creating an exchange focusing on the frailty and insecurities of the human participant and raising issues of contemporary relevance.

 

Our interests in this project are more psychological than mechanical and the marionette has simple movements that show it to be aware of a person's actions and to engage with them subtly. Our intention was not to create so much a perfectly functioning robot but rather to imbue an obviously mechanically actuated marionette with the ability to solicit a physical and emotional dialog with a viewer. A marionette’s ability to abstract human characteristics and their disconnected relationship to us lends them qualities that are unnerving and fascinating at the same time. The marionette has always been a reflection of our own actions and it has played an important role throughout human history. It has always though, been a distortion of humankind. Their heads are bigger, fingers and noses longer and their movements exaggerated, making them “less human”. What would be considered inappropriate behaviors by human actors are acceptable, even expected, in marionettes. This and their performative aspect, in its scale and its historical implications, is extremely intriguing to us. Contemporizing the marionette and its theater using robotics and computer vision to make it “more real” creates a changed dialog with the viewer and allows the interaction between them to become psychologically more intimate and potentially more vulnerable. This project, additionally, by utilizing several advanced technologies in the service of a traditional art form, bridges a gap in the contemporary art experience between the static viewing of the past and the increasingly interactive and dynamic experience to be found in work that allows the art and the viewer to interact in unpredictable ways.