SoundVisionComputation

The “groove” of a song correlates with enjoyment and bodily movement. Recent work has shown that humans often agree whether a song does or does not have groove and how much groove a song has. It is therefore useful to develop algorithms that characterize the quality of groove across songs. We evaluate three unsupervised tempo-invariant models for measuring pairwise musical groove similarity: A temporal model, a timbre-temporal model, and a pitch-timbre-temporal model.

Content-based analysis of video using audio and visual features has previously been used for the automatic tasks of scene/shot segmentation and video summarization. We present new work that extends this research to automatically extract and compare the narrative structure of feature films, discover patterns in the relationship of music, sound, and image, and classify films according to their director using audio, visual, and joint audio-visual features.

From exposing Jackson Pollock forgeries to clarifying the sections of the Federalist Papers written by Alexander Hamilton, computational analysis and machine learning have proven to be powerful tools in the study of authorship. Film scholar Warren Buckland used the statistical analysis of shot lengths and shot types to make a persuasive claim that Tobe Hooper, and not Steven Spielberg as rumored, directed Poltergeist (1982). However, Buckland and other scholars using Cinemetrics have had to manually enter data for these elements.

The EMdrum, a drum electromagnetically actuated in the manner of a loudspeaker, is presented. Design principles are established and implementation is described; in particular, two alternative electromagnetic actuation designs, moving-coil and moving-magnet, are discussed. We evaluate the time-frequency response of the instrument and present a musical application.

SonicTaiji is a mobile instrument designed for the Android Platform. It utilizes accelerometer detection, sound synthesis, and data communication techniques to achieve real-time Taiji sonification. Taiji is an inner-strength martial art aimed at inducing meditative states. In this mobile music application, Taiji movements are sonified via gesture detection, connecting listening and movement. This instrument is a tool for practitioners to enhance the meditative experience of performing Taiji. We describe the implementation of gesture position selection, real-time synthesis, and data mapping.

This project presents a control surface that combines a grid of photocells with a microcontroller to allow a musician to manipulate multiple analog components at once. A brief background on past uses of photocells for music and film composition and instrument-building introduces a few different implementations and performance contexts for the controller. Topics such as implementation, construction, performance scenarios and reflections on past performances of the controller are also discussed.

The optical soundtrack has a long history in experimental film as a means of image sonification. The technique translates image luminance into amplitude along the vertical axis, enabling the sonification of a wide variety of filmed patterns. While the technical challenges of working with film preclude casual exploration of the technique, digital implementation of optical image sonification allows interested users with skill sets outside of film to access this process as a means of sonifying video input.

With an optimal network topology and tuning of hyperparameters, artificial neural networks (ANNs) may be trained to learn a mapping from low level audio features to one or more higher-level representations. Such artificial neural networks are commonly used in classification and re-gression settings to perform arbitrary tasks. In this work we suggest re-purposing auto-encoding neural networks as musical audio synthesizers.

"Resynthesizing Perception" immserses participants within an audiovisual augmented reality using goggles and headphones while they explore their environment. What they hear and see is a computationally generative synthesis of what they would normally hear and see. By demonstrating the associations and juxtapositions the synthesis creates, the aim is to bring to light questions of the nature of representations supporting perception. Two modes of operation are possible.