ACTION: Cross-Modal Cinematics, Auteur Classification, and Audio-Visual Structure in Film

Content-based analysis of video using audio and visual features has previously been used for the automatic tasks of scene/shot segmentation and video summarization. We present new work that extends this research to automatically extract and compare the narrative structure of feature films, discover patterns in the relationship of music, sound, and image, and classify films according to their director using audio, visual, and joint audio-visual features. Our experiments utilize a new open source toolkit called ACTION—Audiovisual Cinematics Toolkit for Interaction, Organization, and Navigation of film content—which provides tools for extracting, segmenting, visualizing, and classifying audio, visual, and joint audio-features from digitized film.