Ever wondered what it’s like to hear movement? Synesthesia is a condition where two or more of an individual’s senses are “linked.” Someone may see colors that go with sounds, or smell numbers. This experimental video I created does not recreate the experience verbatim, however it is heavily inspired by hearing-motion synesthesia.

Whenever an object crosses the threshold at the center, it is transduced directly into sound. The spectrogram of the sound can be seen on the left side of the screen.

Spectrogram from the Threshold Experiment

Top: The image fed into Metasynth. Bottom: The spectrogram of the resulting audio file.

The process for this piece was rather time-consuming, but surprisingly simple. The raw video was cropped to a one-pixel slice and exported as an image sequence, creating over 5000 individual files. These files were then imported into Photoshop and lined up back to back to produce an image that gave time on the horizontal axis. This was the most tedious part of the process.

The next step was to run this image through a program called Metasynth which uses oscillators at different pitches to transduce images into sound.

The resulting sound file was processed slightly in Reaper to level out the volume. A spectrogram was also created that mirrored the dimensions of the original image.

The last step was simply to use After Effects to synchronize the two scrolling image files (blended together), the sound, and the video to create the final product you see above.