FlipMu

FlipMu Live @ the Watusi from FlipMu on Vimeo.

As part of our live show, Owen and I have been working on creating a tightly synchronized audio/visual experience. While we are still experimenting with the best implementation of balancing audio/visual expressivity & composition with live-interaction, we have developed an approach using Quartz Composer, VDMX, and integrating it all with our Ableton live set. Of course, we couldn’t just project onto any old flat surface, so we’ve built a 3D cube to float between our heads and map the projection onto 3 of its faces. While there is lots of work going on in the FlipMu studio/lab right now creating exciting new interactive visual content, as well as more enticing 3D displays beyond a single cube-shaped object, this post will delve into the why and how of our current gig-ready set up. Also, as an added bonus, we’ve put up a custom Quartz Composer plugin we created and use extensively in our live show for download for those who want to attempt a similar approach! Now on to the gory details…

Part 1: Generating Visuals with Quartz Composer

Perhaps we like to make things harder for ourselves, but Owen and I have always been fascinated with programming live generated visual content. Not that we have anything against using rendered video in audio/visual shows (in fact, many of our favorite audio/visual experiences and artsts/VJ’s work primarily with rendered video content), but our current visual show uses 100% live generated content. Plus, we don’t own a fancy video camera to shoot beautiful HD video content, and aren’t composting and After Effects guru’s…yet

Enter Quartz Composer, our personal tool of choice (right now) for programming and generating live visual content. As many of you know, Quartz Composer is a node-based visual programming language provided by Apple. It uses OpenGL, supports most of Apple’s core technologies, and has an active community of developers and experimenters like Kineme and Anton Marini (aka Vade, aka Badass Dude). Best of all, Quartz Compositions can be directly loaded and manipulated from within other Mac OSX VJing / visual program’s, like VDMX.

Below is a screenshot of one of our Quartz visuals which generates a grid of 3D cubes that we manipulate and sequence with the music.

Notes about our Quartz Composer Integration:

What’s the deal with the MIDI Note Receiver in the Quartz patch above? Wouldn’t it make sense to just do all the midi mapping from VDMX?

Yes! Well…sort of. One of the best parts about Quartz Composers tight integration with VDMX is the ability to “publish inputs” and have them show up for direct manipulation and control via automatically created GUI sliders within VDMX. As nicely documented on the Vidvox Wiki, you can even specify special type’s when publishing inputs, allowing VDMX to automatically create GUI objects like RGB sliders…etc. So at first, we published everything in our Quartz patches that we wanted to be controlled from Ableton through VDMX via midi. Next, we simply “midi-learned” the parameters from their respective GUI objects that VDMX amazingly created for us. Which is where we came into our first major issue. When we switched Quartz patches in VDMX, we weren’t sure if VDMX would save the midi mappings. We thought it could possibly be done by saving VDMX presets for each loaded Quartz patch, and then changing the presets via midi messages from Ableton, but in the end we figured we’d just hard define the midi messages within the Quartz patches themselves. This means that each Quartz visual patch is waiting for specific midi messages to drive its visuals. Definitely not the most elegant solution, but it also saved us a lot of time ‘midi-leaning!’– And, because each Quartz patch has different midi requirements, we came up with a sudo-standard for our patches, where the first two midi notes (Quartz Composers C0 and C#0) adjust the color of objects in the patch (RGB, and Alpha respectively).

IMPORTANT NOTE: Middle C in Quartz Composer is NOT the same as middle C in Ableton. We were very confused when we made our Quartz patches listen to specific midi notes, and then got no response when we sent those notes out from Ableton. Basically, Ableton and Quartz are 3 octaves off from eachother, so ‘C0′ in Quartz is actually ‘C3′ in Ableton! Got to love standards…

What is the Midi2Color quartz module in the photo? I don’t see that in my Quartz library…

At first, we were sending individual midi messages to control R,G,B and A for setting colors in Quartz. In an attempt to reduce the amount of midi messages we were sending, we came up with the following solution to use one midi note velocity message to set colors, and one for Alpha. Based on a tutorial example from Apple, we wrote a custom Quartz Plugin (downloadable at the end of the post) for doing the color changing via the velocity values of C0 and C#0. Essentially, we wrote a simple plugin for Quartz Composer which takes in the velocity from a midi note, and then outputs one of 11 pre-defined RGB colors based on the velocity number, and similarly for the alpha value. It then outputs a quartz Color object that can be used in many of the quartz modules. In the future we will hopefully change the MIDI2Color object to allow the outputted colors to be changed on the fly from within the Quartz Inspector, however, the 11 colors we set worked for our purposes and the time-constraints we had for a gig. We have included the source with the plugin if you wanna get your hands dirty and change the colors yourself, or modify the code at your will!

Part 2: Setting Up Content and FX in VDMX

Now that we have programmed all of our custom Quartz visuals, that are ready to go and listen for incoming midi messages, its time to load them into VDMX for being triggered, filtered with effects, and projection mapped. Again, we are probably doing things really dirty here, but bear in mind, we only had one week to create/program all of our Quartz Visuals, purchase and learn VDMX, tightly sequence an hours worth of visuals to music, do projection mapping for the first time, AND build the object to map onto! Why do we do this to ourselves!?

Step One: Content Set-Up

We set up 3 media bins, one for each face of the cube we were going to project onto. Each quartz patch was then added to the media bins 3 times, once for each face. Next we assigned specific midi notes to each midi bin cell, in different midi ranges for each bin, so that we could dynamically trigger/load the clips for each face via Ableton. We set up clip triggering on a dedicated midi channel to avoid note conflicts / confusion. More on this in the next section… Additionally, we set up three layers, one for each face of the cube, and then set each media bin to only control the content of an individual layer. Finally, we created a fourth layer to display an image on a cube we generated in Quartz, so that we could test our projection on a virtual cube before setting up the real thing.

Step Two: FX

Now that all of our media was present, it was time to experiment with various effects. We were able to get drastic amounts of variation and change with relatively simple Quartz patches by experimenting with VDMX FX (many of which are actually made in Quartz). Each layer / cube face got an FX chain, that allowed different FX to be turned on or off via an Ableton midi track. Again, we decided to give FX a dedicated midi channel to avoid midi conflicts with midi clip triggering and sequencing. The most important part of the FX chain was the last effect in each FX chain. That effect is sir Memo Aktens MSA Quad Warp Quartz Patch. Shown in the screenshot above, this is what allowed us to map onto the individual faces of the cube. More information on MSA’s quad warp  can be found here.

In addition to setting up Ableton clips to enable and disable VDMX fx, we connected fx parameters directly (gotta love VDMX’s Midi/OSC learn!) to a few knobs on my Korg NanoKontrol. The knobs were also connected to various audio fx and processing in Ableton, and so, I had realtime control that would manipulate both the audio and the visuals at the same time. Likewise, this led me to learn about VDMX’s great Behavior chain mapping, and curve definitions. For example, in Ableton I have a knob that controls some delay based effects. I wanted that to also control the Wet/Dry of a visual effect in VDMX, but with a very different response curve to how I moved the knob, as well as inverting the control from 0-127 to 127-0, when turning the knob from fully counter-clockwise to fully clockwise. VDMX’s custom behaviors makes this a breeze!

Step 3: Sequencing and Controlling the Visuals & VDMX from Ableton Live

Now that we have all of our visuals set up in VDMX, FX processing, as well as mapping our images to our cube, its time to compose and sequence everything in Ableton. This is perhaps the most labor intensive part, but is where we can really get creative. As shown in the screenshot of Ableton Live above, our Ableton set is setup as follows. We have three midi tracks

1. VDMX-QTZ: If you remember, each Quartz patch we created is internally listening to different midi messages to change various parameters and drawing commands. This track is how we sequence and control everything. Playing with different note lengths, you can get very different results, from interesting stepped motion, to nice smooth motion if you use small note lengths (as pictures above). Also, you can create some beautiful Ableton-art, and definitely create some interesting gradient patterns …with just some simple ramping of x and y positions of visual elements in a Quartz patch, you can very quickly get synchronized motion to make you music come to life!
2. VDMX-FX: This track controls fx in the VDMX fx chains. Each effect in all three of our VDMX layers are midi learned to a specific note, which we then control from here. In this way, we are able to automate effects in VDMX on and off…etc automatically with scene launches.
3. VDMX-Clips: This track is our main VDMX clip launcher. If you look back at the screenshot of VDMX earlier in the post, you’ll notice each clip in our Media Bins are tied to a specific midi note. In this way, we can automatically launch clips on scene launches.

Well! I can keep going into more of the specifics of how we’re doing things, for example, in addition to sequencing the visuals, and tying other elements to live controls on our gear, many of our visuals also listen to incoming audio via my laptops sound-card, but at this time, we’d like to open up to questions you may have. Hopefully we were able to provide some general insight into one possible way to integrate Quartz Composer, VDMX, and Ableton live together for a tightly synchronized, and expressive audio/visual experience. We’d love to hear how you are doing things different, or if you have any ideas on how we can optimize our current setup. Cheers!

And finally…. DOWNLOAD the Quartz MIDI2Color Plugin!

Share This!