FlipMu

Nuance Audio & Sensor Recorder Preview from FlipMu on Vimeo.

Lately I’ve been focusing much of my time on the development of an application, Nuance (working title–suggestions are encouraged), in order to explore various research goals which are part of my PhD.

So what is Nuance?

Nuance is a software application for recording synchronous audio and sensor data. Nuance provides a fast and intuitive interface for rapidly developing custom experiment-centric configurations, which can easily be saved and reloaded at a later time. Custom hyper-instruments and sensor systems (or other serial / osc [soon] devices) can be defined in simple XML-based protocol files, allowing them to be dynamically loaded and recorded using Nuance. Oh yah, and with minimal pain!

Why Nuance?

The primary goal of Nuance is to create a means for capturing synchronous audio & sensor data from musicians & other performers, for analysis and/or machine learning experiments. This task is very important for my PhD research, and I have yet to find a simple third-party solution which is easy to set-up, and requires very little configuration. I figured, why not spend a little time developing an application which would allow this sort of rapid experimental set-up. With Nuance, whenever I have an experiment with a completely different/custom sensor system or instrument, I can define a super-simple xml file for the system, and in a couple minutes create a custom Nuance configuration to capture everything. Best of all, I can save the configuration so I can reload the set-up and capture new data later in the future.

Example Please?

Recently I posted about a sensor system I put together at the request of a colleague of mine for a sort of hyper-saxophone (original article here). The sensor-system consisted of 3 Force Sensing Resistors, 1 Triple axis accelerometer, 2 Push buttons, and was powered by an Arduino. If we wanted to capture this data using Nuance, all we would have to do is program the Arduino to send out the serial data with the following format: SensorStartMessageDATA –so for the force sensing resistors on the saxophone sensors, the Arduino would just be set up to output the sensor name followed by the data and a new line character (achieved by using println in the Arduino source below).

In arduino sudo-code it would look something like this:

void loop() {
  // read the values from the sensors (only first two fsr's shown in this example)
  fsr1Value =  analogRead(fsr);
  fsr2Value =  analogRead(fsr2);

  //serial.print  & serial println for sensors (only first two fsr's shown in this example)
  Serial.print("fsr1");
  Serial.println(fsr1Value);
  Serial.print("fsr2");
  Serial.println(fsr2Value);
  delay(10);
}

Now that the Arduino is sending the serial sensor data in the right format, all you have to do is define a simple XML config file for Nuance so that it can parse the data. Here’s an example for the two FSR’s above.

XML Protocol

<TEST_PROTOCOL>
    <DATA>
        <ITEM ID="Force Resistor 1" start="fsr1"/>
        <ITEM ID="Force Resistor 2" start="fsr2"/>
    </DATA>
</TEST_PROTOCOL>

The ID field is for a human readable name which will be selectable from a drop-down menu in a SensorRecorder object in Nuance. Finally, the start field is the SensorStartMessage you defined in the Arduino code.

Thats it! When you load the config in a Nuance SensorRecorder object, it will automatically populate a drop down menu for you to select the sensor you want the SensorRecorder to listen to, and take care of all the serial parsing behind the scenes. Watch the video above for an example.

Nuance is still in early development, but is coming along quite quickly. Special thanks to Jules for his help, examples, and especially for creating the JUCE library from which Nuance builds off of. Stay tuned for its imminent release!