Project #07.1: Multimodal Guitar

Demo

embed­ded by Embed­ded Video

Abstract

This project aims at study­ing how recent inter­ac­tive and inter­ac­tion tech­nolo­gies would help extend how we play the gui­tar, thus defin­ing the “mul­ti­modal gui­tar”. We inves­ti­gate two axes, 1) “A gestural/polyphonic sensing/processing tool­box to aug­ment gui­tar per­for­mances”, and 2) “An inter­ac­tive gui­tar score fol­low­ing envi­ron­ment for adap­tive learn­ing”. These approaches share quite sim­i­lar tech­no­log­i­cal chal­lenges (sens­ing, analy­sis, pro­cess­ing, syn­the­sis and inter­ac­tion meth­ods) and dis­sem­i­na­tion inten­tions (community-based, low-cost, open-source when­ever pos­si­ble), while lead­ing to dif­fer­ent appli­ca­tions (respec­tively artis­tic and edu­ca­tional), still tar­geted towards expe­ri­enced play­ers and beginners.

A gestural/polyphonic sensing/processing tool­box to aug­ment gui­tar performances

We designed and devel­oped a tool­box for mul­ti­modal gui­tar per­for­mances con­tain­ing the fol­low­ing tools: Poly­phonic Pitch Esti­ma­tion, Fret­board Group­ing, Rear-mounted Pres­sure Sen­sors, Infi­nite Sus­tain, Rear­rang­ing Looper, Smart Har­mo­nizer. The Modal Syn­the­sis tool needs be refined before being released.

An inter­ac­tive gui­tar score fol­low­ing envi­ron­ment for adap­tive learning

We designed a low-cost offline sys­tem for gui­tar score fol­low­ing. An audio modal­ity, poly­phonic pitch esti­ma­tion from a mono­phonic audio sig­nal, is the main source of the infor­ma­tion, while the visual input modal­ity, fin­ger and head­stock track­ing using com­puter vision tech­niques on two web­cams, pro­vides the com­ple­men­tary infor­ma­tion. We built a sta­ble data acqui­si­tion approach towards low infor­ma­tion loss. We built a probability-based fusion scheme so as to han­dle miss­ing data; and unex­pected or mis­in­ter­preted results from sin­gle modal­i­ties so as to have bet­ter multi-pitch tran­scrip­tion results. We designed a visual out­put modal­ity so as to visu­al­ize simul­ta­ne­ously the gui­tar score and feed­back from the score fol­low­ing eval­u­a­tion. The audio modal­ity and parts of the visual input modal­ity are already designed to run in real­time, we need to improve the mul­ti­modal fusion and visu­al­iza­tion so that the whole sys­tem can run in realtime.

Results

The one-month work­shop of this project was run through the eNTERFACE’09 Sum­mer Work­shop on Mul­ti­modal Inter­faces, from July 13th to August 7th, at Casa Paganini, Info­Mus lab, in Gen­ova, Italy.

Doc­u­ments

  • [2010,inproceedings] bib­tex Go to document
    Rebour­sière Loïc, Fris­son Chris­t­ian, Lähdeoja Otso, Mills III John Ander­son, Picard Cécile, Todo­r­off Todor, « Mul­ti­modal­Gui­tar: a Tool­box for Aug­mented Gui­tar Per­for­mances », in « Pro­ceed­ings of the New Inter­faces for Musi­cal Expres­sion++ (NIME++) », Syd­ney, Aus­tralia, 2010.
    @INPROCEEDINGS{MultimodalGuitarNIME2010,
      author = {Lo\"{i}c Reboursi\`{e}re and Christian Frisson and Otso L\"{a}hdeoja and John~Anderson {Mills~III} and C\'{e}cile Picard and Todor Todoroff},
      title = {Multimodal{G}uitar: a Toolbox for Augmented Guitar Performances},
      booktitle = {Proceedings of the New Interfaces for Musical Expression++ (NIME++)},
      year = {2010},
      address = {Sydney, Australia},
      month = {June 15-18},
      url = {http://www.educ.dab.uts.edu.au/nime/PROCEEDINGS/papers/Demo%20O1-O20/P415_Reboursiere.pdf}
    }
  • [2009,inproceedings] bib­tex Go to document
    Fris­son Chris­t­ian, Rebour­sière Loïc, Chu Wen-Yang, Lähdeoja Otso, Mills III John Ander­son, Picard Cécile, Shen Ao, Todo­r­off Todor, « Mul­ti­modal Gui­tar: Per­for­mance Tool­box and Study Work­bench », in « QPSR of the nume­di­art research pro­gram », 2009, pp. 67–84.
    @INPROCEEDINGS{numediart_2009_s07_p1_report,
      author = {Christian Frisson and Lo\"{i}c Reboursi\`{e}re and Wen-Yang Chu and Otso L\"{a}hdeoja and John~Anderson Mills~III and C\'{e}cile Picard and Ao Shen and Todor Todoroff},
      title = {Multimodal Guitar: Performance Toolbox and Study Workbench},
      booktitle = {{QPSR} of the numediart research program},
      year = {2009},
      editor = {Thierry Dutoit and Beno\^{i}t Macq},
      volume = {2},
      number = {3},
      pages = {67-84},
      month = {9},
      organization = {numediart Research Program on Digital Art Technologies},
      url = {http://www.numediart.org/docs/numediart_2009_s07_p1_report.pdf}
    }

Soft­ware

  • The Mul­ti­modal Gui­tar Artis­tic Per­for­mance Tool­box is cur­rently avail­able for Max/MSP 5 (soon for Pure­Data too!):

      Mul­ti­modal­Gui­tar v1.2 (454.0 KiB, 348 hits)

  • Stay tuned for the upcom­ing Mul­ti­modal Gui­tar Score Fol­low­ing Work­bench in Open­In­ter­face!