Music Everywhere
My Role: 3D Interaction Designer, UI/UX Designer
Music Everywhere leverages the strengths of augmented reality technology (Microsoft HoloLens) and the MIDI protocol to build a prototype that will engage guests, and provide essential musical concepts, paced actionable lessons and interactive exercises.
I designed the workable user interface for playing music notes and created 3D virtual bands represents different music styles with guests in AR space in order to increase engagement. Dev website: http://www.etc.cmu.edu/projects/music-everywhere/
I designed the workable user interface for playing music notes and created 3D virtual bands represents different music styles with guests in AR space in order to increase engagement. Dev website: http://www.etc.cmu.edu/projects/music-everywhere/
Producer/Musician: Seth Glickman | Programmer: Shantanu Das, Byunghwan Lee | Develop Time: 15 weeks
Why AR + Piano Improvisation?
|
To study improvisation is to learn by actively, and repeatedly, thinking in musical notes, scales, dynamics, phrasing, articulation, context and theory. It is often thought to increase musical awareness and understanding in beginner, intermediate and advanced students.
Augmented reality provides an ideal environment for this type of hands-on learning. Demonstration, guidance and user feedback can all be displayed and facilitated directly on the piano keys themselves. By spawning virtual musicians, the AR-enabled student can then practice new ideas and techniques covered in interactive lessons at their own pace. |
Improvisation
|
Virtual Band
|
Genre Variety
|
Overlay System
|
Keyboard Navigation System
To access different music lessons, I developed a keyboard-centered navigation system which is intuitive and more reliable than the gesture feature in HoloLens. Users get color overlay feedback when pressing real keys that aligned with the virtual menu.
Development
HoloLens’ FOV (feild of view) causes guests to have to angle their head downward in order to see the cut-off keyboard.
First, Start with overlays on keys, then cut the overlay to half to quarter in order to decrease the dependency of the overlay gradually.
Second, in order to guide users to look up a little bit, we remove the overlay on keys and put markers which point the key to play.
Last, we design floating keys hovering in the air and a mirror keyboard in order to encourage users to look up.
Note Overlay Development
|
|
Final UI
|
Design Decisions
Light blue: stands out in the mixed-reality space. Yellow color: the visual feedback for hitting the right key. Hitting line: all the key should be hit at the top. Stems: indicate pressing time for notes. Numbers: represent the correct finger to hit the key. Overlay size: differentiates black key and white key. Saturation Change: shows a note's life time. Sequence of notes: shows the expectation for the next note. |
UI Playtest
Experience Map
Virtual Bands
Having a guest play along with animated musicians increase engagement and interaction. For many exercises, the virtual players would play along with the guest, all playing the same notes together. In other cases, they could be used to give the guest the simulated experience of playing along with others.
3D Models
Virtual characters based on the real musicians and designed representative art style for each music genre.
|
|
|
|
Stage Design
In order to enhence the experience of different music , I also designed stage for each band after analyzing the proper color, scale,shape and position in HoloLens to make everything visually cohesively.
Animation Design
I designed four animation phases for band improvisation practices: Dropping, Waiting, Playing, Leaving which fit into user's behavior during lessons. Therefore, the animation will be less distractive in order to let guests focus on the mirror key interface more.
|
1. Dropping
3. Playing
|
2. Waiting
4. Leaving
|
Lessons Learned
For UI design, I found inspiration from Rock Band, and adapted its principles to an AR platform. I think leveraging existing design from companies who have researched and iterated their product is important. To develop UI for emerging technologies, adapting what works on other platforms helps us shape new visual design for the new technology as it matures.
Music Everywhere received positive feedback from GDC 2017. People thought AR-assisted learning was a great idea for future music education. We believe we have a significant head start compared with other software in the field, due in large part to our unique approach to improvisational music learning. Throughout the development, we were continually surprised by how many qualified professional musicians, music teachers, music technologists and CMU faculty were willing to spend time brainstorming, sharing experience and advice.
Music Everywhere received positive feedback from GDC 2017. People thought AR-assisted learning was a great idea for future music education. We believe we have a significant head start compared with other software in the field, due in large part to our unique approach to improvisational music learning. Throughout the development, we were continually surprised by how many qualified professional musicians, music teachers, music technologists and CMU faculty were willing to spend time brainstorming, sharing experience and advice.