FU YEN HSIAO
  • LinkedIn
  • Dribbble
  • MEDIUM

Music Everywhere

​My Role: 3D Interaction Designer, UI/UX Designer
Picture
Music Everywhere leverages the strengths of augmented reality technology (Microsoft HoloLens) and the MIDI protocol to build a prototype that will engage guests, and provide essential musical concepts, paced actionable lessons and interactive exercises.
I designed the workable user interface for playing music notes and created 3D virtual bands represents different music styles with guests in AR space in order to increase engagement.  Dev website: http://www.etc.cmu.edu/projects/music-everywhere/

Producer/Musician: Seth Glickman | Programmer: Shantanu Das, Byunghwan Lee | Develop Time: 15 weeks


​Why AR + Piano Improvisation?

Picture
​To study improvisation is to learn by actively, and repeatedly, thinking in musical notes, scales, dynamics, phrasing, articulation, context and theory. It is often thought to increase musical awareness and understanding in beginner, intermediate and advanced students.
 
Augmented reality provides an ideal environment for this type of hands-on learning. Demonstration, guidance and user feedback can all be displayed and facilitated directly on the piano keys themselves.
​By spawning virtual musicians, the AR-enabled student can then practice new ideas and techniques covered in interactive lessons at their own pace.
​
Picture
Improvisation
Picture
Virtual Band
Picture
Genre Variety
Picture
Overlay System
ui

Keyboard Navigation System

To access different music lessons, I developed a keyboard-centered navigation system which is intuitive and more reliable than the gesture feature in HoloLens. Users get color overlay feedback when pressing real keys that aligned with the virtual menu.
Picture


​Development

HoloLens’ FOV (feild of view) causes guests to have to angle their head downward in order to see the cut-off keyboard.
Picture

​First
, Start with overlays on keys, then cut the overlay to half to quarter in order to decrease the dependency of the overlay gradually. 
​Second
, in order to guide users to look up a little bit, we remove the overlay on keys and put markers which point the key to play.
Last, we design floating keys hovering in the air and a mirror keyboard in order to encourage users to look up.
Picture


​Note Overlay Development
​

1.0. Mirror Keyboard
Overlapping of black keys and white keys

Picture

2.0. Charging Bar
Hard to tell when
​to hit the key

Picture

1.1. Accurate Shape 
No expectation for
the next note 

Picture

2.1. Charging Bar II
 Show a sequence of notes
​ for one key?

Picture

3.0. Layer Bar
Very messy
​Too many colors

Picture

3.1. Layer Bar II
Still a mess
​Too many colors

Picture

4.0. Sequence
Need feedback
​after a key is hit

Picture

4.1. Sequence​ II
Need finger numbers
for each key

Picture


Final UI​
​

Picture
Design Decisions
​

Light blue: stands out in the mixed-reality space.
Yellow color: the visual feedback for hitting the right key.
Hitting line: all the key should be hit at the top.
Stems: indicate pressing time for notes.
Numbers: represent the correct finger to hit the key.
Overlay size: differentiates black key and white key.
Saturation Change: shows a note's life time.
Sequence of notes: shows the expectation for the next note.

ux

 
​UI Playtest

Picture


​Experience Map

Picture


​Virtual Bands 

Having a guest play along with animated musicians increase engagement and interaction. For many exercises, the virtual players would play along with the guest, all playing the same notes together. In other cases, they could be used to give the guest the simulated experience of playing along with others.
Picture

3D Models

Virtual characters based on the real musicians and designed representative art style for each music genre. 
Picture
Picture
Picture
Picture


​Stage Design 

In order to enhence the experience of different music , I also designed stage for each band after analyzing the proper color, scale,shape and position in HoloLens to make everything visually cohesively.

Animation Design 

I designed four animation phases for band improvisation practices: Dropping, Waiting, Playing, Leaving which fit into user's behavior during lessons. Therefore, the animation will be less distractive in order to let guests focus on the mirror key interface more.
1. Dropping 
Picture
3. Playing 
Picture
2. Waiting 
Picture
4. Leaving
Picture


Lessons Learned

For UI design, I found inspiration from Rock Band, and adapted its principles to an AR platform. I think leveraging existing design from companies who have researched and iterated their product is important. To develop UI for emerging technologies, adapting what works on other platforms helps us shape new visual design for the new technology as it matures.

Music Everywhere received positive feedback from GDC 2017. People thought AR-assisted learning was a great idea for future music education. We believe we have a significant head start compared with other software in the field, due in large part to our unique approach to improvisational music learning. 
Throughout the development, we were continually surprised by how many qualified professional musicians, music teachers, music technologists and CMU faculty were willing to spend time brainstorming, sharing experience and advice.
​
© 2018 Fu Yen Hsiao
  • LinkedIn
  • Dribbble
  • MEDIUM