Swell Beats – Always On Always Connected Final

This semester, I spent some time exploring Virtual Reality in my Worlds on a Wire class, and I wanted to keep exploring it in my Always On, Always Connected final using WebVR. Although WebVR is technologically behind than the Oculus or Vive, it’s not the hottest take that WebVR has the potential to reach the widest audience of any VR technology.

There are a few ways in which VR can explode. One path that many people are taking is to create really good quality content. Another way that VR can explode, that I want to explode, is to create a strong multiplayer experience in which users can interface and interact with one another.

So for my AOAC final, I decided to create a multiplayer VR music experience in which users can “surf” midi files and play music with one another.


I was particularly inspired by this great band website GeorgeandJonathan.com that created these great visuals of their musical notes with three.js.

I loved the idea of being able to view the MIDI notes of a file, and I wanted to be able to ride one of these notes. Looking at a MIDI editor, it reminded me a lot of an overhead view of surfers, and that gave me the idea to create an experience in which you were a surfer, riding the MIDI waves.




Working with Amanda Lee, we built this application using three.js for the graphics and WebVR, Cordova, and p5.js for the sound. We spent a lot of time learning the architecture of three.js, and we used it to animate clouds and the ground so it looked like the camera was moving through space. We used p5 to map the position of the surfer to different notes. We also ended up spending a lot of time trying to figure out how to get the VR code to work. Here’s a short video of what we built for our final.

What’s next

We have a lot more to work on, and I hope to continue building this throughout my time at ITP. Here are some critical next steps for this project.

  • Rethink the music experience — are people creating notes? Or are there notes that people can surf to to make music?
  • Polish up the visual design
  • Use socket.io to bring other players into the musical experience.
  • Connect the Daydream Controller to move the surfer.
  • Add different musical interactions such as pitch bend (up/down), rests, different instruments.

AOAC Final Preso