Mark Steelman
XR Developer/Enthusiast
My Journey Exploring
The XR Landscape
How I Got Into
Virtual Reality
Throughout my life I’ve always loved movies and video games and the way they immersed me in an alternate world. When I first tried the HTC Vive at a demo Valve gave at Duke in January 2016, it was a mind-blowing moment. At first, I hardly paid attention to the environment and was simply mesmerized by the tracking quality of the controllers and the room boundaries. When I took off the headset, I genuinely felt like I was in another place. The idea of being able to track anything in 3D space at a low cost and to trick the brain with ease got me thinking about the potential applications of this technology, and I was very curious about what would happen when these headsets became available to anyone.
First Steps: Hardware
September 2016
After many months of reading articles and watching videos about VR, I decided to build my own desktop computer and buy an HTC Vive. I was nervous about spending all of the money I made over the summer in one go, but I felt that there was a serious potential for VR to be my path to entrepreneurship and that it would be worth becoming an early adopter. When I got my headset, I realized how little content existed for such incredible hardware and I became eager to create VR experiences that didn't yet exist. I was shocked at how easy and fun it was to get a basic scene running in Unity, and I felt compelled to explore VR development as a career option.
The DiVE and Breakout VR
February 2017
Pianoless
April 2017
My sophomore spring, I switched my major from mechanical engineering to computer science and became a member of Duke's XR research lab, the DiVE (Duke Immersive Virtual Environment). The director of the lab, Dr. Regis Kopper, gave me an unusual but exciting task in order to prove that I could handle working with Unity and VR hardware: to create an interactive VR version of an old Atari game. I ended up creating a 3D twist on the Steve Wozniak and Steve Jobs classic, Breakout (1976), with the paddle taken out into your hand to deflect the ball into bricks. It was simple to create and surprisingly fun to play. From there I went through the DiVE operations training, began giving tours of the DiVE, and started thinking about what I wanted to make next.
In the same semester I took a course called Experimental Interface Design, where we learned about product design and worked on small projects involving different experimental technology interfaces. One of the technologies we got to work with was the Leap Motion controller, a desktop hand tracking device that recently became compatible with VR headsets including the Vive. The class had an open-ended final project where we had to design some experimental interface, which I saw as a perfect opportunity to make an app using the Vive and Leap Motion together. I eventually came up with this idea of making an experience where you could both play and learn the piano without an actual piano. I created a recording mechanism that generates a set of instructions like Guitar Hero where you can play back the recording and have the notes fly into the piano, making it easy and fun to learn how to play that recording. The project started as just a proof of concept, but I think it could really make learning the piano significantly more accessible and I’d love to see the app grow into a multi-user platform where anyone can upload and download recordings, creating an ecosystem of easy-to-learn songs.
VR in Cape Town: Part One
July 2017
Having really enjoyed working on Breakout VR and Pianoless, I wanted to continue working on VR projects during my summer break and was lucky to find this summer abroad program iXperience that was piloting a course on VR in Cape Town, South Africa. The teacher, Gerard Slee, is a Unity expert and architect who started his own VR studio, Tenebris Lab, and his expertise on the VR industry showed me how much more I still have to learn. Through his course I learned about some extremely useful resources for creating VR experiences, notably VRTK, as well as how to use a 360 camera which we brought with us on hikes around Cape Town. iXperience’s program combines their courses with an internship, and I got to continue learning from Gerard by interning for Tenebris Lab. We worked on an initial boss-level of one of the studio’s games, tested and gave feedback on some projects that were further along, and helped with public demos at a gaming convention in Cape Town. I was extremely grateful to finally work on collaborative projects, and to see people working with this new technology all the way in South Africa made me excited to see what kind of global impact XR will make.
September 2017
Devils Cross Reality (DXR)
In my junior fall I wanted to continue working on collaborative XR projects, but unfortunately there was no place at Duke for that, so I took the initiative and started Duke’s first XR club, Devils Cross Reality (DXR). The goal of the club was to create an XR community at Duke and to teach students about what’s going on in the space by providing them resources, equipment, and an opportunity to collaborate on projects with other students. For me this was a chance to gain experience as a leader in the XR space, and I saw it as a stepping stone for eventually starting my own XR company. It wasn’t easy to get things going though, as I had to drag my giant computer and Vive setup across campus for demos, apply for funding, and get people to show up to meetings without being able to enforce commitments. There was a lot of student interest though, and our email list serve quickly grew to around 400 people. Since there are limited classes or other ways to learn about XR at Duke, I started hosting workshops on Unity development for VR through DXR and the club has become the place to go for any students new to XR.
The most personally beneficial outcome of starting DXR was that my passion for XR was finally out in the public, which made it significantly easier to network with others in the industry. I connected with a Durham based XR startup, Lucid Dream, that is run by some Duke alumni and works on custom projects for enterprise clients. They very kindly put me in touch with a woman looking to hire someone to give VR demos at her wedding, and I gladly took the opportunity. This was my first time giving public demos outside of the Duke community, and at this gig it became much clearer to me that there is a very wide range of reactions that people have when they first try VR. Depending on their age, history with video games, eagerness to try new things, and many other factors that are hard to pinpoint, everyone that tried VR had a unique experience. Some wanted to take the headset off within the first minute, others kept it on until I forced them to let someone else get a turn. Overall, the feedback was very positive, and people were glad to have the opportunity.
October 2017
A VR Wedding
Shortly after the wedding gig, I got the opportunity to join a group of students and professors presenting their project for the XPrize engineering competition at the Smithsonian Institution in Washington DC. Various universities were presenting projects at the Smithsonian as part of the ACCelerate festival, “a celebration of creative exploration and research happening at the nexus of science, engineering, arts and design”. Duke’s project was on building a device to map the floor of the ocean, and they wanted to have an ocean VR experience to go along with it to show how a 3D map of the ocean floor could be visualized. I chose to demo the Ocean Rift experience on the Vive, and unlike my previous demos I got to compare people’s reactions to the exact same VR experience. As an event open to the general public, the volume of people I gave demos was far greater than ever before, and so were the variety of reactions. By far the most surprising reaction I witnessed was an older gentleman who was so immersed in the ocean environment that he genuinely believed he could not move his legs (since there was no virtual representation of his legs), and I had to take him out when he started to look like he would fall over. From that moment on I realized how powerful VR immersion could be.
Demoing At The Smithsonian
October 2017
The History of Jazz VR Museum
November 2017
I wanted to continue taking classes with open ended projects so I could block out time to make more VR apps, so in my junior fall I took a class called Virtual Museums, which involved a final project on creating a digitally enhanced museum piece. I also happened to be taking another course on the history of jazz, and my group ended up working with the professor of that course in creating a history of jazz VR museum. The idea was that we would use VR to create a museum experience that went beyond what’s possible with a real-world museum. For every piece in the museum, when you walk up to it a song relevant to that piece would play, making use of spatial audio. We also added features like instruments that would play when you picked them up so you could pretend like you were playing them, a station for arranging different sections of jazz songs to demonstrate how composers arrange songs, and a 360-video recording of a jazz performance at Duke.
January 2018
Researching VR's Educational Potential
In my junior spring I took Duke’s only class on VR, Virtual Reality Systems Research, a graduate course taught by Dr. Kopper (who runs the DiVE lab). The primary purpose of the class was to conduct a research study related to the field of virtual reality. I was really interested in the potential applications of VR in education, and my partner and I decided to research VR’s effect on memory to try and find evidence that 3D visualizations are inherently more memorable. Prior research has shown that more sensory information correlates to better retention of information, and we felt this would provide a good basis for why VR should be used in education. At the end of our study our results showed that people enjoyed memorization with 3D objects more than with flashcards, but our quantitative results on memory retention were mixed and did not fully support our hypothesis. It appeared that 3D visualizations helped memorization in certain situations, but like any psychological study it was difficult to truly understand what factors were at the root of the results we got. Nevertheless, I learned a lot about the process of doing research in the field of VR and got a better understanding of how I might go about future research.
My time working on VR projects in Cape Town in the summer of 2017 was so amazing that I decided to go back the following summer as a teaching assistant for iXperience’s VR course. I realized that there was no better way to continue exploring VR’s potential in education than by teaching a course on VR. I also learned so much from Gerard (the teacher) the year before, and when I got to run the course with him, the value of having a direct mentor who is an expert in the field could not be understated. With 15 students working on VR projects using Unity, I had to help with the harder parts of 15 different projects which exposed me to many obstacles in VR development that I may not have ever encountered on my own. I also got so much better at explaining the more complicated concepts of VR development, which is a great skill for being able to talk to non-experts about XR. I also put together a 360 marketing video for iXperience with footage taken by me and my students. During the internship phase of the program I transitioned to be the intern manager at Tenebris Lab, Gerard’s VR studio. I led project design meetings, delegated project tasks to the interns, and helped them as needed as we collaboratively prototyped two VR apps related to banking and a bow shooter VR game.
May 2018
VR in Cape Town: Part Two
Pioneering VR Development at HackDuke
October 2018
By my senior fall there were no more classes I could take that would allow me to work on XR projects, so to fill the void I decided to make a VR app at HackDuke, a big hackathon hosted at Duke where no VR projects had ever been done before. I hosted a workshop at the beginning of the hackathon on the basics of VR development with Unity and VRTK and then gathered a group of students from the workshop and the HackDuke Slack channel that were interested in making a VR app. We came up with a project idea of using VR to showcase environmental issues related to climate change as a proof of concept of how VR can create an empathetic experience that moves people to act on environmental issues. We put together 3 simple scenes in Unity and demoed them on the Acer Windows Mixed Reality headset. Although our project was extremely scrappy, we ended up with the Novice Winner prize for the Energy & Environment track. I learned that it’s very hard to manage the scope of a VR project in a 24-hour time frame, and in future hackathons I want to go in with a more specific plan in mind.
Expanding Into XR Communities Beyond Duke
November 2018
One day I came across a LinkedIn post about an organization called the ARVR Academy, a group focused on diversity and inclusion in the XR space, and saw they were looking for mentors for their first cohort of mentor groups where women and people from underrepresented backgrounds in XR could learn about XR development. I was surprised to get accepted as a mentor given my age and experience but was very glad to have the opportunity to join what sounded like an awesome community of professionals that are passionate about the world of XR. I was matched with a group of women years ahead of me in work experience that were eager to learn about Unity and other XR tools, and it was inspiring to see how they balanced learning about this emerging technology with full-time jobs and familial responsibilities. The ARVR Academy is an incredible initiative that I’m excited to see grow because it’s imperative that the XR community is filled with people from all backgrounds, and I hope to see more diversity and inclusion in XR than the current tech community fosters.
Computer Science Education Using AR
April 2019
For a while I wanted to get my feet wet with augmented reality development, and I finally did so by designing and conducting a research study on the benefits of using AR for computer science (CS) education. I believe XR is going to play a huge part in the future of education, but in order to get to that future there needs to be much more research to show the pros and cons of integrating XR into different curricula. For most people, learning CS and programming is scary or too hard, and I think XR could be a great medium to make CS more enjoyable and easier to understand with intuitive interfaces and real-world examples in 3D. The core principle behind my project is simple: teaching object-oriented programming (OOP) concepts with 3D objects should be far less abstract than traditional 2D explanations, with the added benefit of being more fun. I developed both a desktop computer and Magic Leap One version of a tutorial on the concept of inheritance and collected data using 18 subjects to compare the two versions of the tutorial. The results of the study showed statistically significant performance improvement for subjects doing the AR tutorial and there was an overwhelming preference for learning with AR. The project was challenging but fun to work on and helped me get a better understanding of the differences between developing for VR vs. AR platforms.