Multi-Device Application: Unity, HoloLens, MRTK and Photon Working Together
Using Photon and MRTK for Professor/Student Collaboration in Classroom Instruction
The University of Connecticut used a custom Mixed Reality application on the HoloLens 2, with Unity, MRTK and Photon to teach Physics concepts with virtual collaboration. Students were able to participate in the instruction using their phones, desktop computers or HoloLens 2.
OVERVIEW
The University of Connecticut (UCONN) was interested in evaluating the use of 3D visualizations to teach basic Physics concepts in the classroom. SphereGen created a custom application on the HoloLens 2 using Unity, MRTK (Mixed Reality Tool Kit) and Photon to allow professors and students to three dimensionally view conceptual electromagnetic fields whose object values correlate to variables in equations. When object values such as mass, speed and magnetic polarization are manipulated, the 3D visualizations change accordingly so students can actively participate in working equations.
Challenges
There were three main challenges in creating this application. The first challenge dealt with the devices themselves. 3D visualizations are best when using the HoloLens 2 headset, however the cost of providing 20-30 headsets per classroom was prohibitive during a prototype evaluation. There had to be a way to use more commonly available devices to allow participation.
The second challenge was related to collaboration. For the students to best understand the concepts, they had to be able to manipulate object values in addition to the Professor adjusting key object values. All users in the application had to be able to see object values adjust in real-time and maintain the spatial perspective of the object values in relation to their position within the viewing environment.
The third challenge involved users joining the application at different times. When someone joins the application, the user must be able to see an accurate representation of the objects as they exist in the application at that point and time, regardless of when they join. This means that spatial points of all changes must be recorded and presented to the newest joiners.
Solution
SphereGen created a Mixed Reality application for the HoloLens 2 using MRTK and Photon to resolve existing challenges. The application was modified with MRTK to be available on smart phones and desktops.
The use of Photon allowed the control and recordation of all objects and their relative spatial assignments. . This allowed the users to witness object changes in real-time and the programming to record and synchronize all necessary data points to present a consistent visual representation for users that join in at different times.
RESULTS
Collaboration
1 Application, Multiple Devices
Student Engagement
THE DETAILS
Cross Platform Deployment
In building the application, the resulting code had to be usable on multiple devices, based on UCONN’s request. Using Unity and the tools present in the MRTK, only one code base was needed to build to multiple platforms. SphereGen was sure that the application would function as expected on our target HoloLens 2 and iOS/Android platforms as the MRTK handles translating common actions into the relevant SDK for each platform. It also allows for platform specific modifications to be made if required.
Shared Experience
The Photon SDK allowed SphereGen to connect all the users to the experience, regardless of device or location. All the objects and settings shared between the users are kept up to date in all sessions and sent to any new users that join and request the current state of all objects.
Users that are co-located are further immersed with all objects sharing a consistent reference point through Azure Spatial Anchors. Not only do these users see the real-time virtual actions of others, but the virtual objects’ physical locations are also in sync.
CONCLUSION
In testing virtual instruction in the classroom, UCONN could teach 5 Physics concepts alongside the traditional teaching method. With the Unity application, students could interact with the laws and equations of Physics to visually grasp the effects of changing forces on electrical and magnetic fields.
Students took turns using all three device types – HoloLens 2, smart phone, and desktop -at different times of instruction. Engagement with the subject material was highest when using the HoloLens 2, as the field of view was truly three dimensional and immersive. However, students could participate and follow along on the other devices without incurring extra cost as desktops were readily available, and almost all students own smart phones.
SphereGen built a collaborative, engaging application for multiple device platforms from one working codebase.
TECHNOLOGY USED
• Unity
• MRTK
• Azure Spatial Anchors
• Photon SDK
• HoloLens 2
• iOS/Android OS
View and download this case study here
WE ASSIST
SMALL + MEDIUM
SIZED BUSINESSES
As a privately-held small business, we understand the goals, needs, and challenges of organizations your size. SphereGen leverages 10+ years of experience to find the ‘best technology fit’ for clients whether you build, buy or customize software and services.
