CS CAPSTONE TECHNOLOGY REVIEW NOVEMBER 10, 2018
NASA FDL LUNAR POSITIONING
PREPARED FOR
INTEL SHASHI JAIN
PREPARED BY GROUP 27-2
INTEL AI/VR TEAM MEGHANA KOLASANI
Abstract
The following technology review document describes different technologies used in topics relevant to the Virtual Reality Demo Application portion of the project, such as in-application data capture mechanisms, application-database interfaces and APIs, and online data storage.
CONTENTS
1 Introduction
2 Online Data Storage
2.1 Central XML-Based Database
2.2 Poly………….. .
2.3 InContext Cloud Storage . . .
3 Application Programming Interface
3.1 OpenVR ……….. .
3.2 WebVR………… .
3.3 OculusSDK ……… .
. . .
. . .
2
2 ………………………………… 2 ………………………………… 2 ………………………………… 3
3 ………………………………… 3 ………………………………… 3 ………………………………… 4
4 ………………………………… 4 ………………………………… 4 ………………………………… 4
5 5
4 Application Data Capture Mechanisms
4.1 VisualCloud . . .
4.2 Oculus Analytics
4.3 VadR …….
5 Conclusion
6 References
…….. …….. ……..
1
1 INTRODUCTION
To address the knowledge transfer part of the problem, a virtual reality demo application will be created to showcase how hard the space navigation problem actually is, by putting a human in the place of the rover. There is already a prototype of this application, but the further work to be done is to deliver an immersive web experience to engage citizen scientists and bring the experience to the masses, so they can understand how the AI model can significantly overcome the challenges of space navigation.
2 ONLINE DATA STORAGE
When it comes to to virtual reality applications there is an excessive amount of data beyond just HD or 4K video images, like 360 degree views of every video viewpoint that need to be stored to maintain the users perspective from every angle. This data must be extracted in a way that will be useful for the application, resulting in detailed environment models that are difficult to produce and maintain. This far exceeds the capacity that most databases have, requiring 10-20 times the storage capacity a standard HD file would need. This data needs to be streamed in real time, so there needs to be a data center with fast access.
2.1 Central XML-Based Database
A central XML-based database uses common XML tools to transform and import data from different sources, as virtual reality needs a wider variety of data sources to be integrated to create content and build detailed applications [1]. This means that multiple users and applications need to be able to access and share one environment model for any updates or modifications. It can be difficult to keep the data model consistent with different applications, which is why the XML n-tiered application architecture can be utilized to more easily maintain the data and apply any application specific pre-processing operations. Data is stored in the database component in a model format until transformations occur to customize the data for different output requirements required by the application. By querying for relevant information in the central store during run time, the data can be configured from its original state to a state that can be directly used by the client. This way, the application and storage back-end are treated as a distinct entity and remain untouched, since the application will use efficient data structures independent of the actual storage format [1].
2.2 Poly
Poly is Googles recently develop platform used to download and browse 3D objects and scenes. Users can access a vast collection of 3D assets and dynamically import them between AR/VR, mobile or desktop applications [2]. The Poly Toolkit for Unity and Unreal Engines which we will also be utilizing for the demo app for this project. The objects can be modified or customized to suit the users specific needs and then get republished. Designing 3D models is time consuming and difficult, and Poly provides a searchable database with thousands of objects and scenes for free. Any users 3D creations can be uploaded and available to others, with a Creative Commons license [2]. Although Poly is integrated for Googles creative VR tools like Tilt Brush and Blocks, the Poly API objects can be used for whatever the user wants. There are massive amounts of content already available so it makes sense to utilize them in a readily available database.
2
2.3 InContext Cloud Storage
Cloud-based VR software like InContext Cloud Storage opens up the possibility of users working together, saving and viewing changes on multiple devices within a virtual environment [3]. Cloud based storage is portable, eliminating the need for expensive on-site servers. Traditional methods can be too limited and time taking and are difficult to personalize. With InContext, users can create, change, and save multiple image and 3D video concepts with stakeholders or team members, with secured access and tests for effectiveness [3]. There is also no storage limit since the data lives in the cloud. InContext is also research friendly, as it provides easy access to get feedback about the simulations. The Total Cost of Ownership for InContext is budget-friendly, since aspects of storage, infrastructure, hosting, security and maintenance are managed by the platform.
3 APPLICATION PROGRAMMING INTERFACE
Virtual reality has a high potential to be used as an interface for an integrated project database. With the use of Virtual Reality Modeling Language, a web-based standard, information can be remotely investigated, and 3D views and worlds can be explored in real time. This way the user can interact with a virtual reality 3D column instead of a traditional column in a database environment [4]. Having an integrated database will make it easier to use virtual reality to classify information.
3.1 OpenVR
Created by Valve, OpenVR is an open source program interface that allows communication with a virtual reality system [5]. This interface can support major headsets like Oculus Rift, HTC Vive, Windows Mixed Reality and most others, with nearly no modifications. Likewise, Unity, Unreal, Godot and other software components can be used with OpenVR backend calls. The OpenVR API is divided into modules that handle different parts of a virtual reality application. The main interface that allows one to interact with connected devices and controllers and collect information is the IVRSystem. The module that allows 3D content to be correctly rendered in the application display within a VR context is the IVRCompositor. There are also two modules, IVRChaperone and IVROverlay, that enable one to access the virtual bounds system information as well as 2D content for the overlay like menus and buttons.
3.2 WebVR
WebVR is a versatile option, as anyone can experience virtual reality in their browser, regardless of device [6]. All they would need is any headset and a compatible browser, without needing to download any additional client, application or hardware. As an open standard not limited to a single platform, virtual reality apps on the web are much more easily accessible. WebVR can still provide an immersive experience, even when run on the browser. The central interface in the WebVR API is the VRDisplay. This interface provides information that will identify the display and its controllers and capabilities. It will also retrieve and submit frame data for every content frame being presented at a consistent rate. Typically the app will get a reference to the display, present, run a rendering loop at a specific refresh rate, draw the displayed scene and display the rendered view to the user [6].
3
3.3 Oculus SDK
The Oculus SDK is a native SDK for the entire Oculus platform, especially for high end headsets like Oculus Rift and Samsung Gear VR. Oculus has a very flexible SDKs and integration for audio, avatar and platform, since popular engines like Unity and Unreal provide out-of-the-box support for this hardware [7]. The SDK contains various tools, libraries and resources for native virtual reality development. A third-party engine integration VrAPI, native application framework for building high-performance virtual reality applications from scratch, additional libraries that support GUI and other functionality, sample project applications served as a reference model are all provided within the SDK [7]. The other SDKs can be used for other features, like the platform SDK for security, community, revenue and engagement.
4 APPLICATION DATA CAPTURE MECHANISMS
When building custom applications, the developers often benefit from user feedback and data about how the application was used and what the results were, in order to improve or make changes to the application. This is especially true in the NASA Lunar Positioning Project, as NASA would like to process and analyze this data.
4.1 VisualCloud
VisualCloud provides the breakthrough opportunity of capturing, storing, transmitting and rendering virtual reality content at scale, so that people can get an immersive experience [8] . Visual Cloud runs on a public cloud with a full system stack that can ingest, store, process and stream large amounts of visual and audio data. This database management system can persist information in a multidimensional array, taking into account space and time and bit and frame rate. The VisualCloud system is composed of the VisualCloud Server (VCS) and the VisualCloud Client (VCC), which is run on a users headset [8]. The VCS captures video, partitions and encodes it, and sends the segments to the client. The VCC retrieves the segments and rebuilds them into a displayable stream to render on the users headset. This improves the performance, compared to processing and storing 3D content the same way 2D content is.
4.2 Oculus Analytics
Oculus provides a relevant tool for measuring application usage, whether its session behavior or user behavior, that identify triggers developers can look into to holistically optimize the application with platform-specific metrics [9]. Oculus delivers rich insights about several different core app metrics like the number of installs or users, revenue, frame rate, battery burn rate, install failures, controller usage, and session length. There are also data filters for applications like by device or carrier or GPU or CPU breakdowns, and updates on real-time processing to identify unexpected crashes or CPU spikes all of which can be easily exported into a csv to better view the data.
4.3 VadR
VadR is another platform for virtual reality analytics to increase engagement and user experience [10]. Since virtual reality is a new media that is more immersive and engaging, the data generated is more complicated, which is why VadR helps provide better analysis for data like interactions, movements and vision better understand user interests and behavior. VadR is intuitive and powerful with relevant metrics and advanced data visualization to quickly gain insight. VadR is easy to customize and users can choose what parameters to modify, whether its user gaze, position and movement, device performance, or heat maps. It also supports multiple platforms like Unity or Unreal and headsets
4
like Vive, Rift and GearVR. The SDK can be easily integrated and the data can be accessed and shared from anywhere, making collaboration a lot easier.
5 CONCLUSION
Several different technologies will be used for different functions for the many aspects of this virtual reality demo application. This document provides a high-level overview of potential options that may help achieve the goal of delivering an immersive experience that satisfies the purpose of the NASA Lunar Positioning Project. InContext cloud storage would be a good option for online data storage, as it is research friendly and portable. Final technical decisions about the APIs will be made after gaining access to existing code and further discussion with the client and the rest of the team, but any of the three mentioned could work well. As for application data capture mechanisms, VadR seems like a good fit since it can be easily integrated with the platforms being used like Unity and Unreal.
6 REFERENCES
[1] G. Reitmayr and D. Schmalstieg, Data Management Strategies for Mobile Augmented Reality, Vienna University of Technology, [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.10.9658rep=rep1type=pdf. [Accessed: 09-Nov-2018].
[2] R. Whitwam, Google launches Poly, a database of 3D objects for app developers, Android Police, 07-Nov-2017. [On- line]. Available: https://www.androidpolice.com/2017/11/01/google-launches-poly-database-3d-objects-app-developers/. [Accessed: 09-Nov-2018].
[3] G. Joseph, Virtual, Reality, and Living in the Cloud, InContext Solutions, 20-Apr-2017. [Online]. Available: https://www.incontextso virtual-store-simulation-experts/virtual-reality-and-living-in-the-cloud. [Accessed 09-Nov-2018].
[4] G. Aouad, T. Child, F. Marir, and P. Brandon, Developing a virtual reality interface for an integrated project database environment, Proceedings. 1997 IEEE Conference on Information Visualization (Cat. No.97TB100165), 2002.
[5] M. Nassi, Introduction to OpenVR 101 Series: What is OpenVR and how to get started with its APIs, The Ghost Howls, 15-Mar-2018. [Online]. Available: https://skarredghost.com/2018/03/15/introduction-to-openvr-101-series-what-is-openvr- and-how-to-get-started-with-its-apis/. [Accessed: 09-Nov-2018].
[6] WebVR API, MDN Web Docs. [Online]. Available: https://developer.mozilla.org/en-US/docs/Web/API/WebVRAPI. [Accessed: 09-Nov-2018].
[7] Mobile SDK Getting Started Guide, Oculus. [Online]. Available: https://developer.oculus.com/documentation/mobilesdk/latest/ intro/. [Accessed: 09-Nov-2018].
[8] B. Haynes, A. Minyaylov, M. Balazinska, L. Ceze, and A. Cheung, VisualCloud Demonstration, Proceedings of the
2017 ACM International Conference on Management of Data – SIGMOD 17, 2017.
[9] Developer Analytics: An Introduction, Oculus. [Online]. Available: https://developer.oculus.com/blog/developer- analytics-an-introduction/. [Accessed: 09-Nov-2018].
[10] Virtual Reality Analytics, Analytics for Virtual Reality Content by VadR. [Online]. Available: https://www.vadr.io/. [Accessed: 09-Nov-2018].
5