Highly Interactive Visualizations in Augmented Reality
Leonel Merino
Visualizations enable us to interact with the visual representation of data and to identify complex patterns. Now imagine that you don‘t need to sit in front of a computer display and use the keyboard and mouse to interact with a visualization, but that you can walk through human-sized and interactive visualizations while selecting elements with hand gestures. These are Situated Visualizations. They allow a new form of interactive analytical thinking. However, existing toolkits for creating such Situated Visualizations are not yet flexible or agile enough to produce usable visualizations. As a first step to achieve Agile Situated Visualizations, we have developed AVAR. AVAR is a toolkit that generates particularly lively, expressive and interactive visualizations.
Visualizations In-Situ
Through Situated Visualizations, complex data sets are made visually perceptible in a new way.
Augmented reality is used to embed these visualizations directly in our physical environment.
Their aim is to close the gap between people and their data by supporting data understanding and decision making in context. Moreover, Situated Visualizations stimulate cognitive aspects such as engagement, embodiment, and recall.
Existing toolkits are not yet flexible or agile enough, which makes the resulting Situated Visualization less applicable. There is a lack of toolkits that guide the authoring of Situated Visualizations and, at the same time, offer ready-to-use building blocks to speed up development.
Our Approach
With our toolkit AVAR, one can quickly create Agile Situated Visualizations that combine real objects with visual representations of data. We hope to promote logical thinking in this way.
The advantage of Agile Situated Visualization is, that one can easily explore multiple facets of a data set without having to leave the immersive environment in augmented reality for changing the parameters.
Users can add previously filtered data and see directly the view changing. AVAR has to overcome three main challenges: short feedback loops when modifying visualizations, integral support for defining and interacting with visualizations, and be expressive enough to define multiple visualization designs, but simple enough to be used in immersive augmented reality.
AVAR: An Agile Situated Visualization Toolkit
AVAR is a prototypical implemention for Agile Situated Visualization. In it, users can interact with the visualization by simply walking through it while employing a Microsoft HoloLens® and a Bluetooth keyboard as input/output devices.
AVAR integrates a fully operational programming language, Pharo Smalltalk, into an immersive environment. Why Pharo? Because it is interpreted and highly reflective. This means no waiting for a script to compile, but working in a live environment, and an eased integration to external environments. In addition it has powerful 2D/3D visualization engines (Roassal2 and Woden), which implement multiple visualization techniques such as parallel coordinates, treemaps, node-link diagrams, and space-time cube matrices.
AVAR also integrates a Unity 3D application that communicates with Pharo as a backend, handles user interaction, and renders a graphical user interface in augmented reality.
We examined the usage of the AVAR toolkit for authoring Situated Visualizations in a study with seven expert Pharo programmers.
We could confirm two things by working with Agile Situated Visualizations. First, that users need to address visualization processes in a continuous loop and not sequentially. And second, that users depend on having multiple features available to express various visualization designs in an iterative fashion.
Summary
Situated Visualizations in immersive augmented reality offer the possibilty to explore visualizations of real world data embedded in their physical context. With Agile Situated Visualizations one can build multiple Situated Visualizations in an agile fashion. This is a new way to examine various facets of a data set.
Our toolkit AVAR is the first step towards applicable Agile Situated Visualizations. With AVAR one can deal with a visualization as a whole without leaving the immersive environment. Our design choices impacted the users‘ behavior, namely spending much more time with the visualization. Especially the live feedback boosted the highly interactively engagement. The employment of multiple available features when programming a Situated Visualization is a real advantage. In the future, we plan to improve our design, investigate other means for interaction, and conduct further evaluations.
About the author
Leonel Merino is the Manager of the Graduate School. He conducts research on the intersection of data visualization, human-computer interaction, software engineering, and virtual and augmented reality.
This issue is a contribution to the DFG2020 campaign Because Research Matters.
Links
- Leonel Merino Profile Page
- Video with demo of Agile Situated Visualizations
- Related Publication
- All INSIGHT Stories
Download
Press Contact