Reactive Reality’s technology drives a wide range of Augmented Reality (AR) applications by enabling quick content generation methods and user immersion. Our methods capture users and objects using image-based modeling and reproduce their appearance using image-based rendering. This approach is inherently scalable (every user has a camera) and realistic (photos and videos don’t look artifical, virtual objects often do). All our methods run on conventional mobile devices without any special sensors and was designed for instant zero latency operation.

User Immersion Through Image Based Rendering

Reactive Reality’s technology enables users to immerse themselves in Augmented Reality scenes with apparel, objects and landmarks. For example, users can see themselves wearing a summer dress while standing in front of the pyramids. With a single swipe, the same user is taken to a fashion show where she is posing as one of the models on the catwalk wearing the designer's clothes. Tilting the user's phone reveals new views of the scene and create a strong sense of immersion and depth.

Being image-based, the technology's fields of application are nearly limitless. Proprietary image-based modeling algorithms turn conventional images and photos into AR objects that can be viewed, tried on, animated and interacted with. Users just take a photo or capture a video to immerse in other worlds and can even create new worlds on their smartphones.

Related publications:

Stefan Hauswiesner, Philipp Grasmug: Method and system for producing output images and method for generating image-related databases. Filed at EPO May 2015 Link

Stefan Hauswiesner, Efficient Image-based Augmentations, PhD Thesis, Graz University of Technology, 2013 PDF

Stefan Hauswiesner, Philipp Grasmug, Denis Kalkofen, Dieter Schmalstieg: Frame Cache Management for Multi-frame Rate Systems. Proceedings of the 8th International Symposium on Visual Computing (ISVC), 2012 PDF

Efficient Mesh Fitting

Our mesh fitting algorithm uses differential coordinates to represent garments and other non-rigid objects. As a consequence, we can model fit, stretch and stiffness as a linear system of equations. The mesh tesselation and correspondence search algorithms are specifically optimized for mobile CPUs. As a result, our solver converges within fractions of a second on any recent smartphone or tablet. Our proprietary rendering methods utilize mobile graphical processors to achieve a high level of performance and keep the battery usage low.

Related publications:

Michael Donoser, Martin Hirzer, Dieter Schmalstieg: Multiple Model Fitting by Evolutionary Dynamics. International Conference on Pattern Recognition, ICPR 2014, Stockholm, Sweden, 2014 PDF

Markus Steinberger, Michael Kenzel, Bernhard Kainz, Joerg Mueller, Peter Wonka, Dieter Schmalstieg: 
Parallel Generation of Architecture on the GPUComputer Graphics Forum, 33(2), 2014 PDF

Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Temporal Coherence in Image-based Visual Hull Rendering. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2013 PDF

Markus Steinberger, Bernhard Kainz, Bernhard Kerbl, Stefan Hauswiesner, Michael Kenzel, Dieter Schmalstieg: Softshell: Dynamic Scheduling on GPUs. ACM Transactions on Graphics, Proceedings of SIGGRAPH Asia 2012 PDF

Stefan Hauswiesner, Rostislav Khlebnikov, Markus Steinberger, Matthias Straka, Gerhard Reitmayr: Multi-GPU Image-based Visual Hull Rendering. Proceedings of the Eurographics Symposium on Parallel Graphics and Visualization, Sardinia, Italy, 2012 PDF

Image Based Modeling and Machine Learning

Reactive Reality transforms conventional images into augmented reality objects. Content can be product images from webshops or photos the user takes. Our methods use machine learning to identify the type of an image and shape fitting to obtain a geometric model. The key to this approach lies in how to obtain shape priors for fitting. We developed a backend application that enables us to add new object categories within minutes - which in turn enables users to add as many objects into categories as they like. This enables unprecedented scalability of the content generation.

Related publications:

Thomas Richter-Trummer, Jinwoo Park, Denis Kalkofen, and Dieter Schmalstieg. Instant Mixed Reality Lighting from Casual Scanning. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR'16), Merida, Mexico, 2016 PDF

Stefan Hauswiesner, Philipp Grasmug: Method and system for generating garment model data. Filed at EPO December 2014, Link

Michael Donoser and Dieter Schmalstieg: Discrete-Continuous Gradient Orientation Estimation for Faster Image SegmentationIn Proc. IEEE Computer Vision and Pattern Recognition 2014, Columbus, OH, USA, 2014 PDF

Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Virtual Try-On Through Image-based Rendering. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2013 PDF

Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Image-Based Clothes Transfer. Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, Switzerland, 2011 PDF

Body Pose and Shape Detection on Mobile Devices

In our mission to roll out augmented reality on mobile platforms, we developed a specialized set of algorithms for articulated, deformable model detection. Reactive Reality cuts down on the number of memory operations by using SIMD and GPU parallel instruction sets. We use a pipeline of different methods, including SVM-supported analysis of image features and probabilistic pose space modeling. These methods can be used to recognize a user's body position and shape, but can also be used to locate vehicles, machines, furniture and other articulated or rigid objects in images.

Related publications:

Thomas Krinninger, One-Shot 3D Body-Measurement, Master's Thesis, Graz University of Technology, 2016 PDF

Matthias Straka, Stefan Hauswiesner, Matthias Rüther, Horst Bischof: Rapid Skin: Estimating the 3D Human Pose and Shape in Real-Time. Proceedings of 3DimPVT, Zürich, Switzerland, 2012 PDF

Matthias Straka, Stefan Hauswiesner, Matthias Rüther, Horst Bischof: Simultaneous Shape and Pose Adaption of Articulated Models using Linear Optimization. Proceedings of the 12th European Conference on Computer Vision (ECCV), 2012 PDF

Matthias Straka, Stefan Hauswiesner, Matthias Rüther, Horst Bischof: Skeletal Graph Based Human Pose Estimation in Real-Time, Proceedings of the British Machine Vision Conference (BMVC), 2011 PDF

Andreas Hartl, Lukas Gruber, Clemens Arth, Stefan Hauswiesner, Dieter Schmalstieg: Rapid Reconstruction of Small Objects on Mobile Phones. Proceedings of the Embedded Computer Vision Workshop (held in conjunction with CVPR), 2011 PDF

Matthias Straka, Stefan Hauswiesner, Matthias Rüther, Horst Bischof: A Free-Viewpoint Virtual Mirror with Marker-Less User Interaction. Proceedings of the 17th Scandinavian Conference on Image Analysis (SCIA), 2011 PDF

3D Reconstruction for Mixed Reality

Virtual avatars are common to a large range of applications such as telepresence systems, virtual try-on applications or games. Avatars need to look like the users they represent to create a sense of immersion. Reactive Reality's technology enables users to create realistic avatars that look like themselves on their home PC, gaming console or smartphone. Our technology is based on a highly efficient solver for non-linear equations that automatically adapts a parametric avatar model to given color and depth data.

Related publications:

Christoph Bauernhofer, Dense Reconstruction On Mobile Devices, Master's Thesis, Graz University of Technology, 2016 PDF

Bernhard Kainz, Stefan Hauswiesner, Gerhard Reitmayr, Markus Steinberger, Raphael Grasset, Lukas Gruber, Eduardo Veas, Denis Kalkofen, Hartmut Seichter, Dieter Schmalstieg: OmniKinect: Real-Time Dense Volumetric Data Acquisition and Applications. Symposium on Virtual Reality Software and Technology (VRST), 2012 PDF

Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Free Viewpoint Virtual Try-On With Commodity Depth Cameras. Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry (VRCAI), ACM SIGGRAPH, Hong Kong, 2011 PDF

Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Coherent Image-Based Rendering of Real-World Objects. Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, San Francisco, CA, 2011 PDF