Scalable Image-based Modeling Through Machine Learning
Reactive Reality’s technology transforms conventional 2D photos into photo-realistic 3D objects. Unlike 3D scans, these 3D objects include semantic information which enables advanced mobile AR use cases, including home furnishing and fashion try-on. Capturing the photos is quick and does not require any special skills or hardware. All processing steps are fully automated.
Our methods are based on convolutional neural networks (CNNs) and other machine learning methods to identify the type of image and obtain an accurate geometric model. Moreover, shape templates are used to derive semantics for each object. All algorithms are integrated into our powerful backend tool that can be used to create an unlimited amount of AR content and achieve an unprecedented level of scalability.
- Stefan Hauswiesner, Philipp Grasmug: Method and system for generating garment model data. Filed at EPO December 2014 Link
- Thomas Richter-Trummer, Jinwoo Park, Denis Kalkofen, and Dieter Schmalstieg. Instant Mixed Reality Lighting from Casual Scanning. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR'16), Merida, Mexico, 2016 PDF
- Christoph Bauernhofer, Dense Reconstruction On Mobile Devices, Master's Thesis, Graz University of Technology, 2016 PDF
- Michael Donoser and Dieter Schmalstieg: Discrete-Continuous Gradient Orientation Estimation for Faster Image Segmentation. In Proc. IEEE Computer Vision and Pattern Recognition 2014, Columbus, OH, USA, 2014 PDF
- Bernhard Kainz, Stefan Hauswiesner, Gerhard Reitmayr, Markus Steinberger, Raphael Grasset, Lukas Gruber, Eduardo Veas, Denis Kalkofen, Hartmut Seichter, Dieter Schmalstieg: OmniKinect: Real-Time Dense Volumetric Data Acquisition and Applications. Symposium on Virtual Reality Software and Technology (VRST), 2012 PDF
- Andreas Hartl, Lukas Gruber, Clemens Arth, Stefan Hauswiesner, Dieter Schmalstieg: Rapid Reconstruction of Small Objects on Mobile Phones. Proceedings of the Embedded Computer Vision Workshop (held in conjunction with CVPR), 2011 PDF
Photo-Realistic AR Through Image Based Rendering
Reactive Reality’s image-based modeling technology creates extremely detailed 3D models from simple 2D photos. The technology combines photogrammetry with machine learning to derive geometry as well as semantics from a set of input images. The process works for all materials, including fine structures, soft and fuzzy materials and can even capture objects that are otherwise not suitable for photogrammetry (objects with plain, untextured surfaces). As a result, the technology can be used for all kinds of products in e-commerce.
- Stefan Hauswiesner, Philipp Grasmug: Method and system for producing output images and method for generating image-related databases. Filed at EPO May 2015 Link
- Stefan Hauswiesner, Efficient Image-based Augmentations, PhD Thesis, Graz University of Technology, 2013 PDF
- Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Virtual Try-On Through Image-based Rendering. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2013 PDF
- Stefan Hauswiesner, Philipp Grasmug, Denis Kalkofen, Dieter Schmalstieg: Frame Cache Management for Multi-frame Rate Systems. Proceedings of the 8th International Symposium on Visual Computing (ISVC), 2012 PDF
- Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Image-Based Clothes Transfer. Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, Switzerland, 2011 PDF
Accurate Fit Simulation Through Mesh Adaption
The adaptation and augmentation of clothes on a user’s image is achieved through advanced mesh fitting algorithms. Our proprietary algorithms use differential coordinates to represent garments and other non-rigid objects, allowing us to model fit, stretch and stiffness as a linear system of equations. The mesh tessellation and correspondence search algorithms are specifically optimized for mobile CPUs. As a result, our solver converges within a fraction of a second on any recent smartphone or tablet model. Our proprietary rendering methods utilize mobile graphical processors to achieve a high level of performance and keep battery usage low.
- Thomas Krinninger, One-Shot 3D Body-Measurement, Master's Thesis, Graz University of Technology, 2016 PDF
- Michael Donoser, Martin Hirzer, Dieter Schmalstieg: Multiple Model Fitting by Evolutionary Dynamics. International Conference on Pattern Recognition, ICPR 2014, Stockholm, Sweden, 2014 PDF
- Markus Steinberger, Michael Kenzel, Bernhard Kainz, Joerg Mueller, Peter Wonka, Dieter Schmalstieg: Parallel Generation of Architecture on the GPU. Computer Graphics Forum, 33(2), 2014 PDF
- Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Temporal Coherence in Image-based Visual Hull Rendering. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2013 PDF
- Matthias Straka, Stefan Hauswiesner, Matthias Rüther, Horst Bischof: Rapid Skin: Estimating the 3D Human Pose and Shape in Real-Time. Proceedings of 3DimPVT, Zürich, Switzerland, 2012 PDF
- Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Free Viewpoint Virtual Try-On With Commodity Depth Cameras. Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry (VRCAI), ACM SIGGRAPH, Hong Kong, 2011 PDF
Real-time Body Pose and Shape Detection Through Deep Learning
Detecting the user’s full body pose and shape in an image is a very challenging technical problem that Reactive Reality solves through convolutional neural networks (CNNs) and probabilistic pose space modeling. We cut down on the number of memory operations by using SIMD and GPU parallel instruction sets, resulting in a real-time body detection and tracking system that runs on conventional smartphones.
These methods can be used to recognize a user's body position and shape and even create a full 3D avatar resembling the user. Realistic avatars are required by a wide range of applications such as telepresence systems, virtual try-on applications or games.
- Matthias Straka, Stefan Hauswiesner, Matthias Rüther, Horst Bischof: Simultaneous Shape and Pose Adaption of Articulated Models using Linear Optimization. Proceedings of the 12th European Conference on Computer Vision (ECCV), 2012 PDF
- Markus Steinberger, Bernhard Kainz, Bernhard Kerbl, Stefan Hauswiesner, Michael Kenzel, Dieter Schmalstieg: Softshell: Dynamic Scheduling on GPUs. ACM Transactions on Graphics, Proceedings of SIGGRAPH Asia 2012 PDF
- Stefan Hauswiesner, Rostislav Khlebnikov, Markus Steinberger, Matthias Straka, Gerhard Reitmayr: Multi-GPU Image-based Visual Hull Rendering. Proceedings of the Eurographics Symposium on Parallel Graphics and Visualization, Sardinia, Italy, 2012 PDF
- Matthias Straka, Stefan Hauswiesner, Matthias Rüther, Horst Bischof: Skeletal Graph Based Human Pose Estimation in Real-Time, Proceedings of the British Machine Vision Conference (BMVC), 2011 PDF
- Matthias Straka, Stefan Hauswiesner, Matthias Rüther, Horst Bischof: A Free-Viewpoint Virtual Mirror with Marker-Less User Interaction. Proceedings of the 17th Scandinavian Conference on Image Analysis (SCIA), 2011 PDF
- Stefan Hauswiesner, Matthias Straka, Gerhard Reitmayr: Coherent Image-Based Rendering of Real-World Objects. Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, San Francisco, CA, 2011 PDF