A Novel Distributed Multimodality Interaction Framework for Augmented Reality Objects Using Internet of Things

Augmented reality is the art of rendering intelligent computer-generated content in digital visual perception of a user. A camera input device continually captures real world scene, which is analyzed by a context extraction program. The rendering program then renders artificial objects into the scene based on this extra content. User can interact with such computer-generated content. Resolving the interaction in real time when the interaction is generated from multiple modalities and combining them with context is a huge challenge in Augmented reality and other mixed reality programs. Several past techniques have been proposed towards this direction, but they are more specific to perception, limited context and interaction modalities. In this work we propose a novel framework for interacting with AR objects over IoT using multimodal interaction and show that such a framework can be extended to wide range of other perceptual frameworks like gaming and virtual reality. We show that out method is more robust and fast in comparison to traditional IP command-based techniques. The paper presents method from developing to deploying a 3D human like character in artificial scenes and demonstrate user interaction with this with different modalities. A complete framework for Unity 3D has been developed with IoT enabled interaction which can be extended to other games and artificial context aware renderings.