A 3D tele-immersion system based on live captured mesh geometry

  • Authors:

  • Rufael Mekuria;Michele Sanna;Stefano Asioli;Ebroul Izquierdo;Dick C. A. Bulterman;Pablo Cesar

  • Affiliations:

  • Centrum voor Wiskunde en Informatica, XG Amsterdam;Queen Mary, University of London, London UK;Queen Mary, University of London, London UK;Queen Mary, University of London, London UK;Centrum voor Wiskunde en Informatica, XG Amsterdam;Centrum voor Wiskunde en Informatica, XG Amsterdam

  • Venue:
  • Proceedings of the 4th ACM Multimedia Systems Conference
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

3D Tele-immersion enables participants in remote locations to share, in real-time, an activity. It offers users natural interactivity and immersive experiences, but it challenges current networking solutions. Work in the past has mainly focused on the efficient delivery of image-based 3D videos and on the realistic rendering and reconstruction of geometry-based 3D objects. The contribution of this paper is a complete media pipeline that allows for geometry-based 3D tele-immersion. Unlike previous approaches, that stream videos or video plus depth estimate, our streaming module can transmit the live-reconstructed 3D representations (triangle meshes). Based on a set of comparative experiments, this paper details the architecture and describes a novel component that can efficiently stream geometry in real-time. This component includes both a novel fast local compression algorithm and a rateless packet protection scheme geared towards the requirements imposed by real-time transmission of live-capture mesh geometry. Tests on a large dataset show an encoding and decoding speed-up of over 10 times at similar compression and quality rates, when compared to the high-end MPEG-4 SC3DMC mesh encoder. The implemented rateless code ensures complete packet loss protection of the triangle mesh object and avoids delay introduced by retransmissions. This approach is compared to a streaming mechanism over TCP and outperforms it at packet loss rates over 2% and/or latencies over 9 ms in terms of end-to-end transmission delay. As reported in this paper, the component has been successfully integrated into a larger tele-immersive environment that includes beyond state of the art 3D reconstruction and rendering modules. This resulted in a prototype that can capture, compress transmit and render triangle mesh geometry in real-time over the internet.