Research Papers

Object Tracking With a Range Camera for Augmented Reality Assembly Assistance

[+] Author and Article Information
Rafael Radkowski

Assistant Professor
Department of Mechanical Engineering,
Iowa State University,
Ames, IA 50011
e-mail: rafael@iastate.edu

Contributed by the Computers and Information Division of ASME for publication in the JOURNAL OF COMPUTING AND INFORMATION SCIENCE IN ENGINEERING. Manuscript received January 2, 2015; final manuscript received November 5, 2015; published online January 13, 2016. Editor: Bahram Ravani.

J. Comput. Inf. Sci. Eng 16(1), 011004 (Jan 13, 2016) (8 pages) Paper No: JCISE-15-1002; doi: 10.1115/1.4031981 History: Received January 02, 2015; Revised November 05, 2015

This paper introduces a 3D object tracking method for an augmented reality (AR) assembly assistance application. The tracking method relies on point clouds; it uses 3D feature descriptors and point cloud matching with the iterative closest points (ICP) algorithm. The feature descriptors identify an object in a point cloud; ICP align a reference object with this point cloud. The challenge is to achieve high fidelity while maintaining camera frame rates. The point cloud and reference object sampling density are one of the key factors to meet this challenge. In this research, three-point sampling methods and two-point cloud search algorithms were compared to assess their fidelity when tracking typical products of mechanical engineering. The results indicate that a uniform sampling maintains the best fidelity at camera frame rates.

Copyright © 2016 by ASME
Your Session has timed out. Please sign back in to continue.


Azuma, R. , 1997, “ A Survey of Augmented Reality,” Presence Teleoperators Virtual Environ., 6(2–3), pp. 355–385. [CrossRef]
Radkowski, R. , Herrema, J. , and Oliver, J. , 2015, “ Augmented Reality Based Manual Assembly Support With Visual Features for Different Degree of Difficulty,” J. Hum. Comput. Interact., 31(5), pp. 337–349. [CrossRef]
Henderson, S. , and Feiner, S. , 2011, “ Augmented Reality in the Psychomotor Phase of a Procedural Task,” 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 191–200.
Hou, L. , and Wang, X. , 2013, “ A Study on the Benefits of Augmented Reality in Retaining Working Memory in Assembly Tasks: A Focus on Differences in Gender,” Autom. Constr., 32(2013), pp. 38–45. [CrossRef]
Westerfield, G. , Mitrovic, A. , and Billinghurst, M. , 2013, “ Intelligent Augmented Reality Training for Assembly Tasks,” 16th International Conference on Artificial Intelligence in Education (AIED 2013), H. Chad, Land, K. Yacef, J. Mostow, and P. Pavlik, eds., pp. 542–551.
Peniche, A. , Treffetz, H. , Diaz, C. , and Paramo, G. , 2012, “ Combing Virtual and Augmented Reality to Improve the Mechanical Assembly Training Process in Manufacturing,” American Conference on Applied Mathematics, pp. 292–297.
Yuan, M. L. , Ong, S. K. , and Nee, A. Y. C. , 2008, “ Augmented Reality for Assembly Guidance Using a Virtual Interactive Tool,” Int. J. Prod. Res., 46(7), pp. 1745–1767. [CrossRef]
Wiedenmaier, S. , Oehme, O. , Schmidt, L. , and Luczak, H. , 2003, “ Augmented Reality (AR) for Assembly Processes Design and Experimental Evaluation,” Int. J. Hum. Comput. Interact., 16(3), pp. 497–514. [CrossRef]
Kato, H. , and Billinghurst, M. , 1999, “ Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System,” 2nd International Workshop on Augmented Reality (IWAR 99), pp. 85–94.
Neumann, U. , and You, S. , 1998, “ Natural Feature Tracking for Augmented-Reality,” IEEE Transactions on Multimedia, pp. 53–64.
Salas-Moreno, R. F. , Newcombe, R. A. , Strasdat, H. , Kelly, P. H. J. , and Davison, A. J. , 2013, “ Slam++: Simultaneous Localisation and Mapping at the Level of Objects,” 2013 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1352–1359.
Radkowski, R. , and Oliver, J. , 2013, “ Natural Feature Tracking Augmented Reality for On-Site Assembly Assistance Systems,” Human Computer Interaction International Conference, pp. 281–290.
Besl, P. , and McKay, N. , 1992, “ A Method for Registration of 3D-Shapes,” PAMI., 14(2), pp. 239–256. [CrossRef]
Raczynski, A. , and Gussmann, P. , 2004, “ Services and Training Through Augmented Reality,” 1st European Conference on Visual Media Production (CVMP’04), pp. 263–271.
Zenati, N. , Zerhouni, N. , and Achour, K. , 2004, “ Assistance to Maintenance in Industry Process Using an Augmented Reality System,” 2004 IEEE international Conference on Industrial Technology (KIT), Vol. 2, pp. 848–852.
Reiners, D. , Stricker, D. , Klinker, G. , and Müller, S. , 1998, “ Augmented Reality for Construction Tasks: Doorlock Assembly,” 1st International Workshop on Augmented Reality (IWAR 98), pp. 31–46.
Neumann, U. , and Majoros, A. , 1998, “ Cognitive, Performance, and Systems Issues for Augmented Reality Applications in Manufacturing and Maintenance,” Virtual Reality Annual International Symposium, pp. 4–11.
Robertson, C. , MacIntyre, B. , and Walker, B. , 2008, “ An Evaluation of Graphical Context When the Graphics Are Outside of the Task Area,” 7th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2008), IEEE Computer Society, Washington, DC, pp. 73–76.
Khuong, B. M. , Kiyokawa, K. , Mashita, T. , Miller, A. , Takemura, H. , and La Viola, J. J. , 2014, “ The Affectiveness of an AR-Based Context-Aware Assembly Support System in Object Assembly,” IEEE Virtual Reality (VR), pp. 57–62.
Radkowski, R. , and Oliver, J. , 2014, “ Monocular Depth Cues for Augmented Reality Applications to Enhance Spatial Perception Tasks,” Tools and Methods of Competitive Engineering (TMCE 2014), pp. 749–768.
Wang, Y. , Shen, Y. , Liu, D. , Wei, S. , and Zhu, C. , 2010, “ Key Technique of Assembly System in an Augmented Reality Environment,” 2010 Second International Conference on Computer Modeling and Simulation, pp. 133–137.
Song, J. , Jian, Q. , Sun, H. , and Gao, X. , 2009, “ Study of the Perception Mechanisms and Method of Virtual and Real Objects in Augmented Reality Assembly Environments,” 4th IEEE Conference on Industrial Electronics and Applications, pp. 1452–1456.
Baird, K. M. , and Barfield, W. , 1999, “ Evaluating the Effectiveness of Augmented Reality Displays for a Manual Assembly Task,” Virtual Reality, 4(4), pp. 250–259. [CrossRef]
Korn, O. , Schmidt, A. , and Hörz, T. , 2013, “ Augmented Manufacturing: A Study With Impaired Persons on Assistive Systems Using In-Situ Projection,” 6th International Conference on Pervasive Technologies Related to Assistive Environment (PETRA 2013), pp. 21:1–21:8.
Tang, A. , Owen, C. , Biocca, F. , and Mou, W. , 2003, “ Comparative Effectiveness of Augmented Reality in Object Assembly,” Conference on Human Factors in Computing Systems (CHI’03), pp. 73–80.
Webel, S. , Bockholt, U. , Engelke, T. , Gavish, N. , Olbrich, M. , and Preusche, C. , 2011, “ An Augmented Reality Training Platform for Assembly and Maintenance Skills,” J. Rob. Auton. Syst., 61(4), pp. 398–403. [CrossRef]
Nilsson, S. , and Johansson, B. , 2007, “ Fun and Usable: Augmented Reality Instructions in a Hospital Setting,” 19th Australasian Conference on Computer-Human Interaction (OZCHI 07), pp. 123–130.
Miller, A. , White, B. , Charbonneau, E. , Kanzler, Z. , and LaViola, J. J., Jr. , 2012, “ Interactive 3D Model Acquisition and Tracking of Building Block Structures,” IEEE Trans. Visualization Comput. Graphics, 18(4), pp. 651–659. [CrossRef]
Gupta, A. , Fox, D. , Curless, B. , and Cohen, M. , 2012, “ Duplotrack: A Real-Time System for Authoring and Guiding Duplo Block Assembly,” 25th Annual ACM Symposium on User Interface Software and Technology, pp. 389–402.
Newcombe, R. A. , Izadi, S. , Hilliges, O. , Molyneaux, D. , Kim, D. , Davison, A. J. , Kohli, P. , Shotton, J. , Hodges, S. , and Fitzgibbon, A. , 2011, “ Kinectfusion: Real-Time Dense Surface Mapping and Tracking,” International Symposium on Mixed and Augmented Reality, pp. 127–136.
Garrett, T. , Debernardis, S. , Radkowski, R. , Chang, C. K. , Fiorentino, M. , Uva, A. E. , and Oliver, J. , 2014, “ Rigid-Object Tracking for Real-Time Augmented Reality,” ASME 2014 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, pp. 1–10.
Rusinkiewicz, S. , and Levoy, M. , 2001, “ Efficient Variants of the ICP Algorithm,” IEEE International Workshop on 3D Digital Imaging and Modeling (3DIM), pp. 145–152.
Drost, B. , Ulrich, M. , Navab, N. , and Ilic., S. , 2010, “ Model Globally, Match Locally: Efficient and Robust 3D Object Recognition,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 998–1005.
Turk, G. , and Levoy, M. , 1994, “ Zippered Polytgon Meshes From Range Images,” SIGGRAPH '94, Computer Graphics Proceedings, pp. 311–318.
Masuda, T. , Sakaue, K. , and Yokoya, N. , 1996, “ Registration and Integration of Multiple Range Images for 3D-Model Construction,” CVPR, pp. 879–883.
Greenspan, M. , and Yurick, M. , 2003, “ Approximate k-d Tree Search for Efficient ICP,” Fourth International Conference on 3-D Digital Imaging and Modeling (3DIM 2003), pp. 442–448.
Godin, G. , Rioux, M. , and Baribeau, R. , 1994, “ Three-Dimensional Registration Using Range and Intensity Information,” Proc. SPIE 2350, Videometrics III, pp. 279–290.
Zhang, Z. , 1994, “ Iterative Point Matching for Registration of Free-Form Curves and Surfaces,” Int. J. Comput. Vision, 13(2), pp. 119–152. [CrossRef]
Arun, K. , Huang, T. S. , and Blostein, S. D. , 1987, “ Least-Squares Fitting of Two 3-D Point Sets,” IEEE Trans. Pattern Anal. Mach. Intell., PAMI-9(5), pp. 698–700. [CrossRef]


Grahic Jump Location
Fig. 1

Overview of the tracking process

Grahic Jump Location
Fig. 2

(a) A typical part that needs to be tracked in the addressed research area. (b) The associated 3D model that is used as reference object for tracking.

Grahic Jump Location
Fig. 3

(a) The input depth image Rt and (b) the resulting vertex map Vt

Grahic Jump Location
Fig. 4

Camera pose estimation: an initial position for reference model Pi and the result from ICP ([Rt]) are used to calculate the camera pose

Grahic Jump Location
Fig. 5

Iterations to minimize the distance between a reference object Pi and an environment point cloud X

Grahic Jump Location
Fig. 6

The AR application with (a) the reference object overlaid over the physical part and (b) with a virtual object superimposed on the physical part

Grahic Jump Location
Fig. 7

The test setup to record the datasets

Grahic Jump Location
Fig. 8

Screenshot of the simulation tool icptracksim

Grahic Jump Location
Fig. 9

Number of iterations from different camera positions

Grahic Jump Location
Fig. 11

Comparison of RMS over number of iterations

Grahic Jump Location
Fig. 12

Sample RMS error of the second experiment



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In