Research Papers

Conceptual Three-Dimensional Modeling Using Intuitive Gesture-Based Midair Three-Dimensional Sketching Technique

[+] Author and Article Information
Jinmiao Huang

ABB Inc.,
Bloomfield, CT 06002
e-mail: jinmiao.huang@us.abb.com

Rahul Rai

Manufacturing and Design (MAD) Lab,
Department of Mechanical and Aerospace
University at Buffalo (UB)-SUNY,
Buffalo, NY 14260
e-mail: rahulrai@buffalo.edu

1Corresponding author.

Contributed by the Computers and Information Division of ASME for publication in the JOURNAL OF COMPUTING AND INFORMATION SCIENCE IN ENGINEERING. Manuscript received February 16, 2018; final manuscript received July 20, 2018; published online September 5, 2018. Assoc. Editor: Francesco Ferrise.

J. Comput. Inf. Sci. Eng 18(4), 041014 (Sep 05, 2018) (13 pages) Paper No: JCISE-18-1042; doi: 10.1115/1.4040982 History: Received February 16, 2018; Revised July 20, 2018

We introduce an intuitive gesture-based interaction technique for creating and manipulating simple three-dimensional (3D) shapes. Specifically, the developed interface utilizes low-cost depth camera to capture user's hand gesture as the input, maps different gestures to system commands and generates 3D models from midair 3D sketches (as opposed to traditional two-dimensional (2D) sketches). Our primary contribution is in the development of an intuitive gesture-based interface that enables novice users to rapidly construct conceptual 3D models. Our development extends current works by proposing both design and technical solutions to the challenges of the gestural modeling interface for conceptual 3D shapes. The preliminary user study results suggest that the developed framework is intuitive to use and able to create a variety of 3D conceptual models.

Copyright © 2018 by ASME
Your Session has timed out. Please sign back in to continue.


Robertson, B. F. , and Radcliffe, D. F. , 2009, “ Impact of Cad Tools on Creative Problem Solving in Engineering Design,” Comput.-Aided Des., 41(3), pp. 136–146. [CrossRef]
Sharma, A. , Madhvanath, S. , Shekhawat, A. , and Billinghurst, M. , 2011, “ Mozart: A Multimodal Interface for Conceptual 3D Modeling,” 13th International Conference on Multimodal Interfaces (ICMI), Alicante, Spain, Nov. 14–18, pp. 307–310. https://www.researchgate.net/publication/221052364_MozArt_A_Multimodal_Interface_for_Conceptual_3D_Modeling
Rubine, D. , 1991, “ Specifying Gestures by Example,” 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '91), Providence, RI, Apr. 27–30, pp. 329–337.
Takahashi, T. , and Kishino, F. , 1992, “ A Hand Gesture Recognition Method and Its Application,” Syst. Comput. Jpn., 23(3), pp. 38–48. [CrossRef]
Radhakrishnan, S. , Lin, Y. , Zeid, I. , and Kamarthi, S. , 2013, “ Finger-Based Multitouch Interface for Performing 3D CAD Operations,” Int. J. Human-Comput. Stud., 71(3), pp. 261–275. [CrossRef]
Seth, A. , Su, H.-J. , and Vance, J. M. , 2006, “ Sharp: A System for Haptic Assembly and Realistic Prototyping,” ASME Paper No. DETC2006-99476.
Edit, V. , 2008, “ Using Hand Motions in Conceptual Shape Design,” Product Engineering, Springer, Berlin, pp. 367–382.
Segen, J. , and Kumar, S. , 1998, “ Gesture Vr: Vision-Based 3D Hand Interace for Spatial Interaction,” Sixth ACM International Conference on Multimedia, Bristol, UK, Sept. 13–16, pp. 455–464.
Wang, R. , Paris, S. , and Popović, J. , 2011, “ 6D Hands: Markerless Hand-Tracking for Computer Aided Design,” 24th Annual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, Oct. 16–19, pp. 549–558. https://homes.cs.washington.edu/~jovan/papers/wang-2011-6dh.pdf
Bai, H. , Gao, L. , El-Sana, J. , and Billinghurst, M. , “ Free-Hand Interaction for Handheld Augmented Reality Using an RGB-Depth Camera,” SIGGRAPH Asia 2013 Symposium on Mobile Graphics and Interactive Applications, Hong Kong, Nov. 19–22, Paper No. 22.
Stannus, S. , Fu, W.-T. , and Lucieer, A. , 2014, “ Natural 7DOF Input for 3D Navigation,” 26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design, Sydney, Australia, Dec. 2–5, pp. 216–219.
Song, J. , Cho, S. , Baek, S.-Y. , Lee, K. , and Bang, H. , 2014, “ Gafinc: Gaze and Finger Control Interface for 3D Model Manipulation in Cad Application,” Comput.-Aided Des., 46, pp. 239–245. [CrossRef]
Vinayak, Murugappan, S. , Piya, C. , and Ramani, K. , 2013, “ Handy-Potter: Rapid Exploration of Rotationally Symmetric Shapes Through Natural Hand Motions,” ASME J. Comput. Inf. Sci. Eng., 13(2), p. 021008. [CrossRef]
Han, Y. C. , and Han, B-J. , 2014, “ Virtual Pottery: A Virtual 3D Audiovisual Interface Using Natural Hand Motions,” Multimedia Tools Appl., 73(2), pp. 917–933. [CrossRef]
Vinayak, K. , and Ramani , 2015, “ A Gesture-Free Geometric Approach for Mid-Air Expression of Design Intent in 3D Virtual Pottery,” Comput.-Aided Des., 69, pp. 11–24. [CrossRef]
Colombo, G. , Facoetti, G. , Rizzi, C. , and Vitali, A. , 2016, “ Mixed Reality to Design Lower Limb Prosthesis,” Comput.-Aided Des. Appl., 13(6), pp. 799–807. [CrossRef]
Fuge, M. , Yumer, M. E. , Orbay, G. , and Kara, L. B. , 2012, “ Conceptual Design and Modification of Freeform Surfaces Using Dual Shape Representations in Augmented Reality Environments,” Comput.-Aided Des., 44(10), pp. 1020–1032. [CrossRef]
Jinmiao, H. , Prakhar, J. , and Rahul, R. , 2018, “ Gesture-Based System for Next Generation Natural and Intuitive Interfaces,” Ai Edam, 15, p. 1.
Olsen, L. , Samavati, F. F. , Sousa, M. C. , and Jorge, J. A. , 2009, “ Sketch-Based Modeling: A Survey,” Comput. Graph., 33(1), pp. 85–103. [CrossRef]
Takeo, I. , Satoshi, M. , and Hidehiko, T. , 2007, “ Teddy: A Sketching Interface for 3D Freeform Design,” ACM SIGGRAPH 2007 Courses, ACM, New York, p. 21
Nealen, A. , Igarashi, T. , Sorkine, O. , and Alexa, M. , 2007, “ Fibermesh: Designing Freeform Surfaces With 3D Curves,” ACM Trans. Graph. (TOG), 26(3), p. 41. [CrossRef]
Karpenko, O. , Hughes, J. F. , and Raskar, R. , 2002, “ Free-Form Sketching With Variational Implicit Surfaces,” Comput. Graph. Forum, 21(3), pp. 585–594. [CrossRef]
Tai, C.-L. , Zhang, H. , and Fong, J. C.-K. , 2004, “ Prototype Modeling From Sketched Silhouettes Based on Convolution Surfaces,” Comput. Graph. Forum, 23(1), pp. 71–83. [CrossRef]
Alexe, A. , Gaildrat, V. , and Barthe, L. , 2004, “ Interactive Modelling From Sketches Using Spherical Implicit Functions,” Third International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa, Stellenbosch, South Africa, Nov. 3–5, pp. 25–34.
Schmidt, R. , Wyvill, B. , Sousa, M. C. , and Jorge, J. A. , 2006, “ Shapeshop: Sketch-Based Solid Modeling With Blobtrees,” ACM SIGGRAPH 2006 Courses, ACM, New York, p. 14
Gharib, I. , and Qin, S. , 2013, “ Integration of Sketch-Based Conceptual Design and Commercial Cad Systems for Manufacturing,” Int. J. Adv. Manuf. Technol., 68(9–12), pp. 2669–2681. [CrossRef]
Zeleznik, R. C. , Herndon, K. P. , and Hughes, J. F. , 2007, “ Sketch: An Interface for Sketching 3D Scenes,” ACM SIGGRAPH 2007 Courses, ACM, New York, p. 19.
Jatupoj, P. , 2005, “ Sketchboard: The Simple 3D Modeling From Architectural Sketch Recognition,” Proceedings of the 10th International Conference on Computer Aided Architectural Design Research in Asia, CAADRIA, New Delhi, India, pp. 3–22.
Qin, S. F. , Wright, D. K. , and Jordanov, I. N. , 2000, “ From On-Line Sketching to 2D and 3D Geometry: A System Based on Fuzzy Knowledge,” Comput.-Aided Des., 32(14), pp. 851–866. [CrossRef]
Chaudhuri, S. , Kalogerakis, E. , Guibas, L. , and Koltun, V. , 2011, “ Probabilistic Reasoning for Assembly-Based 3D Modeling,” ACM Trans. Graph. (TOG), 30(4), p. 35. [CrossRef]
Kalogerakis, E. , Chaudhuri, S. , Koller, D. , and Koltun, V. , 2012, “ A Probabilistic Model for Component-Based Shape Synthesis,” ACM Trans. Graph., 31(4), p. 55.
Chaudhuri, S. , Kalogerakis, E. , Giguere, S. , and Funkhouser, T. , 2013, “ Attribit: Content Creation With Semantic Attributes,” 26th Annual ACM Symposium on User Interface Software and Technology, St. Andrews, UK, Oct. 8–11, pp. 193–202.
Jaiswal, P. , Huang, J. , and Rai, R. , 2016, “ Assembly-Based Conceptual 3D Modeling With Unlabeled Components Using Probabilistic Factor Graph,” Comput.-Aided Des., 74, pp. 45–54. [CrossRef]
Sung, M. , Su, H. , Kim, V. G. , Chaudhuri, S. , and Guibas, L. , 2017, “ Complement Me: Weakly-Supervised Component Suggestions for 3D Modeling,” ACM Trans. Graph. (TOG), 36(6), p. 226. [CrossRef]
Krupka, E. , Karmon, K. , Bloom, N. , Freedman, D. , Gurvich, I. , Hurvitz, A. , Leichter, I. , Smolin, Y. , Tzairi, Y. , Vinnikov, A. , and Bar-Hillel, A. , 2017, “ Toward Realistic Hands Gesture Interface: Keeping It Simple for Developers and Machines,” CHI Conference on Human Factors in Computing Systems, Denver, CO, May 6–11, pp. 1887–1898.
Vitali, A. , and Rizzi, C. , 2017, “ A Virtual Environment to Emulate Tailor's Work,” Comput.-Aided Des. Appl., 14(5), pp. 671–679. [CrossRef]
Microsoft, 2018, “ Gestures Mixed Reality,” Microsoft, Redmond, WA, accessed Aug. 8, 2018, https://docs.microsoft.com/en-us/windows/mixed-reality/gestures
The CGAL Project, 2018, CGAL User and Reference Manual, 4.12 ed., CGAL Editorial Board.
Chew, L. P. , 1989, “ Constrained Delaunay Triangulations,” Algorithmica, 4(1–4), pp. 97–108. [CrossRef]
Mapari, R. B. , and Kharat, G. , 2016, “ American Static Signs Recognition Using Leap Motion Sensor,” Second International Conference on Information and Communication Technology for Competitive Strategies, Udaipur, India, Mar. 4–5, p. 67.
Marin, G. , Dominio, F. , and Zanuttigh, P. , 2014, “ Hand Gesture Recognition With Leap Motion and Kinect Devices,” IEEE International Conference on Image Processing (ICIP), Paris, France, Oct. 27–30, pp. 1565–1569.
Zeiler, M. D. , and Fergus, R. , 2014, “ Visualizing and Understanding Convolutional Networks,” European Conference on Computer Vision, Zurich, Switzerland, Sept. 6–12, pp. 818–833. https://cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf
Simonyan, K. , and Zisserman, A. , 2014, “ Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv Preprint: 1409:1556. https://arxiv.org/abs/1409.1556
Russakovsky, O. , Deng, J. , Su, H. , Krause, J. , Satheesh, S. , Ma, S. , Huang, Z. , Karpathy, A. , Khosla, A. , Bernstein, M. , Berg, A. C. , and Fei-Fei, L. , 2015, “ ImageNet Large Scale Visual Recognition Challenge,” Int. J. Comput. Vision (IJCV), 115(3), pp. 211–252. [CrossRef]
Chollet, F. , 2015, “ Keras,” Keras, accessed Aug. 7, 2018, https://keras.io
Dalal, N. , and Triggs, B. , 2005, “ Histograms of Oriented Gradients for Human Detection,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, CA, June 20–25, pp. 886–893.
Hammond, T. , and Davis, R. , 2005, “ Ladder, a Sketching Language for User Interface Developers,” Comput. Graph., 29(4), pp. 518–532. [CrossRef]
Kara, L. B. , and Stahovich, T. F. , 2005, “ An Image-Based, Trainable Symbol Recognizer for Hand-Drawn Sketches,” Comput. Graph., 29(4), pp. 501–517. [CrossRef]
Rabiner, L. R. , 1990, “ Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition,” Readings in Speech Recognition, Morgan Kaufmann Publishers Inc., San Francisco, CA, pp. 267–296
Viterbi, A. , 1967, “ Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm,” IEEE Transactions Inf. Theory, 13(2), pp. 260–269. [CrossRef]
Tsang, S. , Balakrishnan, R. , Singh, K. , and Ranjan, A. , 2004, “ A Suggestive Interface for Image Guided 3D Sketching,” SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria, Apr. 24–29, pp. 591–598.
Wigdor, D. , and Wixon, D. , 2011, Brave Nui World: Designing Natural User Interfaces for Touch and Gesture, Elsevier, Burlington, MA.
Nielsen, M. , Störring, M. , Moeslund, T. B. , and Granum, E. , 2003, “ A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI,” International Gesture Workshop, Genoa, Italy, Apr. 15–17.
Thakur, A. , and Rai, R. , 2015, “ User Study of Hand Gestures for Gesture Based 3D CAD Modeling,” ASME Paper No. DETC2015-46086.
Mansutti, A. , Rodriguez, M. C. , Bordegoni, M. , and Cugini, U. , 2017, Tactile Display for Virtual 3D Shape Rendering, Springer, Cham, Switzerland.


Grahic Jump Location
Fig. 1

Gesture-based modeling pipeline

Grahic Jump Location
Fig. 2

Hand gesture defined in our interface

Grahic Jump Location
Fig. 3

Gesture-based mirroring process

Grahic Jump Location
Fig. 4

Gesture combination for assembly process

Grahic Jump Location
Fig. 5

Gestures for camera navigation: tumbling (left) and zooming (right)

Grahic Jump Location
Fig. 6

Single-strokes stored in our system

Grahic Jump Location
Fig. 7

Primitives database

Grahic Jump Location
Fig. 8

Pc is the intensity value for palm center: (a) shows the segmented hand with cutoff threshold Pc-50, a portion of fingers would also become absent, (b) shows the segmented hand with cutoff threshold Pc-100, unnecessary background information would often become included, (c) Canny edge detection process, and (d) normalized hand

Grahic Jump Location
Fig. 9

Leap motion captured same hand gesture image with different rotation angle

Grahic Jump Location
Fig. 10

Different view angles for a rectangle sketch in the air

Grahic Jump Location
Fig. 11

Performance on with/without HMMs for different tests

Grahic Jump Location
Fig. 12

Stroke before and after preprocessing

Grahic Jump Location
Fig. 13

Gesture-based modeling concept overview, with sample models created from the developed interface

Grahic Jump Location
Fig. 14

Sample models used for testing the modeling time in different interfaces

Grahic Jump Location
Fig. 15

Accepted models recreated from modeling experts based on the shown samples in Fig. 14

Grahic Jump Location
Fig. 16

This figure shows the average modeling time with standard deviations for ten users recreate the models shown in Fig. 14

Grahic Jump Location
Fig. 17

Models created in the developed interface by novice user



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In