Research Papers

Freehand Gesture and Tactile Interaction for Shape Design

[+] Author and Article Information
Monica Bordegoni, Giandomenico Caruso, Umberto Cugini

Department of Mechanical Engineering,
Politecnico di Milano,
Via La Masa 1,
Milano 20156, Italy

Mario Covarrubias

Department of Mechanical Engineering,
Politecnico di Milano,
Via La Masa 1,
Milano 20156, Italy
e-mail: mario.covarrubias@polimi.it

Contributed by the Computers and Information Division of ASME for publication in the JOURNAL OF COMPUTING AND INFORMATION SCIENCE IN ENGINEERING. Manuscript received January 18, 2016; final manuscript received March 16, 2016; published online November 7, 2016. Assoc. Editor: Charlie C.L. Wang.

J. Comput. Inf. Sci. Eng 16(4), 041002 (Nov 07, 2016) (7 pages) Paper No: JCISE-16-1026; doi: 10.1115/1.4033230 History: Received January 18, 2016; Revised March 16, 2016

This paper presents a novel system that allows product designers to design, experience, and modify new shapes of objects, starting from existing ones. The system allows designers to acquire and reconstruct the 3D model of a real object and to visualize and physically interact with this model. In addition, the system allows designer to modify the shape through physical manipulation of the 3D model and to eventually print it using a 3D printing technology. The system is developed by integrating state-of-the-art technologies in the sectors of reverse engineering, virtual reality, and haptic technology. The 3D model of an object is reconstructed by scanning its shape by means of a 3D scanning device. Then, the 3D model is imported into the virtual reality environment, which is used to render the 3D model of the object through an immersive head mounted display (HMD). The user can physically interact with the 3D model by using the desktop haptic strip for shape design (DHSSD), a 6 degrees of freedom servo-actuated developable metallic strip, which reproduces cross-sectional curves of 3D virtual objects. The DHSSD device is controlled by means of hand gestures recognized by a leap motion sensor.

Copyright © 2016 by ASME
Your Session has timed out. Please sign back in to continue.


Racasan, R. , Popescu, D. , and Dragomir, M. , 2011, “ Integrating the Concept of Reverse Engineering in Medical Applications,” 2010 IEEE International Conference on Automation Quality and Testing Robotics (AQTR), Cluj-Napoca, Romania, May 28–30.
Bradley, C. , and Currie, B. , 2005, “ Advances in the Field of Reverse Engineering,” Comput. Aided Des. Appl., 2(5), pp. 697–706.
Yatani, K. , Partridge, K. , Bern, M. , and Newman, M. , 2008, “ Escape: A Target Selection Technique Using Visually-Cued Gestures,” Conference on Human Factors in Computing Systems, pp. 285–294.
Bragdon, A. , and Ko, H.-S. , 2011. “ Gesture Select: Acquiring Remote Targets on Large Displays Without Pointing,” Conference on Human Factors in Computing Systems, pp. 187–196.
Vogel, D. , and Balakrishnan, R. , 2005, “ Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays,” Annual ACM Symposium on User Interface Software and Technology, UIST, pp. 33–42.
Vogel, D. , and Balakrishnan, R. , 2004, “ Interactive Public Ambient Displays: Transitioning From Implicit to Explicit, Public to Personal, Interaction With Multiple Users,” Annual ACM Symposium on User Interface Software and Technology, UIST, pp. 137–146.
Guimbretière, F. , and Nguyen, C. , 2012, “ Bimanual Marking Menu for Near Surface Interactions,” Conference on Human Factors in Computing Systems, pp. 825–828.
Ren, G. , and O'Neill, E. , 2013, “ 3D Selection With Freehand Gesture,” Comput. Graphics, 37(3), pp. 101–120. [CrossRef]
Iwata, H. , Yano, H. , Nakaizumi, F. , and Kawamura, R. , 2001, “ Project FEELEX: Adding Haptic Surface to Graphics,” 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH’01, ACM, New York, pp. 469–476.
Leithinger, D. , Follmer, S. , Olwal, A. , Luescher, S. , Hogge, A. , Lee, J. , and Ishii, H. , 2013, “ Sublimate: State-Changing Virtual and Physical Rendering to Augment Interaction With Shape Displays,” SIGCHI Conference on Human Factors in Computing Systems, CHI’13, pp. 1441–1450.
Rosen, D. , Nguyen, A. , and Wang, H. , 2003, “ On the Geometry of Low Degree-of-Freedom Digital Clay Human Computer Interface Devices,” ASME Paper No. DETC2003/CIE-48295.
Bosscher, P. , and Ebert-Uphoff, I. , 2003, “ A Novel Mechanism for Implementing Multiple Collocated Spherical Joints,” IEEE International Conference on Robotics and Automation, ICRA’03, Taipei, Taiwan, Sept. 14–19, Vol. 1, pp. 336–341.
Klare, S. , Forssilow, D. , and Peer, A. , 2013, “ Formable Object—A New Haptic Interface for Shape Rendering,” World Haptics Conference, Daejeon, South Korea, Apr. 14–15, pp. 61–66.
Bordegoni, M. , Ferrise, F. , Covarrubias, M. , and Antolini, M. , 2010, “ Haptic and Sound Interface for Shape Rendering,” Presence: Teleoperators Virtual Environ., 19(4), pp. 341–363. [CrossRef]
Bordegoni, M. , Ferrise, F. , Covarrubias, M. , and Antolini, M. , 2011, “ Geodesic Spline Interface for Haptic Curve Rendering,” IEEE Trans. Haptics, 4(2), pp. 111–121. [CrossRef] [PubMed]
Covarrubias, M. , Bordegoni, M. , and Cugini, U. , 2013, “ Continuous Surface Rendering, Passing From Cad to Physical Representation,” Int. J. Adv. Rob. Syst., 10, pp. 1–13. [CrossRef]
“Structure Sensor,” Last accessed Mar. 2, 2016, https://www.structure.io/
“Skanect,” Last accessed Mar. 2, 2016, https://www.scanect.occipital.com/
Levien, R. L. , 2009. “ From Spiral to Spline: Optimal Techniques in Interactive Curve Design,” Ph.D. thesis, University of California, Berkeley, CA.
Cameron, C. , DiValentin, L. , Manaktala, R. , McElhaney, A. , Nostrand, C. , Quinlan, O. , Sharpe, L. , Slagle, A. , Wood, C. , Zheng, Y. Y. , and Gerling, G. , 2011, “ Hand Tracking and Visualization in a Virtual Reality Simulation,” Systems and Information Engineering Design Symposium (SIEDS), Charlottesville, VA, Apr. 29, pp. 127–132.
Wigdor, D. , and Wixon, D. , 2011, Brave NUI World: Designing Natural User Interfaces for Touch and Gesture, 1st ed., Morgan Kaufmann Publishers, San Francisco, CA.
Oculus, “Oculus Rift,” Last accessed Mar. 2, 2016, https://www.oculus.com/en-us/
Leap Motion, “Leap Motion,” Last accessed Mar. 2, 2016, https://www.leapmotion.com/
Unity, “Unity3D,” Last accessed Mar. 2, 2016, https://www.unity3d.com/
Arduino, “Arduino,” Last accessed Mar. 2, 2016, https://www.arduino.cc/


Grahic Jump Location
Fig. 1

Product design process: (a) traditional approach and (b) proposed approach

Grahic Jump Location
Fig. 2

Structure sensor [17] and steps of the process used for scanning the geometry of an object

Grahic Jump Location
Fig. 3

MEC spline approach for representing the 2D cross section curve: (c) frontal view

Grahic Jump Location
Fig. 4

Interaction modalities using gestures tracked and recognized by the LeapMotion sensor. In (a) and (b), the user sees the virtual environment displayed on a desktop monitor and in (c) and (d) the user wears the Oculus Rift HMD.

Grahic Jump Location
Fig. 5

Example of global modification of the vacuum cleaner: (a) original CAD model, (b) mesh-editing grid for global and local modification, and (c) results for global modification modality

Grahic Jump Location
Fig. 6

System components and integration

Grahic Jump Location
Fig. 7

Examples of target curves selected on the digital model and physically rendered by the haptic strip: (a) the three target curves, (b) target curve “A,” (c) target curve “B,” and (d) target curve “C”

Grahic Jump Location
Fig. 8

Moving the virtual strip through the pinch gesture: (a) the pinch is recognized by LeapMotion, (b) the user moves the strip, and (c) continuous movement up to the desired position

Grahic Jump Location
Fig. 9

Moving the 3D model through the hand and/or the index finger: (a) the virtual strip “floating” in space and (b) the 3D model movement is driven by the hand and/or the index finger through leap motion



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In