0
Research Papers

Handling Perception Uncertainty in Simulation-Based Singulation Planning for Robotic Bin Picking

[+] Author and Article Information
Nithyananda B. Kumbla

Department of Mechanical Engineering,
University of Maryland,
College Park, MD 20742
e-mail: nkumbla@umd.edu

Shantanu Thakar

Department of Aerospace & Mechanical
Engineering,
University of Southern California,
Los Angeles, CA 90089
e-mail: sthakar@usc.edu

Krishnanand N. Kaipa

Department of Mechanical & Aerospace
Engineering,
Old Dominion University,
Norfolk, VA 23529
e-mail: kkaipa@odu.edu

Jeremy Marvel

Intelligent Systems Division,
National Institute of Standards and Technology,
Gaithersburg, MD 20899
e-mail: jeremy.marvel@nist.gov

Satyandra K. Gupta

Department of Aerospace & Mechanical
Engineering,
University of Southern California,
Los Angeles, CA 90089
e-mail: skgupta@usc.edu

1Corresponding author.

Contributed by the Computers and Information Division of ASME for publication in the JOURNAL OF COMPUTING AND INFORMATION SCIENCE IN ENGINEERING. Manuscript received June 12, 2017; final manuscript received December 14, 2017; published online March 15, 2018. Editor: Bahram Ravani. This work is in part a work of the U.S. Government. ASME disclaims all interest in the U.S. Government's contributions.

J. Comput. Inf. Sci. Eng 18(2), 021004 (Mar 15, 2018) (10 pages) Paper No: JCISE-17-1115; doi: 10.1115/1.4038954 History: Received June 12, 2017; Revised December 14, 2017

Robotic bin picking requires using a perception system to estimate the posture of parts in the bin. The selected singulation plan should be robust with respect to perception uncertainties. If the estimated posture is significantly different from the actual posture, then the singulation plan may fail during execution. In such cases, the singulation process will need to be repeated. We are interested in selecting singulation plans that minimize the expected task completion time. In order to estimate the expected task completion time for a proposed singulation plan, we need to estimate the probability of success and the plan execution time. Robotic bin picking needs to be done in real-time. Therefore, candidate singulation plans need to be generated and evaluated in real-time. This paper presents an approach for utilizing computationally efficient simulations for generating singulation plans. Results from physical experiments match well with the predictions obtained from simulations.

FIGURES IN THIS ARTICLE
<>
Copyright © 2018 by ASME
Your Session has timed out. Please sign back in to continue.

References

Kumbla, N. B. , Thakar, S. , Kaipa, K. N. , Marvel, J. A. , and Gupta, S. K. , 2017, “Simulation Based On-Line Evaluation of Singulation Plans to Handle Perception Uncertainty in Robotic Bin Picking,” ASME Paper No. MSEC2017-2955.
Buchholz, D. , 2016, “ Depth Map Based Pose Estimation,” Bin-Picking, Springer, New York, pp. 39–56. [CrossRef]
Kuo, H. Y. , Su, H. R. , Lai, S. H. , and Wu, C. C. , 2014, “ 3D Object Detection and Pose Estimation From Depth Image for Robotic Bin Picking,” IEEE International Conference on Automation Science and Engineering (CASE), Taipei, Taiwan, Aug. 18–22, pp. 1264–1269.
Sansoni, G. , Bellandi, P. , Leoni, F. , and Docchio, F. , 2014, “ Optoranger: A 3D Pattern Matching Method for Bin Picking Applications,” Optics and Lasers in Engineering, Vol. 54, Elsevier, Amsterdam, The Netherlands, pp. 222–231. [CrossRef]
Pretto, A. , Tonello, S. , and Menegatti, E. , 2013, “ Flexible 3D Localization of Planar Objects for Industrial Bin-Picking With Monocamera Vision System,” IEEE International Conference on Automation Science and Engineering (CASE), Madison, WI, Aug. 17–20, pp. 168–175.
Rodrigues, J. J. , Kim, J. S. , Furukawa, M. , Xavier, J. , Aguiar, P. , and Kanade, T. , 2012, “ 6D Pose Estimation of Textureless Shiny Objects Using Random Ferns for Bin-Picking,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vilamoura, Portugal, Oct. 7–12, pp. 3334–3341.
Boughorbel, F. , Zhang, Y. , Kang, S. , Chidambaram, U. , Abidi, B. , Koschan, A. , and Abidi, M. , 2003, “ Laser Ranging and Video Imaging for Bin Picking,” Assem. Autom., 23(1), pp. 53–59. [CrossRef]
Horn, B. , and Ikeuchi, K. , 1983, “Picking Parts Out of a Bin,” Artificial Intelligence Laboratory, Cambridge, MA, DTIC Document No. AIM-746. https://dspace.mit.edu/bitstream/handle/1721.1/5642/AIM-746.pdf?sequence=2
Akizuki, S. , and Hashimoto, M. , 2015, “ Stable Position and Pose Estimation of Industrial Parts Using Evaluation of Observability of 3D Vector Pairs,” J. Rob. Mechatronics, 27(2), pp. 174–181. [CrossRef]
Fuchs, S. , Haddadin, S. , Keller, M. , Parusel, S. , Kolb, A. , and Suppa, M. , 2010, “ Cooperative Bin-Picking With Time-of-Flight Camera and Impedance Controlled DLR Lightweight Robot III,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan, Oct. 18–22, pp. 4862–4867.
Harada, K. , Nagata, K. , Tsuji, T. , Yamanobe, N. , Nakamura, A. , and Kawai, Y. , 2013, “ Probabilistic Approach for Object Bin Picking Approximated by Cylinders,” IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany, May 6–10, pp. 3742–3747.
Liu, M. Y. , Tuzel, O. , Veeraraghavan, A. , Taguchi, Y. , Marks, T. K. , and Chellappa, R. , 2012, “ Fast Object Localization and Pose Estimation in Heavy Clutter for Robotic Bin Picking,” Int. J. Rob. Res., 31(8), pp. 951–973. [CrossRef]
Papazov, C. , Haddadin, S. , Parusel, S. , Krieger, K. , and Burschka, D. , 2012, “ Rigid 3D Geometry Matching for Grasping of Known Objects in Cluttered Scenes,” Int. J. Rob. Res., 31(4), pp. 538–553. [CrossRef]
Pronobis, A. , and Caputo, B. , 2007, “ Confidence-Based Cue Integration for Visual Place Recognition,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), San Diego, CA, Oct. 29–Nov. 2, pp. 2394–2401.
Bicchi, A. , and Kumar, V. , 2000, “ Robotic Grasping and Contact: A Review,” IEEE International Conference on Robotics and Automation (ICRA), San Francisco, CA, Apr. 24–28, pp. 348–353.
Bohg, J. , Morales, A. , Asfour, T. , and Kragic, D. , 2014, “ Data-Driven Grasp Synthesis—A Survey,” IEEE Trans. Rob., 30(2), pp. 289–309. [CrossRef]
Dupuis, D. C. , Léonard, S. , Baumann, M. A. , Croft, E. A. , and Little, J. J. , 2008, “ Two-Fingered Grasp Planning for Randomized Bin-Picking,” Robotics: Science and Systems 2008 Manipulation Workshop-Intelligence in Human Environments, June 28. http://www.cs.ubc.ca/~little/links/linked-papers/RSS_MAN_08_two_fingered_grasp.pdf
Ellekilde, L. P. , Jorgensen, J. A. , Kraft, D. , Kruger, N. , Piater, J. , and Petersen, H. G. , 2012, “ Applying a Learning Framework for Improving Success Rates in Industrial Bin Picking,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vilamoura, Portugal, Oct. 7–12, pp. 1637–1643.
Kendall, A. , and Cipolla, R. , 2016, “ Modelling Uncertainty in Deep Learning for Camera Relocalization,” IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, May 16–21, pp. 4762–4769.
Zheng, Y. , and Qian, W. H. , 2005, “ Coping With the Grasping Uncertainties in Force-Closure Analysis,” Int. J. Rob. Res., 24(4), pp. 311–327. [CrossRef]
Berenson, D. , Srinivasa, S. S. , and Kuffner, J. J. , 2009, “ Addressing Pose Uncertainty in Manipulation Planning Using Task Space Regions,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), St. Louis, MO, Oct. 10–15, pp. 1419–1425.
Chang, L. , Smith, J. R. , and Fox, D. , 2012, “ Interactive Singulation of Objects From a Pile,” IEEE International Conference on Robotics and Automation (ICRA), Saint Paul, MN, May 14–18, pp. 3875–3882.
Kaipa, K. N. , Kankanhalli-Nagendra, A. S. , Kumbla, N. B. , Shriyam, S. , Thevendria-Karthic, S. S. , Marvel, J. A. , and Gupta, S. K. , 2016, “ Addressing Perception Uncertainty Induced Failure Modes in Robotic Bin-Picking,” Robotics and Computer-Integrated Manufacturing, Vol. 42, Elsevier, Amsterdam, The Netherlands, pp. 17–38. [CrossRef]
Kaipa, K. N. , Kankanhalli-Nagendra, A. S. , Kumbla, N. B. , Shriyam, S. , Thevendria-Karthic, S. S. , Marvel, J. A. , and Gupta, S. K. , 2016, “ Enhancing Robotic Unstructured Bin-Picking Performance by Enabling Remote Human Interventions in Challenging Perception Scenarios,” IEEE International Conference on Automation Science and Engineering (CASE), Fort Worth, TX, Aug. 21–25, pp. 639–645.
Kaipa, K. N. , Shriyam, S. , B, N. , and Gupta, S. K. , 2016, “Resolving Occlusions Through Simple Extraction Motions in Robotic Bin Picking,” ASME Paper No. MSEC2016-8661.
Kaipa, K. N. , Shriyam, S. , Kumbla, N. B. , and Gupta, S. K. , 2015, “ Automated Plan Generation for Robotic Singulation From Mixed Bins,” IROS Workshop on Task Planning for Intelligent Robots in Service and Manufacturing, Hamburg, Germany, Oct. 2, pp. 45–50. http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=216DAF63796EA6003DAEE4F2C196FBBC?doi=10.1.1.709.9295&rep=rep1&type=pdf
Kaipa, K. N. , Kumbla, N. B. , and Gupta, S. K. , 2015, “ Characterizing Performance of Sensorless Fine Positioning Moves in the Presence of Grasping Position Uncertainty,” IROS Workshop on Task Planning for Intelligent Robots in Service and Manufacturing, Hamburg, Germany, Oct. 2, pp. 51–56. http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=043268EBCA8E7183D05E26B3974A66BD?doi=10.1.1.708.6122&rep=rep1&type=pdf
Ilies, H. T. , 2009, “ Continuous Collision and Interference Detection for 3D Geometric Models,” ASME J. Comput. Inf. Sci. Eng., 9(2), p. 021007. [CrossRef]
Akgunduz, A. , Banerjee, P. , and Mehrotra, S. , 2005, “ A Linear Programming Solution for Exact Collision Detection,” ASME J. Comput. Inf. Sci. Eng., 5(1), pp. 48–55. [CrossRef]
Kim, B. , and Rossignac, J. , 2003, “ Collision Prediction,” ASME J. Comput. Inf. Sci. Eng., 3(4), pp. 295–301. [CrossRef]

Figures

Grahic Jump Location
Fig. 1

Example of a singulation plan

Grahic Jump Location
Fig. 2

Illustration of perception uncertainty in the part location—estimated posture might differ from the actual part posture

Grahic Jump Location
Fig. 3

Possible outcomes of a singulation plan: (a) successful grasp; certain uncertainties like rotation about zg being updated during grasp. (b) Successful grasp; certain uncertainties like rotation about yg and translation along zg will propagate through the successive stages. (c) Failure due to collision with the part upon lowering the gripper due to uncertainty in translation along yg. (d) Failure due to grasp miss as the uncertainty in translation along xg is high.

Grahic Jump Location
Fig. 4

Illustration of seven grasp strategies for the chosen part (row 1) and corresponding seven drop-off strategies (row 2)

Grahic Jump Location
Fig. 5

Illustration of grasp parameters for strategy 1

Grahic Jump Location
Fig. 6

Illustration of uncertainty update during plan execution. (a) Uncertainty in translation along yg being updated during grasping action; (b) Uncertainty in rotation about xg being updated during grasp. (c) Uncertainty in rotation about zg being updated during grasp. (d) Uncertainty in rotation about yg and translation about zg being updated during drop-off on a flat surface.

Grahic Jump Location
Fig. 7

Parallel jaw grippers of Baxter and the bounding box approximation of the grippers for collision check

Grahic Jump Location
Fig. 8

Swept volume of the approximated gripper bounding box during (a) approach, (b) grasp, and (c) extract phase. The darker cuboids indicate the approximated bounding box of the gripper and the lighter cuboids indicate the swept volume of the approximated bounding box during the execution. The grasp points on the part and the gripper are marked with a thick dot.

Grahic Jump Location
Fig. 14

Error between estimated execution time and actual time for execution

Grahic Jump Location
Fig. 13

Comparison of success probability estimated by the simulator and physical trials

Grahic Jump Location
Fig. 12

Experimental data: collision check between gripper and part during approach

Grahic Jump Location
Fig. 11

Experimental setup with Baxter research robot equipped with an Asus Xtion Pro camera mounted on the left arm and a parallel jaw gripper on the right arm

Grahic Jump Location
Fig. 10

Illustration of grasp strategies that are feasible for an estimated part pose. Grasp strategy 6 is chosen for execution as it has minimum expected completion time (Table 3).

Grahic Jump Location
Fig. 9

Illustration of successful grasp under uncertainty (row 1) and failure due to grasp miss (row 2). The yellow–red dot indicates the ideal grasp point on the part. The yellow bounding box indicates the tolerance region for every grasp strategy. In row 1, the bounding box of the grasp tolerance touches the bounding box of the grippers during grasp. In row 2 the bounding box of the grasp tolerance does not touch the bounding box of the grippers during grasp.

Tables

Errata

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In