Research Papers

An Unsupervised Machine Learning Approach to Assessing Designer Performance During Physical Prototyping

[+] Author and Article Information
Matthew L. Dering

Computer Science and Engineering,
Pennsylvania State University,
University Park, PA 16801
e-mail: dering@cse.psu.edu

Conrad S. Tucker

Industrial Engineering,
Pennsylvania State University,
University Park, PA 16801
e-mail: ctucker4@psu.edu

Soundar Kumara

Industrial Engineering,
Pennsylvania State University,
University Park, PA 16801
e-mail: u1o@engr.psu.edu

Contributed by the Computers and Information Division of ASME for publication in the JOURNAL OF COMPUTING AND INFORMATION SCIENCE IN ENGINEERING. Manuscript received September 1, 2016; final manuscript received July 12, 2017; published online November 13, 2017. Assoc. Editor: Monica Bordegoni.

J. Comput. Inf. Sci. Eng 18(1), 011002 (Nov 13, 2017) (10 pages) Paper No: JCISE-16-2066; doi: 10.1115/1.4037434 History: Received September 01, 2016; Revised July 12, 2017

An important part of the engineering design process is prototyping, where designers build and test their designs. This process is typically iterative, time consuming, and manual in nature. For a given task, there are multiple objects that can be used, each with different time units associated with accomplishing the task. Current methods for reducing time spent during the prototyping process have focused primarily on optimizing designer to designer interactions, as opposed to designer to tool interactions. Advancements in commercially available sensing systems (e.g., the Kinect) and machine learning algorithms have opened the pathway toward real-time observation of designer's behavior in engineering workspaces during prototype construction. Toward this end, this work hypothesizes that an object O being used for task i is distinguishable from object O being used for task j, where i is the correct task and j is the incorrect task. The contributions of this work are: (i) the ability to recognize these objects in a free roaming engineering workshop environment and (ii) the ability to distinguish between the correct and incorrect use of objects used during a prototyping task. By distinguishing the difference between correct and incorrect uses, incorrect behavior (which often results in wasted time and materials) can be detected and quickly corrected. The method presented in this work learns as designers use objects, and infers the proper way to use them during prototyping. In order to demonstrate the effectiveness of the proposed method, a case study is presented in which participants in an engineering design workshop are asked to perform correct and incorrect tasks with a tool. The participants' movements are analyzed by an unsupervised clustering algorithm to determine if there is a statistical difference between tasks being performed correctly and incorrectly. Clusters which are a plurality incorrect are found to be significantly distinct for each node considered by the method, each with p ≪ 0.001.

Copyright © 2018 by ASME
Your Session has timed out. Please sign back in to continue.


Tuarob, S. , and Tucker, C. S. , 2015, “ Automated Discovery of Lead Users and Latent Product Features by Mining Large Scale Social Media Networks,” ASME J. Mech. Des., 137(7), p. 071402. [CrossRef]
Tuarob, S. , and Tucker, C. S. , 2015, “ Quantifying Product Favorability and Extracting Notable Product Features Using Large Scale Social Media Data,” ASME J. Comput. Inf. Sci. Eng., 15(3), p. 031003. [CrossRef]
Wassenaar, H. J. , Chen, W. , Cheng, J. , and Sudjianto, A. , 2005, “ Enhancing Discrete Choice Demand Modeling for Decision-Based Design,” ASME J. Mech. Des., 127(4), pp. 514–523. [CrossRef]
Hoyle, C. , Chen, W. , Ankenman, B. , and Wang, N. , 2009, “ Optimal Experimental Design of Human Appraisals for Modeling Consumer Preferences in Engineering Design,” ASME J. Mech. Des., 131(7), p. 071008. [CrossRef]
Agard, B. , and Kusiak, A. , 2004, “ Data-Mining-Based Methodology for the Design of Product Families,” Int. J. Prod. Res., 42(15), pp. 2955–2969. [CrossRef]
Kusiak, A. , and Smith, M. , 2007, “ Data Mining in Design of Products and Production Systems,” Annu. Rev. Control, 31(1), pp. 147–156. [CrossRef]
Tucker, C. S. , and Kim, H. M. , 2011, “ Trend Mining for Predictive Product Design,” ASME J. Mech. Des., 133(11), p. 111008. [CrossRef]
Gershenson, J. K. , Prasad, G. J. , and Zhang, Y. , 2004, “ Product Modularity: Measures and Design Methods,” J. Eng. Des., 15(1), pp. 33–51. [CrossRef]
Kurtoglu, T. , Campbell, M. I ., Arnold , C. B., Stone , R. B., and Mcadams , D. A. , 2009, “ A Component Taxonomy as a Framework for Computational Design Synthesis,” ASME J. Comput. Inf. Sci. Eng., 9(1), p. 011007. [CrossRef]
Bonjour, E. , Deniaud, S. , Dulmet, M. , and Harmel, G. , 2009, “ A Fuzzy Method for Propagating Functional Architecture Constraints to Physical Architecture,” ASME J. Mech. Des., 131(6), p. 061002. [CrossRef]
Wang, G. G. , and Shan, S. , 2007, “ Review of Metamodeling Techniques in Support of Engineering Design Optimization,” ASME J. Mech. Des., 129(4), pp. 370–380. [CrossRef]
Apley, D. W. , Liu, J. , and Chen, W. , 2006, “ Understanding the Effects of Model Uncertainty in Robust Design With Computer Experiments,” ASME J. Mech. Des., 128(4), pp. 945–958. [CrossRef]
Hannah, R. , Joshi, S. , and Summers, J. D. , 2012, “ A User Study of Interpretability of Engineering Design Representations,” J. Eng. Des., 23(6), pp. 443–468. [CrossRef]
Gerber, E. , and Carroll, M. , 2012, “ The Psychological Experience of Prototyping,” Des. Stud., 33(1), pp. 64–84. [CrossRef]
Yang, M. C. , 2005, “ A Study of Prototypes, Design Activity, and Design Outcome,” Des. Stud., 26(6), pp. 649–669. [CrossRef]
Houde, S. , and Hill, C. , 1997 , “ What Do Prototypes Prototype,” Handbook of Human-Computer Interaction, Vol. 2, Elsevier, Amsterdam, The Netherlands, pp. 367–381. [CrossRef]
Eppinger, S. , and Whitney, D. , 1995, “ Accelerating Product Development by the Exchange of Preliminary Product Design Information,” ASME J. Mech. Des., 117(4), pp. 491–498. [CrossRef]
Teizer, J. , Venugopal, M. , and Walia, A. , 2008, “ Ultrawideband for Automated Real-Time Three-Dimensional Location Sensing for Workforce, Equipment, and Material Positioning and Tracking,” Transp. Res. Rec., 2081, pp. 56–64. [CrossRef]
Lim, Y.-K. , Stolterman, E. , and Tenenberg, J. , 2008, “ The Anatomy of Prototypes: Prototypes as Filters, Prototypes as Manifestations of Design Ideas,” ACM Trans. Comput.-Hum. Interact., 15(2), p. 7. [CrossRef]
Dalal, N. , and Triggs, B. , 2005, “ Histograms of Oriented Gradients for Human Detection,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, CA, June 20–25, pp. 886–893.
Rublee, E. , Rabaud, V. , Konolige, K. , and Bradski, G. , 2011, “ ORB: An Efficient Alternative to SIFT or SURF,” IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, Nov. 6–13, pp. 2564–2571.
Chi, S. , and Caldas, C. H. , 2011, “ Automated Object Identification Using Optical Video Cameras on Construction Sites,” Comput.-Aided Civ. Infrastruct. Eng., 26(5), pp. 368–380. [CrossRef]
Bo, L. , Ren, X. , and Fox, D. , 2011, “ Depth Kernel Descriptors for Object Recognition,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), San Francisco, CA, Sept. 25–30, pp. 821–826.
Ren, X. , Bo, L. , and Fox, D. , 2012, “ RGB-(D) Scene Labeling: Features and Algorithms,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, June 16–21, pp. 2759–2766.
Russakovsky, O. , Deng, J. , Su, H. , Krause, J. , Satheesh, S. , Ma, S. , Huang, Z. , Karpathy, A. , Khosla, A. , Bernstein, M. , Berg, A. C. , and Fei-Fei, L. , 2015, “ ImageNet Large Scale Visual Recognition Challenge,” Int. J. Comput. Vision, 115(3), pp. 211–252. [CrossRef]
Krizhevsky, A. , Sutskever, I. , and Hinton, G. E. , 2012, “ ImageNet Classification With Deep Convolutional Neural Networks,” 25th International Conference on Neural Information Processing Systems (NIPS), Lake Tahoe, NV, Dec. 3–6, pp. 1097–1105. http://dl.acm.org/citation.cfm?id=2999257
Simonyan, K. , and Zisserman, A. , 2014, “ Very Deep Convolutional Networks for Large-Scale Image Recognition,” preprint arXiv:1409.1556.
Dunleavy, M. , and Dede, C. , 2014, “ Augmented Reality Teaching and Learning,” Handbook of Research on Educational Communications and Technology, Springer, New York, pp. 735–745. [CrossRef]
Kosmadoudi, Z. , Lim, T. , Ritchie, J. , Louchart, S. , Liu, Y. , and Sung, R. , 2013, “ Engineering Design Using Game-Enhanced CAD: The Potential to Augment the User Experience With Game Elements,” Comput.-Aided Des., 45(3), pp. 777–795. [CrossRef]
Jezernik, A. , and Hren, G. , 2003, “ A Solution to Integrate Computer-Aided Design (CAD) and Virtual Reality (VR) Databases in Design and Manufacturing Processes,” Int. J. Adv. Manuf. Technol., 22(11–12), pp. 768–774. [CrossRef]
Bourdot, P. , Convard, T. , Picon, F. , Ammi, M. , Touraine, D. , and Vézien, J.-M. , 2010, “ VR–CAD Integration: Multimodal Immersive Interaction and Advanced Haptic Paradigms for Implicit Edition of CAD Models,” Comput.-Aided Des., 42(5), pp. 445–461. [CrossRef]
Verlinden, J. , and Horváth, I. , 2009, “ Analyzing Opportunities for Using Interactive Augmented Prototyping in Design Practice,” J. Artif. Intell. Eng. Des. Anal. Manuf., 23(3), pp. 289–303. [CrossRef]
Fiorentino, M. , Uva, A. E. , Monno, G. , and Radkowski, R. , 2012, “ Augmented Technical Drawings: A Novel Technique for Natural Interactive Visualization of Computer-Aided Design Models,” ASME J. Comput. Inf. Sci. Eng., 12(2), p. 024503. [CrossRef]
Vélaz, Y. , Arce, J. R. , Gutiérrez, T. , Lozano-Rodero, A. , and Suescun, A. , 2014, “ The Influence of Interaction Technology on the Learning of Assembly Tasks Using Virtual Reality,” ASME J. Comput. Inf. Sci. Eng., 14(4), p. 041007. [CrossRef]
Bradski, G. R. , and Davis, J. W. , 2002, “ Motion Segmentation and Pose Recognition With Motion History Gradients,” Mach. Vision Appl., 13(3), pp. 174–184. [CrossRef]
Ribeiro, P. C. , Santos-Victor, J. , and Lisboa, P. , 2005, “ Human Activity Recognition From Video: Modeling, Feature Selection and Classification Architecture,” International Workshop on Human Activity Recognition and Modelling (HAREM), Oxford, UK, Sept. 9, pp. 61–78. https://www.researchgate.net/publication/237448747_Human_Activity_Recognition_from_Video_modeling_feature_selection_and_classification_architecture
Li, W. , Zhang, Z. , and Liu, Z. , 2010, “ Action Recognition Based on a Bag of 3D Points,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), San Francisco, CA, June 13–18, pp. 9–14.
Morato, C. , Kaipa, K. N. , Zhao, B. , and Gupta, S. K. , 2014, “ Toward Safe Human Robot Collaboration by Using Multiple Kinects Based Real-Time Human Tracking,” ASME J. Comput. Inf. Sci. Eng., 14(1), p. 011006. [CrossRef]
Behoora, I. , and Tucker, C. S. , 2014, “ Quantifying Emotional States Based on Body Language Data Using Non Invasive Sensors,” ASME Paper No. DETC2014-34770.
Tucker, C. , and Kumara, S. , 2015, “ An Automated Object-Task Mining Model for Providing Students With Real Time Performance Feedback,” American Society for Engineering Education Annual Conference and Exposition (ASEE), Seattle, WA, June 14–17. https://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0ahUKEwi3i6ig4-fVAhWR2YMKHeNwA1EQFggwMAE&url=https%3A%2F%2Fwww.asee.org%2Fpublic%2Fconferences%2F56%2Fpapers%2F13126%2Fdownload&usg=AFQjCNEmkbMFMpGjZUWtqYfK-2gnAoQtfw
Lopes, M. , and Santos-Victor, J. , 2005, “ Visual Learning by Imitation With Motor Representations,” IEEE Trans. Syst. Man Cybern., Part B, 35(3), pp. 438–449. [CrossRef]
Jiang, Y. , Lim, M. , and Saxena, A. , 2012, “ Learning Object Arrangements in 3D Scenes Using Human Context,” preprint arXiv:1206.6462.
Koppula, H. S. , Gupta, R. , and Saxena, A. , 2013, “ Learning Human Activities and Object Affordances From RGB-D Videos,” Int. J. Rob. Res., 32(8), pp. 951–970. [CrossRef]
Yu, G. , Liu, Z. , and Yuan, J. , 2014, “ Discriminative Orderlet Mining for Real-Time Recognition of Human-Object Interaction,” Asian Conference on Computer Vision (ACCV), Singapore, Nov. 1–5, pp. 50–65. https://eeeweba.ntu.edu.sg/computervision/Research%20Papers/2014/Discriminative%20Orderlet%20Mining%20For%20Real-time%20Recognition%20of%20Human-Object%20Interaction.pdf
Wieling, M. , and Hofman, W. , 2010, “ The Impact of Online Video Lecture Recordings and Automated Feedback on Student Performance,” Comput. Educ., 54(4), pp. 992–998. [CrossRef]
Chen, P. M. , 2004, “ An Automated Feedback System for Computer Organization Projects,” IEEE Trans. Educ., 47(2), pp. 232–240. [CrossRef]
Calvo, R. A. , and Ellis, R. A. , 2010, “ Students' Conceptions of Tutor and Automated Feedback in Professional Writing,” J. Eng. Educ., 99(4), pp. 427–438. [CrossRef]
Patel, R. A. , Hartzler, A. , Pratt, W. , Back, A. , Czerwinski, M. , and Roseway, A. , 2013, “ Visual Feedback on Nonverbal Communication: A Design Exploration With Healthcare Professionals,” Seventh International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), Venice, Italy, May 5–8, pp. 105–112.
Lamancusa, J. S. , 2006, “ The Reincarnation of the Engineering “Shop”,” ASME Paper No. DETC2006-99723.
Le, Q. V . , 2013, “ Building High-Level Features Using Large Scale Unsupervised Learning,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vancouver, BC, Canada, May 26–31, pp. 8595–8598.
Massey, F. J., Jr. , 1951, “ The Kolmogorov-Smirnov Test for Goodness of Fit,” J. Am. Stat. Assoc., 46(253), pp. 68–78. [CrossRef]


Grahic Jump Location
Fig. 1

An outline of the method presented in this work

Grahic Jump Location
Fig. 2

Top: the RGB image data gathered from a scene (in RGB order); bottom: the depth information given by the sensor (in millimeters)

Grahic Jump Location
Fig. 3

A student beginning the “hammer screw” task in the engineering lab. For the two types of tasks (hammering a screw and a nail), this method determines if the tool they are using is efficient.

Grahic Jump Location
Fig. 4

The transformed skeletons of the shortest and tallest participants in the study

Grahic Jump Location
Fig. 5

Cluster residency by activity label

Grahic Jump Location
Fig. 6

Distribution plots for inlying and outlying distances



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In