10th IFAC Symposium on Robot Control International Federation of Automatic Control September 5-7, 2012. Dubrovnik, Croatia
Discrete fuzzy grasp affordance for robotic manipulators D. Eizicovits, M. Yaacobovich, S. Berman Department of Industrial Engineering and Management, Ben-Gurion University of the Negev, Beer-Sheva, Israel. E-mail: [email protected]
bgu.post.ac.il Abstract: Grasp affordance determines the object-hand relative configurations which lead to successful grasps. Generation and representation of grasp affordances can increase achieved grasp quality and be integrated in path planning algorithms facilitating increased efficiency. Grasp quality is determined by various measures and may have a major impact on task success. Fuzzy grasp affordance can be defined based on a fuzzy grasp quality grade and enhance the previously Boolean notion of grasp affordance. Fuzzy grasp affordances can be represented using a discrete manifold. This facilitates integration of data from various sources and representation optimization using evolutionary algorithms. A method for construction of a discrete fuzzy grasp affordance manifold is presented and demonstrated for apple selective harvesting. The affordance constructed is based on learning from human demonstration. It includes quality grade determination, manifold structure determination, cell quantization, and smoothing. An algorithm for adaptation of the computed manifold to different manipulators and grippers is developed and implemented for two different end effectors. Additionally a method for online integration of the developed affordance is presented. Keywords: Robotic manipulators, fuzzy logic, learning. Robotic path planning has also been studied extensively (Bertram et al. 2006). When multiple degrees of freedom are available in the robotic arm and/or when obstacles are present in the environment path planning forms a complex problem. Additionally, it has been shown that when multiple points are to be reached within a robotic task, e.g. in palletizing or harvesting tasks, the order in which the points are accessed can greatly influence total task execution time (Edan and Nof, 1995).
1. INTRODUCTION Grasp planning can be divided into hand configuration planning (grasp and contact point) and arm transport planning (the path). There are indications that these two processes are separately planned controlled by humans (Jeannerod, 1981). Both components are affected by the task, manipulator, end effector, and object characteristics. Grasp planning is an essential part of object manipulation. Enhancing robotic grasp planning capabilities is crucial for facilitating online robotic operation in dynamic, unstructured environments.
Affordances are intrinsic properties of an object that allow a particular type of manipulation (Gibson, 1977). Affordance defines the relation between an agent and the environment based on the expected outcome of the agent actions on the environment (Şahin et al., 2007). Grasp affordance refers to the relative configuration of an end effector near an object that would yield a successful grasp (De Granville et al., 2006).
Much work has been dedicated to grasp contact point determination, hand configuration synthetises and optimization (Pollard and Wolf, 2005; Li et al., 2007). Yet, especially for dexterous end effectors, due to the large search space, grasp optimization is commonly done off-line. For online contact point determination various sub-optimal synthesis algorithms have proved capable of generating successful grasps (Daoud et al., 2011).
Detry et al., (2010) suggested a method for determination of the grasp affordance density for a given object and end effector. The grasp success probability is represented using a spatial density function in object coordinate space. The grasp quality encoded is a binary success measure where the grasp is deemed either successful if the object was grasped and manipulated, or unsuccessful. The success probability is determined based on either a model (Miller et al., 2003) or on human demonstration (De Granville et al., 2009). Constructing the density function is a complex task. Grasp specimens are clustered into several clusters based on grasp types. A density function is constructed for each cluster and the combined density is created by concatenation of all constructed density functions (Detry et al., 2011).
Determination of grasp quality has also received much attention as it is a major component of grasp synthesis algorithms. A commonly used measure is the magnitude of the largest worst-case disturbance wrench that can be resisted by a grasp of unit strength (Mirtich and Canny, 1994). This measure is suitable for pick and place operations. A more complex measure suitable for a wider range of tasks was suggested by Pollard (Pollard and Wolf, 2005). It compares the task wrench space, i.e., the space of resultant forces and torques that must be applied on the target object in order to complete the task, to the grasp wrench space. This measure quantifies how forcefully the end effector would have to squeeze the object to achieve the required task wrenches.
978-3-902823-11-3/12/$20.00 © 2012 IFAC
IFAC SYROCO 2012 September 5-7, 2012. Dubrovnik, Croatia
Representing grasp quality as a binary parameter is essential to the derivation of the affordance density since it is rooted in probability theory, but it limits the usability of the grasp affordance to cases where such a representation is suitable. We suggest enhancing the notion of grasp affordance to include non-binary grasp quality measures. This can be achieved by using a fuzzy based representation of grasp quality. In addition fuzzy logic facilitates integration of several grasp quality measures, it allows combination of both discrete and continues measures, and can deal with partial information and data perturbations. The grasp affordance density function suggested by (Detry et al., 2011) is a continuous function. We suggest a discrete representation of grasp affordance over 2D manifold covering the object to be grasped. The discrete representation may be less compact than the continuous one yet it facilitates the use of advanced optimization methods and is highly suitable for online path planning.
2.2 Fuzzy grasp affordance Fuzzy grasp affordance represents a quality grade of a grasp, while in contrast probabilistic grasp affordance represents the probability of a grasp to be successful. In that the fuzzy approach facilitates using and augmenting various grasp quality measures rather than a single binary success/fail measure. Representation of multiple, non-binary grasp quality measures is in-line with human studies that show subjects optimize more than one quality measure when grasping objects (Friedman and Flash, 2007). To form the fuzzy grasp affordance each grasp quality measure is fuzzified and a fuzzy rule base is constructed for augmenting the different measures. The final grasp quality grade is computed by defuzzification of the rule base output. We opt to use the centroid defuzzification method since it is most suitable for integration of multiple inputs. 2.3 Discrete grasp affordance manifold
Development of robots for selective harvesting has been studied during the last few decades (Sarig, 1993; Baeten et al. 2007). However although technology exist for detaching fruits from trees, e.g., trees shakers and robotic systems which are manually operated, (Peterson, 2005) the problem of using automation for selective harvesting of soft fruits is still, for the most part, unsolved. The complex, uncertain and dynamic nature of the agriculture environment forms a major challenge for constructing economic, fully automated, selective harvesting machines (Edan et al., 2009). Grasping and manipulating soft fruit successfully and quickly, is among the primary issues that need to be tackled for achieving this goal.
Using a discrete representation of the grasp affordance facilitates augmentation of affordance learning from multiple sources and adaptation of the affordance manifold when additional input is available. The discrete representation simplifies the adaptation of evolutionary optimization and data manipulation algorithms for optimizing the affordance representation. There are two major issues that need to be considered when determining the affordance manifold: the topology of the manifold and the cell granularity. Both increased topology precision and increased representation granularity lead to increased complexity and memory demands.
Using the notion of fuzzy grasp affordance for representation of knowledge about the suitable grasps can assist improved path planning and reduction of harvesting time while avoiding fruit damage. In addition formation of the discrete fuzzy affordance manifold can be integrated with path optimization algorithms both at the single fruit reach-to-grasp level and for harvesting sequence optimization. This paper is organized as follows: section 2 presents the method including the discrete fuzzy affordance manifold construction and its integration with in an online system; section 3 presents a case study forming a discrete fuzzy affordance manifold for apple harvesting; finally conclusions and future research are presented in section 4.
The grasp affordance manifold represents the end effector configuration with respect to the grasped object. In the general case the grasp affordance manifold should include the spatial representation of all six degrees of freedom of the configuration space. Yet this representation forms a very large structure. To reduce the complexity of the representation we fix three degrees of freedom and reduce the manifold to a surface that enclosed the objects. The surface determines the distance of the wrist from the object during the grasp and the surface is aligned such that it is perpendicular to the wrist orientation (pitch and yaw). The three remaining degrees of freedom are the 2D wrist position on the surface and the rotation about the axis perpendicular to the surface at the wrist position (roll). For further reduction of the representation complexity, we construct the surface as an amalgamation of primitive sub-surfaces, i.e., sub surfaces that are, planar, cylindrical, spherical, or conic. This is in line with the work of Miller at al. (2003) who modelled objects using shape primitives and showed good grasps can be devised based on this representation. Increasing the number of sub-surfaces can lead to a more accurate representation yet will increase model complexity. The best trade-off can be found using decision criteria such as the Bayesian information criterion (BIC), which is a criterion for model selection that compromises between the number of parameters and the model fit:
2. METHOD 2.1 Overview In this section we first present the fuzzy affordance concept, we then outline the general structure of the discrete grasp manifold and how to adapt it for a specific object. Several such manifolds should be created to cover the variety of objects that the robotic apparatus must handle during run time operation. Mapping of the discrete fuzzy grasp manifold between non-identical agents is described facilitating robotic affordance creation based on human demonstration. Finally we discuss the use of the affordance manifold during runtime operation.
IFAC SYROCO 2012 September 5-7, 2012. Dubrovnik, Croatia
Where L is the maximized value of the likelihood function, n is the number of data points in the model and k is the number of parameters.
the manipulator is unable to reach part of the configurations, the mapping must take this into account and all configurations which the manipulator cannot access must be mapped to configurations that are part of its configuration space. This can be done by performing a non-linear constraint search on the manipulators configuration space for finding the configuration closest to the unreachable configuration (Tarokh, Kim 2007).
Selecting the representation granularity is another major issue that forms a trade-off between accuracy demands and model complexity. The granularity should be optimized for each object, each sub-surface and each parameter (position and roll). Though for a fully optimal representation the subsurface representation and granularity should be optimized together, a two stage optimization, which is much simpler to perform, may suffice for most practical cases.
2.5 Run-time operation The affordance module includes a library of affordance manifolds and a model of the objects for which these manifolds have been constructed. The model reflects the number and type of sub-surfaces used to construct the affordance manifold. During Run-time operation there are several tasks that should be performed by the affordance handling module.
2.4 Mapping grasps affordances between non identical agents Learning from demonstration (LFD) can be used for learning grasp affordances. Since in the general case the robot manipulator and end effector are not anthropomorphic, a mapping must be devised between the affordance manifold learned from the human teacher and the manifold suitable for the robot (De Granville et al., 2006). To simplify the mapping the dissimilarities in the arm and in the end effector are addressed separately.
The run-time operation is described in Figure 1. When the system has identified an object which it is about to grasp, the affordance module needs to find the affordance manifold that was constructed for an object that is most similar to the object at hand. The similarity is defined using a distance metric in accordance to the object model used. In case the manifold found is of an object with a similarity grade higher than a threshold, the manifold is transferred to the path planning module. In case the similarity grade is lower than the threshold, the module should announce that the object is unknown and request a learning session for constructing an additional affordance manifold. The affordance manifolds can be updated during run time based on success reports from the system following the manipulation operation.
The difference between the human hand and the robotic end effector is expected to be the most prevalent issue influencing the required mapping. It is assumed that some resemblance can be found between the two otherwise, LFD may not be a good way for learning the affordance manifold for the robot at hand. The main issues that must be addressed when devising the mapping are the number of fingers and the relative end effector size. When the robotic manipulator has a different number of fingers then the human demonstrator a mapping must be devised for the grasp itself. The virtual finger method (Iberall, 1997) can be used for performing such a mapping. Using this mapping the human fingers are mapped to virtual fingers aligned with the fingers of the robotic end effector. Each virtual finger can represent up to four actual fingers. In case of a mapping of the human hand to a common two-jaw parallel gripper the four fingers (index, middle, ring, pinky) are mapped to one jaw (finger) and the thumb is mapped to the second jaw. For more complex end effector the mapping strived to achieve stability between the forces and torques of the different virtual fingers. When there is a size difference between the robotic end effector and the hand of the human demonstrator the distance of the wrist position from the object must be adjusted. That is the normal distance of the affordance manifold from the object must be adjusted according to the size difference between the hand and the end effector. In addition in case the maximum aperture of the robotic end effector is smaller than that of the human hand, affordances requiring apertures unreachable by the robotic end effector must be extracted from the database. When the robotic arm has six or more degrees of freedom and its configuration space fully contains all configurations in the vicinity of the object, then an identity mapping can be devised for the arm position. When this is not the case and
Figure 1: methodology for online operation
IFAC SYROCO 2012 September 5-7, 2012. Dubrovnik, Croatia
3. CASE STUDY – ROBOTIC SELECTIVE HARVESTING OF APPLES 3.1 LFD data collection Data of human reach-to-grasp movements towards fruit with an intention to pick them were collected from 15 subjects in an apple picking experiment. The experiment took place in an apple orchard in Belgium and the pickers were professional apple pickers. The experiment included collection of both objective data (finger contact pressure and hand trajectory) and subjective data based on subject oral report regarding perceived grasp quality. The data were used to find hand orientation and position at the initial grasp moment. The full details of the experiment can be found in Yaacobovich et al. (2012).
Figure 3: Left - a cylindrical cover of an apple. Right - a spherical cover of the same apple. The spherical cover was constructed such that its Z axis was aligned with the peduncle of the apple. The intersection points between the wrist approach direction and the tangent to the cover were defined as the grasping position on the sphere. The two position parameters on the spherical surface (θ, ) were divided into cells, were each cell was of 18X18 degrees. Since there were not enough measurements to account for the roll parameter it was disregarded at this stage. Thus the representation was made up of 400 cells. Each cell received the average fuzzy quality grade of all measured grasps that fell within it. A cell without any recoded sample received grade of 0, since there is no knowledge regarding it. The manifold was projected on a 2D surface, and smoothed using a 5*5 Gaussian kernel and then projected back (Figure 4). The data smoothing was done under the assumption that the distribution of the grasp quality is normal. From figure 4C it is apparent that the frontal and downward side of the apple is more adequate for grasping the apple when compared to other regions of the manifold cover.
3.2 The fuzzy grasp affordance for apple selective harvesting Two different grasp quality measures were defined based on the available inputs: the subjective assessment of the grasp quality and the maximum applied finger pressure. The direction of the measures is opposite, i.e. low force and/or high subjective assessment are preferred. Each of the measures was fuzzified using membership functions (Figure 2). A fuzzy rule (Table 1) was formed mapping the two inputs into a fuzzy quality grade. The fuzzy quality grade was defuzzified to create a single continues grade between 0 and 1 using the centroid defuzzification method. Degree of membership
Degree of membership
B. Quantized surface
Maximal Applied Force [V]
Figure 2: Fuzzification of the subjective (on the left) and objective (on the right) quality measures. Table 1. Fuzzy rule matrix Force Output H M L Subjective L B B M M B M G H M G G L- low, M- medium, H- high, B- bad, M- medium, G- good
C. Gaussian Smoothing
3.3. Constructing the discrete grasp affordance manifold In the construction of the affordance manifold there is a trade-off between accuracy demands and model complexity. For the apple we selected a spherical cover manually without a through optimization process. To visualize the reasoning for our selection we depict a spherical and cylindrical covers in Figure 3. The cylindrical cover is less appropriate and in addition it is constructed from three sub-surfaces, while the spherical cover is constructed from a single surface.
Figure 4: In all three images the arrow represents the direction of the peduncle. A. depicts the measured grasp locations (dots). B. depicts the quantization into cell patches where the quality grade is lower for darker cells and black cells are empty cells. C. depicts the manifold after Gaussian smoothing. 256
IFAC SYROCO 2012 September 5-7, 2012. Dubrovnik, Croatia
3.3. Translation to the robotic apparatus
the little finger were mapped to the second finger capable of spread motion the case of the pneumatic gripper (Figure 6). There was no need for additional mapping since the Barrett hand allows execution of palm grasps from a distance similar to that of the human hand.
A robotic apparatus was constructed indoors to test the affordance manifold. The experimental setup included a 6 degree of freedom MOTOMAN UP61 manipulator (Figure 5) and two different types of grippers: the Barrett hand2 and the SCHUNK pneumatic gripper3. In addition a physical model of an apple was printed using a 3D printer (Figure 5).
Figure 6: Illustration of the finger mapping between the human hand and the Barrett hand. 3.4. Demonstration of mapped affordance An initial test of the mapped affordances found that all tested grasp configurations produced stable grasps. An example is given below were a robotic wrist configuration established according to the affordance manifold of each apparatus is depicted in (Figure 7).
Figure 5: The MOTOMAN UP6 arm holding the apple model using the SCHUNK pneumatic gripper. Since the MOTOMAN UP6 has 6 degrees of freedom it can reach every configuration in the task space therefore an identity mapping was formed for the affordance configuration. The end effectors used are not anthropomorphic and are of different size than the human hand therefor mapping based on the virtual finger concept were constructed separately for each end effectors. The SCHUNK pneumatic gripper has only two fingers and can apply a soft pressure on the grasped object (which in the case of fruits is very important). Each finger of the gripper is bigger than the average human finger, and the gripper cannot wrap the object in a palmer grasp as executed by manual human subjects. Thus the grasps used by the gripper had to be force opposition grasps with a 2cm gap between the gripper wrist and the grasped object. Since the gripper has only two fingers the translation using the virtual finger is straight forward. The thumb was mapped to the one finger and the rest of the fingers were mapped to the second finger. In order to overcome the issue of the gap between the wrist and the object, the affordance manifold was extended to fit the larger distance of the gripper from the object.
Figure 7: Grasp configuration according to the affordance manifold of each manipulator: the SCHUNK pneumatic gripper, a human hand and the Barrett gripper.
4. CONCLUSIONS AND FUTURE WORK
The Barrett hand has three fingers and as for the SCHUNK gripper each finger is a bit bigger than the average human finger. However the Barrett hand is capable of wrapping the object in a palmer grasp since it has two joints in each finger. The mapping between the human fingers hand and the Barrett hand was done as following: the thumb was mapped to the Barrett thumb finger (which does not perform the spread motion). The index and the middle finger were mapped two one of the fingers capable of spread motion and the ring and
A discrete fuzzy affordance manifold for representation of grasp quality has been developed. The constructed affordance manifold can be used in run-time for supporting reach-tograsp path planning and job sequencing. An affordance manifold was constructed for an apple selective harvesting task based on LFD. We are currently working on constructing an affordance manifold for a pepper selective harvesting task to test the concept for objects with different characteristics. In addition we are working on the determining the number of affordance manifolds required for run-time operation for getting a good fit to the object verity in apple and pepper harvesting. Finally
http://www.motoman.com http://www.barrett.com 3 http://www.schunk.com 2
IFAC SYROCO 2012 September 5-7, 2012. Dubrovnik, Croatia
we are working on enhancing the initial manifold constructed from the human demonstration with additional data from other sources such as model-based grasp synthesis and experimental results from the real system.
Iberall, T. 1997, "Human prehension and dexterous robot hands", The International Journal of Robotics Research, vol. 16, no. 3, pp. 285-299. Jeannerod, M. 1981, "Intersegmental coordination during reaching at natural visual objects", Attention and performance IX, vol. 9, pp. 153-168. Li, Y., Fu, J.L. & Pollard, N.S. 2007, "Data-driven grasp synthesis using shape matching and task-based pruning", IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 4, pp. 732-747. Miller, A.T., Knoop, S., Christensen, H.I. & Allen, P.K. 2003, "Automatic grasp planning using shape primitives", IEEE International Conference on Robotics and Automation., pp. 1824. Mirtich, B. & Canny, J. 1994, "Easily computable optimum grasps in 2-D and 3-D", IEEE International Conference on Robotics and Automation, pp. 739-747. Peterson, D. 2005, "Harvest mechanization progress and prospects for fresh market quality deciduous tree fruits", HortTechnology, vol. 15, no. 1, pp. 72-75. Pollard, N.S. & Wolf, A. 2005, "5 Grasp Synthesis from Example: Tuning the Example to a Task or Object", Multi-point Interaction with Real and Virtual Objects, , pp. 77-90. Şahin, E., Çakmak, M., Doğar, M.R., Uğur, E. & Üçoluk, G. 2007, "To afford or not to afford: A new formalization of affordances toward affordance-based robot control", Adaptive Behavior, vol. 15, no. 4, pp. 447. Sarig, Y. 1993, "Robotics of fruit harvesting: A state-of-theart review", Journal of Agricultural Engineering Research, vol. 54, no. 4, pp. 265-280. Tarokh, M. & Kim, M. 2007, "Inverse kinematics of 7-DOF robots and limbs by decomposition and approximation", Robotics, IEEE Transactions on, vol. 23, no. 3, pp. 595600. Yaacobovich, M., Eizicovits, D., Berman, S. 2012, "Grasp Affordance for Robotic Selective Harvesting based on Human Demonstrations", CIGR 2012, unpublished.
5. ACKNOWLEDGMENTS This work is supported by the European Commission in the 7th Framework Programme (CROPS GA no 246252). The authors thank Prof. Wouter Saeys, and Tien Thanh Nguyen from the University of Leuven, Dr. Jochen Hemming and Bart van Tuijl from Wageningen University for their support in the LFD field experiments. REFERENCES Baeten, J., Donné, K., Boedrij, S., Beckers, W. & Claesen, E. 2007, "Autonomous fruit picking machine: A robotic apple harvester", 6th International Conference on Field and Service Robotics - FSR 2007, Chamonix: France. Bertram, D., Kuffner, J., Dillmann, R. & Asfour, T. 2006, "An integrated approach to inverse kinematics and path planning for redundant manipulators", IEEE International Conference on Robotics and Automation (ICRA 06), pp. 1874. Daoud, N., Gazeau, J.P., Zeghloul, S. & Arsicault, M. 2011, "A fast grasp synthesis method for online manipulation", Robotics and Autonomous Systems, vol. 59, no. 6, June 2011, pp. 421–427. De Granville, C., Southerland, J. & Fagg, A.H. 2006, "Learning grasp affordances through human demonstration", Proceedings of the International Conference on Development and Learning (ICDL 06). De Granville, C., Wang, D., Southerland, J., Fagg, A.H. & Platt, R. 2009, "Grasping Affordances: Learning to connect vision to Hand Action", The Path to Autonomous Robots, pp. 1-22. Detry, R., Başeski, E., Popović, M., Touati, Y., Krüger, N., Kroemer, O., Peters, J. & Piater, J. 2010, "Learning continuous grasp affordances by sensorimotor exploration", From Motor Learning to Interaction Learning in Robots, pp. 451-465. Detry, R., Kraft, D., Kroemer, O., Bodenhagen, L., Peters, J., Krüger, N. & Piater, J. 2011, "Learning grasp affordance densities", Paladyn, Journal of Behavioral Robotics, vol. 2, no. 1, pp. 1-17. Edan, Y., Han, S. & Kondo, N. 2009, "Automation in agriculture", Springer Handbook of Automation, , pp. 1095-1128. Edan, Y. & Nof, S.Y. 1995, "Motion economy analysis for robotic kitting tasks", The International Journal of production research, vol. 33, no. 5, pp. 1231-1227. Friedman, J. & Flash, T. 2007, "Special issue: original article task-dependent selection of grasp kinematics and stiffness in human object manipulation", Cortex, vol. 43, pp. 444-460. Gibson, J.J. 1977, "The theory of affordances", Perceiving acting and knowing: Toward an ecological psychology, , pp. 67-82.