For simple grippers
- Obtain a Point Cloud or Depth image of the objects of interest
- Pre-process the data to remove extraneous information
- Segmentation: Cluster points that are close enough to identify individual objects
- Filter clusters by size, geometry or other features to down-select objects of interest
- Planning: Compute center-of-mass and principal axes of relevant clusters
- Collision-checking: Generate possible grasps and check for collisions with point clouds
- Execution: Physically test a grasp by monitoring jaw distances, as well as forces and torques at the wrist
Depending on the task, these steps will be easier or harder or even skippable.
For multi-fingered Hands
- do a very similar thing but utilize random sampling to find best options