roboninja.cs.columbia.edu Open in urlscan Pro
128.59.16.27  Public Scan

Submitted URL: http://roboninja.cs.columbia.edu/
Effective URL: https://roboninja.cs.columbia.edu/
Submission: On November 02 via api from US — Scanned from DE

Form analysis 0 forms found in the DOM

Text Content

ROBONINJA: LEARNING AN ADAPTIVE CUTTING POLICY FOR MULTI-MATERIAL OBJECTS

We introduce RoboNinja, a learning-based cutting system for multi-material
objects (i.e., soft objects with rigid cores such as avocados or mangos). In
contrast to prior works using open-loop cutting actions to cut through
single-material objects (e.g., slicing a cucumber), RoboNinja aims to remove the
soft part of an object while preserving the rigid core, thereby maximizing the
yield. To achieve this, our system closes the perception-action loop by
utilizing an interactive state estimator and an adaptive cutting policy. The
system first employs sparse collision information to iteratively estimate the
position and geometry of an object's core and then generates closed-loop cutting
actions based on the estimated state and a tolerance value. The "adaptiveness"
of the policy is achieved through the tolerance value, which modulates the
policy's conservativeness when encountering collisions, maintaining an adaptive
safety distance from the estimated core. Learning such cutting skills directly
on a real-world robot is challenging. Yet, existing simulators are limited in
simulating multi-material objects or computing the energy consumption during the
cutting process. To address this issue, we develop a differentiable cutting
simulator that supports multi-material coupling and allows for the generation of
optimized trajectories as demonstrations for policy learning. Furthermore, by
using a low-cost force sensor to capture collision feedback, we were able to
successfully deploy the learned model in real-world scenarios, including objects
with diverse core geometries and soft materials.

--------------------------------------------------------------------------------


PAPER

Latest version: arXiv
Robotics: Science and Systems (RSS) 2023



Code is available on GitHub.

--------------------------------------------------------------------------------


TEAM

Zhenjia Xu 1
Zhou Xian 2
Xingyu Lin 3
Cheng Chi 1
Zhiao Huang 4
Chuang Gan 5†
Shuran Song 1†

1 Columbia University           2 CMU           3 UC Berkeley           4 UC San
Diego           5 UMass Amherst & MIT-IBM Lab


BIBTEX

@inproceedings{xu2023roboninja,
	title={RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects},
	author={Xu, Zhenjia and Xian, Zhou and Lin, Xingyu and Chi, Cheng and Huang, Zhiao and Gan, Chuang and Song, Shuran},
	booktitle={Proceedings of Robotics: Science and Systems (RSS)},
	year={2023}
}

--------------------------------------------------------------------------------


TECHNICAL SUMMARY VIDEO (WITH AUDIO)

--------------------------------------------------------------------------------


REAL-WORLD EVALUAITON

(1) AVOCADO



The robot first lowers the knife to cut into the avocado. It then encounters a
collision with the core inside. The robot then retracts its state to a few steps
earlier. Meanwhile, the collision signal is used for an updated estimation of
the core state. The robot then continues the cutting process based on the
updated state estimation. After a few collisions, our system is able to
iteratively update the estimation of the core and generates a physically
plausible cutting trajectory to cut off most of the flesh of the avocado. To
better resemble real-world scenarios, we execute the same policy multiple times
with different initial rotation angles to cut off more soft flesh from different
directions.

(2) MANGO



To effectively cut through fiber-rich material, such as mango skin, we add an
additional horizontal, repetitively back-and-forth slicing primitive, in
addition to our vertical cutting trajectory.

(3) MORE FRUITS



Note that in practice, the knife may exhibit visible deformations and the object
pose may also be changed during cutting, which could disrupt the accuracy of
state estimation due to misleading collision positions. However, our cutting
policy is robust enough to complete the task even with inaccurate estimations of
the core.

(4) BONE-IN-MEAT (OXTAIL)



In order to resemble real-world situations more realistically, we employ a
bimanual setup, where one arm with a parallel-jaw gripper (WSG50) holds the bone
and the other arm performs the cutting action.

--------------------------------------------------------------------------------


REAL-WORLD SETUP



We design and construct a compact cutting tool equipped with a force sensor. The
force is measured by a strain gauge as an analog electrical signal. The signal
is then converted to a digital signal and transmitted to the robot controller
through an A/D converter and a Raspberry Pi Zero, respectively. We also 3D print
8 in-distribution and 5 out-of-distribution geometries from the test set for
evaluaiton. As for the soft material, we use Kinect Sand as a proxy because of
its stable physics property.

--------------------------------------------------------------------------------


EVALUAITON ON 3D PRINTED CORES AND KINETIC SAND

(1) IN-DISTRIBUTION GEOMETRIES



(2) NOVEL GEOMETRIES



Our simulation-trained policy demonstrates strong generalization capabilities,
effectively handling both in-distribution and out-of-distribution cores in a
real-world setting. With only a few collisions, it is able to accurately
estimate the core geometry and cut off the majority of the sand with a smooth
cutting trajectory.

--------------------------------------------------------------------------------


COMPARISION IN SIMULATION

In the 2D view (bottom), the ground truth geometry is shown in gray, and the
estimation is shown in brown. In both views, the forward trajectory of the knife
is demonstrated in blue, and the retraction trajectories due to collision are
visualized in red. The cutting trajectory of [RL] is very jittering and becomes
too conservative after a few collisions. [Greedy] strictly follows the contour
of the estimated geometry, leading to exceeding energy consumption during abrupt
rotations. Both [NN] and [Non-Adaptive] get stuck can’t complete the cutting
task within 10 collisions. [RoboNinja] is able to iteratively update the
estimate of the core after each collision and adaptively adjust the cutting
trajectory with optimized energy consumption.

--------------------------------------------------------------------------------


ACKNOWLEDGEMENTS

We would like to thank Huy Ha, Zeyi Liu, and Mandi Zhao for their helpful
feedback and fruitful discussions. This work was supported in part by NSF Awards
2037101, 2132519, 2037101, and Toyota Research Institute. Dr. Gan was supported
by the DARPA MCS program and gift funding from MERL, Cisco, and Amazon. We would
like to thank Google for the UR5 robot hardware. The views and conclusions
contained herein are those of the authors and should not be interpreted as
necessarily representing the official policies, either expressed or implied, of
the sponsors.

--------------------------------------------------------------------------------


CONTACT

If you have any questions, please feel free to contact Zhenjia Xu.