ICRA 2025 Submission
Wei-Cheng Tseng, Ellina Zhang, Krishna Murthy Jatavallabhula, Florian Shkurti
Our method takes a few multi-view images of a scene and their corresponding camera poses as input (a) converts them into their Gaussian splatting representation, (b) learns a dynamics model over these representations, and (c) performs visual model-predictive control for granular material manipulation, which requires view synthesis and dynamics rollouts.
Our method contains the following components. (a) The dynamics model f, conditioned on the input action, predicts the temporal evolution of the scene representation. During planning time, we calculate the task objective and backpropagate the gradients to optimize the action sequence (b) The dynamics model f, conditioned on the input action , predicts the temporal evolution of the scene representation . During planning time, we calculate the task objective and backpropagate the gradients to optimize the action sequence such that the granular media can get closed to the target observations.
Our framework represent the granular material and the scene with Gaussian Splatting, which enables us do novel-view synthesis.
Experimental Setup: The robotic manipulator, equipped with a pusher at the end-effector, moves object piles within the workspace. Four calibrated RGBD cameras mounted around the workspace provide visual observations of the environment. We show the results of following tasks.
Splitting: pushing the piles into multiple target regions.
Collecting: pushing the piles into a target region.
Qualitative Results: We show the qualitative results below. From the results we got, granular material finally match the pattern of the target observation after manipulated from their initial configuration.
Manipulation Visualization: We show the manipulation recording here. After several pushes of manipulator, the granular media can reach the target region.
@inproceedings{icra_submission_tseng,
author = {Wei-Cheng Tseng, Ellina Zhang, Krishna Murthy Jatavallabhula, Florian Shkurti},
title = {Granular Material Manipulation with Gaussian Splatting},
year = {2024}
}