Neural Field Movement Primitives for <br> Joint Modelling of Scenes and Motions

Neural Field Movement Primitives for
Joint Modelling of Scenes and Motions

Ahmet E Tekden¹ Marc P. Deisenroth² Yasemin Bekiroglu^1,2

¹Chalmers University of Technology ²University College London

Abstract

This paper presents a novel Learning from Demonstration (LfD) method that uses neural fields to learn new skills efficiently and accurately. It achieves this by utilizing a shared embedding to learn both scene and motion representations in a generative way. Our method smoothly maps each expert demonstration to a scene-motion embedding and learns to model them without requiring hand-crafted task parameters or large datasets. It achieves data efficiency by enforcing scene and motion generation to be smooth with respect to changes in the embedding space. At inference time, our method can retrieve scene-motion embeddings using test time optimization, and generate precise motion trajectories for novel scenes. The proposed method is versatile and can employ images, 3D shapes, and any other scene representations that can be modeled using neural fields. Additionally, it can generate both end-effector positions and joint angle-based trajectories. Our method is evaluated on tasks that require accurate motion trajectory generation, where the underlying task parametrization is based on object positions and geometric scene changes. Experimental results demonstrate that the proposed method outperforms the baseline approaches and generalizes to novel scenes. Furthermore, in real-world experiments, we show that our method can successfully model multi-valued trajectories, it is robust to the distractor objects introduced at inference time, and it can generate 6D motions.

Paper

Ahmet E Tekden, Marc P. Deisenroth, Yasemin Bekiroglu
Neural Field Movement Primitives for Joint Modelling of Scenes and Motions

Interpolations between demonstrations

Bilinear interpolation between embeddings of four demonstrations. Scene images for these demonstration are given on the each corresponding corner.

Move the marker to see the scenes and motions generated for different embeddings.

The scene function shows the reconstructed scene image. The motion function illustrates the corresponding minimum implicit cost value for each x and y indexes.

Supplementary Video

Code

Code is coming soon.