Questions of the robot not completing the goal in the result visualizations of Kuka stacking experiment.

[mls1999725]

Thanks for your excellent work. When I ran the Kuka branch cube stacking experiment with your pre-trained model, I encountered the following problems. Can you help me check if these results are normal? What are the possible reasons? Is there a way to solve these problems?

For conditional stacking and rearrangement stacking, too many unsuccessful single stacking sequences may cause the step length to be too long or fail to reach the goal within the specified step length.

I ran conditional_kuka_planning_eval.py and rearrangment_kuka_planning_eval.py, where each goal only runs three steps as shown in this line of code:

diffuser/scripts/rearrangment_kuka_planning_eval.py

Line 150 in 7c4d4da

for i in range(3):

My understanding is that only when the block is successfully placed will it enter the next sequence of the goal, as shown in this line of code:

diffuser/gym_stacking/env.py

Line 237 in 7c4d4da

self.progress = min(self.progress + 1, 2)

and

diffuser/gym_stacking/env.py

Line 244 in 7c4d4da

self.progress = min(self.progress + 1, len(self.stacks) - 1)

. But the goal is not just three sequences, as shown in this line of code:

diffuser/gym_stacking/env.py

Line 296 in 7c4d4da

self.goal = np.random.permutation(4)

and

diffuser/gym_stacking/env.py

Line 321 in 7c4d4da

self.stacks.append((top_cube, bottom_cube))

. I then got some visualizations of running the sequence repeatedly with no successful placements, as shown in the video below, and in the bracketed sequence, the former is the top block and the latter is the bottom block.

Conditional Stacking

1. Block stacking steps performed by the robot arm is (red->green)->(red->green)->(red->green)

The Google Drive link for the video is https://drive.google.com/file/d/1uhnEpErjG14RYjCO7X47mMoNoRxa4z06/view?usp=sharing

When completing the third sequence (red->green), the arm picked up both the red and green blocks and did not place the red block on the green block, causing the sequence to fail again.

2. Block stacking steps performed by the robot arm is (blue->red)->(blue->red)->(blue->red)

The Google Drive link for the video is https://drive.google.com/file/d/1e-wPCw4i65pChGQmkgUgmjWN-kcVVs9X/view?usp=sharing

The arm executed the sequence (blue->red) three times and still failed, while the conditional stacking required three different sequences to complete the goal.

3. Block stacking steps performed by the robot arm is (yellow->red)->(yellow->red)->(yellow->red)

The Google Drive link for the video is https://drive.google.com/file/d/1vqscHsiqHdIviNmac02pTrGaJRhZVEi5/view?usp=sharing

After executing the sequence (yellow->red) three times, it still fails, and the execution path of the arm in the third time is very long。

Rearrangement Stacking

1. Block stacking steps performed by the robot arm is (blue->red)->(blue->red)->(green->blue)

The Google Drive link for the video is https://drive.google.com/file/d/1hDqxbseyXW--Y091hSVzH21dzDwqHElH/view?usp=sharing

In the third step, when placing the green block on the blue, it fails and destroys the previous (blue->red) stack.

2. Block stacking steps performed by the robot arm is (red->blue)->(red->blue)->(green->red)

The Google Drive link for the video is https://drive.google.com/file/d/1xpGmXMMXGiQdnr2wumrFrX6te1Z-vhHN/view?usp=sharing

In the third step, when placing the green block on the red, it fails and destroys the previous (red->blue) stack.

3. Block stacking steps performed by the robot arm is (red->yellow)->(red->yellow)->(red->yellow)

The Google Drive link for the video is https://drive.google.com/file/d/13ZKOpwvNPDM39qMABWi9ZIj8niuQJsTx/view?usp=sharing

The arm executed the (red->yellow) sequence three times and still failed.

4. Block stacking steps performed by the robot arm is (red->yellow)->(red->yellow)->(blue->green)

The Google Drive link for the video is https://drive.google.com/file/d/1oYZrUdA7v3nffQYJHkDEyd1Rw2HAxVSy/view?usp=sharing

The execution of the third sequence (blue->green) fails, and the goal of rearrangement stacking may be more than three steps, which makes the final goal impossible to achieve.

I think the reason why the above goal cannot be achieved within three steps is that there are too many unsuccessful single-step sequences, and increasing the total step length will lead to too much time consumption. So how to increase the probability of single-step success? Do I need to modify the guidance function and retrain the diffusion model to strengthen the constraints?

[yilundu]

Hi -- you can probably increase the success rate by either improving the guidance process (for instance you could run a couple steps of MCMC) or by explicitly training the network to be conditioned constraints.

[mls1999725]

Hi -- you can probably increase the success rate by either improving the guidance process (for instance you could run a couple steps of MCMC) or by explicitly training the network to be conditioned constraints.

Thanks a lot!

I don't quite understand the explicit training of the network to be conditioned constraints. Can you describe it in more detail? Does it mean that I need to retrain your diffusion model? Because your model already adds the conditioned constraints: 1) maximize the probability that the final state of the trajectory matches the goal configuration, and 2) enforce the contact constraint between the end effector and the cube during stacking (for 2, I found that the contact constraint was not used as a condition in the code, but instead the L1 distance between the xy coordinates of the cubes was used, as shown in this line:

diffuser/diffusion/denoising_diffusion_pytorch.py

Line 454 in 7c4d4da

dist = -0.1 * torch.abs(stack_xy - place_xy).mean(dim=-1).mean(dim=-1)

. I wonder if this difference really exists?)

I changed the video links to Google Drive in the above issue for your reference.

[mls1999725]

Looking forward to your reply. Thank you very much for your help.

[yilundu]

Yes -- you during the training time of the model -- you can just directly condition it on the desired constraint. This can lead to better performance.