Skip to content
This repository has been archived by the owner on Dec 1, 2020. It is now read-only.

Improve PDB state machine to handle master connection loss better #35

Open
martijnvandermarel opened this issue Aug 27, 2019 · 2 comments
Assignees

Comments

@martijnvandermarel
Copy link
Contributor

martijnvandermarel commented Aug 27, 2019

pdb_lpc_state_machines

We want the PDB to disable HV when the master crashes. Especially when sending torque commands over EtherCAT, when the master software crashes and the last sent torque command is repeated, joints may keep moving.
IMotionCubes go to fault state a short time after losing EtherCAT communication and stop actuating the joints. However, this time may be too large (endstops may already be hit before IMC shuts down)

Therefore it is an option to have the PDB detect when the master has stopped (through crash or manual stop), and consequently disable the high voltage nets. There is already a method of detecting the master state built into the current version of the exoskeleton ( https://github.com/project-march/hardware-interface/blob/5dd978e977f44678940d917413d8f83eb10187ac/march_hardware_interface/src/march_hardware_interface.cpp#L315 in the hardware interface toggles a bit every cycle (pinging), while

if(mosi.masterOk != lastMasterOk){ // Master sent a masterOk signal
checks whether the master is still pinging).
However, what's missing is something that disables HV when the pinging is not detected. The current version of the state machine on the PDB is pictured in the figure above on the left side.

It is not possible to disable HV in the LV_Ok state, because when the master is starting up the exoskeleton, the master needs to be able to turn on HV nets before it starts pinging.
As a result, we will need an extra state. The proposed solution is on the right in the figure above.
From an Idle state where HV is disabled, the Master_Startup state can be reached if some signal is sent from master to PDB notifying the PDB that the master would like to start the startup procedure. The PDB then enables HV, and the master starts up. When the master crashes, the PDB goes back to Idle state and disables HV, until the master is ready to start up again.

Notes:

  • Make sure to reset the master_to_startup signal when the master is done with startup. Otherwise, this signal stays true, even when the master crashes, causing the PDB to go from Idle to Master_Startup state immediately.
  • The master has proved to be a bit bad when it comes to exact timing. You want the PDB to check for a master presence for a rather small period of time for safety reasons, but the master may not always meet this requirement and fail to ping on time even though it hasn't crashed. In case HV is then disabled, the IMotionCubes will crash with undervoltage errors. This has occured in the past. Possible solution: a realtime kernel on the master?
  • For the master_to_startup signal, you will also need to adapt the
    hardware-interface. I made an issue for it in that repository: Improve master-PDB communication so master crash can be detected better hardware-interface#141
  • No additional EtherCAT variable should be needed for the master_to_startup signal. Currently, the pinging flips a single bit in a 8-bit MOSI variable, so 7 bits are unused and should be more than enough to send a master_to_startup message.
@martijnvandermarel
Copy link
Contributor Author

@bjornminderman Maybe a good issue to look at to understand the PDB software. Feel free to fix it if you want :)

@martijnvandermarel
Copy link
Contributor Author

@JitskedeVries @sopje @TimBuckers @rarriens This issue describes everything I currently know about the PDB HV disabling when master crashes.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Development

No branches or pull requests

2 participants