Random brakes with: Exceeded Goal Threshold Error %s

[PatrickVibild]

Hi all.

I have been working with eDO robot for the past 8 months. Now delivering my thesis in the next week i would like to request some help with the code, since I dont know if my robot is malfunctioning or there is an actual error with the controller.

I have notice that the robot randomly every 3-5 min will stop working and throw me an error with
joint_trajectory_action_server: Exceeded Goal Threshold Error edo_joint_X

I have reduced the speed of the robot joints even more, in case that was an issue. joints limits in Moveit package is set up to 0.3.

I have seen the code where this error is been generated:

        def check_goal_state():
            for error in self._get_current_error(joint_names, last.positions):
                if (self._goal_error[error[0]] > 0 and self._goal_error[error[0]] < math.fabs(error[1])):
                    return error[0]
            if (self._stopped_velocity > 0.0 and
                max([abs(cur_vel) for cur_vel in self._get_current_velocities(joint_names)]) > self._stopped_velocity):
                return False
            else:
                return True

        result = check_goal_state()
        if result is True:
            rospy.loginfo("%s: Joint Trajectory Action Succeeded" % (self._action_name,))
            self._result.error_code = self._result.SUCCESSFUL
            self._server.set_succeeded(self._result)
        elif result is False:
            rospy.logerr("%s: Exceeded Max Goal Velocity Threshold" % (self._action_name,))
            self._result.error_code = self._result.GOAL_TOLERANCE_VIOLATED
            self._server.set_aborted(self._result)
        else:
            rospy.logerr("%s: Exceeded Goal Threshold Error %s" % (self._action_name, result,))
            self._result.error_code = self._result.GOAL_TOLERANCE_VIOLATED
            self._server.set_aborted(self._result)

My question is: Have any one else found issues with random brakes on the robot. And if not, can anyone guide me a bit why I am been thrown a Threshold error with the robot?

In the system I am developing I notice that the times provided by ROS are having some issues. I am also wondering the Timestamps that ROS provides can have any impact on Threshold issues?

[ymollard]

I think your robot autocollision triggers (because of the engaged forces, not actual collisions) and then the JTAS reports Goal Threshold Error because so PID cannot follow the requested trajectory (motors have been braked).

There is an ongoing discussion about autocollision here, can you disable it and then test again?

[PatrickVibild]

I think your robot autocollision triggers (because of the engaged forces, not actual collisions) and then the JTAS reports Goal Threshold Error because so PID cannot follow the requested trajectory (motors have been braked).

There is an ongoing discussion about autocollision here, can you disable it and then test again?

Hi,

I tested the proposed solution in that discussion and the robot takes no more than 10-15 trajectories to fail and brake.

[ymollard]

When brakes are released but no motion is being executed, can you try to push the arm with your hand? Brakes should not engage, if they do then you didn't disabled autocollision successfully. Beware, several solutions are proposed in the discussions, some may not work for your robot.

[PatrickVibild]

Things get out of control, the robot braked itself during the calibration of the first joint.

I tested to push the joints with my hands and I can confirm that the auto-collision is disabled, and I get continuous brakes during any trajectory. I am unable to run the robot for more than 5-10 movements without this issue happening.

@ymollard do you have anything else I could check for the robot? presenting the robot in less than 3 weeks, and I would like to have a reliable robot if possible.

[PatrickVibild]

I saw today this next warnings on the robot:

[WARN] [1622557291.080114]: /joint_trajectory_action_server: Executing requested joint trajectory 11.3 sec
....
[ERROR] [1622557302.963336241]: 
Invalid Trajectory: start point deviates from current robot state more than 0.01
joint 'edo_joint_6': expected: 3.74527, current: 3.75976
....
[ERROR] [1622557328.535490]: /joint_trajectory_action_server: Exceeded Goal Threshold Error edo_joint_1
[ WARN] [1622557329.494003703]: Controller  failed with error GOAL_TOLERANCE_VIOLATED: 
[ WARN] [1622557329.494144575]: Controller handle  reports status ABORTED

I can see that the eDO_states compares the trajectorie_points with the current_state and then throws errors if the gap between the state and the trajectorie point is to big. Could the issue be generated from some issues with the timestamps?

[ymollard]

Are you also having such problems when using the Android application only to navigate between 2 viapoints for instance? (No custom ROS started and a fresh reboot).

[PatrickVibild]

I havent used the android app in the past 10 months, but I will give it a try for a while and check if I have random brakes using the Android app.

…

On Mon, 7 Jun 2021 at 19:08, Yoan Mollard ***@***.***> wrote: Are you also having such problems when using the Android application only to navigate between 2 viapoints for instance? (No custom ROS started and a fresh reboot). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKQD6UNMJ3HPOLUJGJXPFIDTRT4J5ANCNFSM45ULT3YQ> .

[PatrickVibild]

Are you also having such problems when using the Android application only to navigate between 2 viapoints for instance? (No custom ROS started and a fresh reboot).

I just tested the app. Just to make sure that everything was working correctly I flashed the robot with the latest version 3.1.1 and installed the Android APP 3.1.1.

I am unable to calibrate the robot, once I unbrake the robot it takes 5 seconds to get automatically braked again.
I have recorded a video of the issue showcasing the issue with the android app: https://www.youtube.com/watch?v=Dw2jdbFtjcU

Something that I notice during the flashing that i get an exception during the process.

comandi tmux:                                                                                                                                                                                                                                                               [49/49]
^B %     dividi verticalmente
^B "     dividi orizzontalmente
^B ^D    detach
rostopic is a command-line tool for printing information about ROS Topics.

Commands:
        rostopic bw     display bandwidth used by topic
        rostopic delay  display delay of topic from timestamp in header
        rostopic echo   print messages to screen
        rostopic find   find topics by type
        rostopic hz     display publishing rate of topic    
        rostopic info   print information about active topic
        rostopic list   list active topics
        rostopic pub    publish data to topic
        rostopic type   print topic or field type

Type rostopic <command> -h for more detailed usage, e.g. 'rostopic echo -h'

edo@raspberrypi:~ $ ./updateFw.sh 
./updateFw.sh: line 46: kill: (14104) - No such process
20190620095157_800
Flashing Bridge USB...
There are no processes keeping the serial line open
speed 57600 baud; line = 0; min = 1; time = 0; -brkint -icrnl -imaxbel -opost -onlcr -isig -icanon -iexten -echo -echoe -echok -echoctl -echoke
20190620095157_959
[INFO] [1561024319.646569]: ROS Serial Python Node
[INFO] [1561024319.679638]: Connecting to /dev/ttyACM0 at 57600 baud
[ERROR] [1561024321.802635]: Tried to publish before configured, topic id 125
[ERROR] [1561024321.807355]: Tried to publish before configured, topic id 125
[INFO] [1561024321.853629]: Note: publish buffer size is 512 bytes
[INFO] [1561024321.856378]: Setup publisher on usb_jnt_state [edo_core_msgs/JointStateArray]
[INFO] [1561024321.871611]: Setup publisher on usb_jnt_version [edo_core_msgs/JointFwVersion]
[INFO] [1561024321.895680]: Note: subscribe buffer size is 512 bytes
[INFO] [1561024321.898445]: Setup subscriber on algo_jnt_ctrl [edo_core_msgs/JointControlArray]
[INFO] [1561024321.928083]: Setup subscriber on machine_jnt_config [edo_core_msgs/JointConfigurationArray]
[INFO] [1561024321.960880]: Setup subscriber on machine_init [edo_core_msgs/JointInit]
[INFO] [1561024322.007413]: Setup subscriber on machine_jnt_calib [edo_core_msgs/JointCalibration]
[INFO] [1561024322.035031]: Setup subscriber on machine_jnt_reset [edo_core_msgs/JointReset]
[INFO] [1561024322.064577]: Setup subscriber on machine_jnt_version [std_msgs/UInt8]
Request to the USB Board to reboot
20190620095209_973
publishing and latching message for 3.0 seconds
Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/rosserial_python/serial_node.py", line 85, in <module>
    client.run()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosserial_python/SerialClient.py", line 503, in run
    self.requestTopics()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosserial_python/SerialClient.py", line 389, in requestTopics
    self.port.flushInput()
  File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 500, in flushInput
    termios.tcflush(self.fd, TERMIOS.TCIFLUSH)
termios.error: (5, 'Input/output error')
[WARN] [1561024331.589900]: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
/home/edo/flashUsbFirmware: line 79: kill: (14146) - No such process
20190620095214_233
There are no processes keeping the serial line open
Request to the USB Board to enter into the ST's bootloader
20190620095215_354
20190620095217_369
dfu-util: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
dfu-util: can't detach
Bridge USB flashed successfully
20190620095229_689

<================== Start Describe output ====================>
88265F10 241664 2008 edo-joint_bo 1 edo_1
DFB95621 241664 2008 edo-joint_bo 2 edo_2
A4649B03 110592 2008 edo-gripper_ 7 edo_7
B645CFB5 241664 2008 edo-joint_bo 3 edo_3
E91FFAB8 241664 2008 edo-joint_bo 4 edo_4
D9B311DA 241664 2008 edo-joint_bo 5 edo_5
92B09DAE 241664 2008 edo-joint_bo 6 edo_6

.....

I will try now with the previous version and check the android application on V3.0.1

[PatrickVibild]

I saw this answer related to the rosserial issue and they claim to be a disconnection form the USB.

Since I have flashed the robot with the V3.1 I am unable to start it up or flash it with another version.

during the script of updateFw.sh I am getting always errors updating the USB bridge.

I have tried to contact Comau some time ago through their forum without any luck. Quite desperate since I am presenting my thesis on 2 weeks and I have a non working robot. Is there a possibility that you could link me with a Comau engineer related with this task? @ymollard

[ymollard]

If even with the app from the manufacturer your robot engages brakes unexpectedly, I advise that you to get it touch with the support via your robot reseller, I don't know anyone at Comau.
The I/O error does not sound good it might be a hardware issue. Neither the DFU signature warning, although it is flashed anyway it looks abnormal to me.