Got ERROR when running test for rocketmq #55

rhelbrent · 2022-05-13T07:26:08Z

rhelbrent
May 13, 2022

I'm trying to set up openchaos testing for rocketmq environment, after running command

chaos.sh --driver driver-rocketmq/rocketmq.yaml --install

at OS level, or running command

bin/chaos.sh --driver driver-rocketmq/rocketmq.yaml --install --restart

at container level, always got ERRORs:

"Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*"
"Queue model setupCluster fail"

Found detail logs in ~/logs/openchaos.log:

[root@client1 logs]# cat openchaos.log 
2022-05-12 17:10:04.001 [main] INFO - --------------- CHAOS TEST --- DRIVER : RocketMQ---------------
2022-05-12 17:10:04.033 [main] INFO - Initial driver: {
  "name" : "RocketMQ",
  "driverClass" : "io.openchaos.driver.rocketmq.RocketMQDriver",
  "nodes" : [ "rocketmq1", "rocketmq2" ],
  "metaNodes" : [ "rocketmq1", "rocketmq2" ],
  "isOrderTest" : false,
  "endToEndLatencyCheck" : true,
  "pull" : false,
  "metaNodesParticipateInFault" : false,
  "isUploadImage" : false,
  "ossEndPoint" : null,
  "ossAccessKeyId" : null,
  "ossAccessKeySecret" : null,
  "bucketName" : null
}
2022-05-12 17:10:04.048 [main] INFO - Node rocketmq1 download rocketmq for nameserver ...
2022-05-12 17:10:31.312 [main] ERROR - Node rocketmq1 setup nameserver node failed
java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.rmi.ServerException: 1
	... 8 more
2022-05-12 17:10:31.323 [main] ERROR - Queue model setupCluster fail
java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:62) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	... 4 more
Caused by: java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	... 4 more
2022-05-12 17:10:31.323 [main] ERROR - Failed to ready chaos test.
java.lang.RuntimeException: java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:164) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:62) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
Caused by: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
Caused by: java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
2022-05-12 17:10:31.325 [main] INFO - Teardown client
2022-05-12 17:10:31.328 [main] ERROR - Failed to run chaos test.
java.lang.NullPointerException: null
	at io.openchaos.driver.rocketmq.RocketMQDriver.shutdown(RocketMQDriver.java:185) ~[ChaosControl.jar:?]
	at io.openchaos.model.QueueModel.shutdown(QueueModel.java:223) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.clearAfterException(ChaosControl.java:483) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:371) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
2022-05-12 17:10:31.328 [main] INFO - Teardown client
2022-05-12 17:10:31.328 [main] INFO - Test stop
2022-05-12 17:10:31.331 [main] INFO - Teardown client
2022-05-12 17:11:05.095 [Thread-1] INFO - Teardown client
2022-05-12 17:36:36.766 [main] INFO - --------------- CHAOS TEST --- DRIVER : RocketMQ---------------
2022-05-12 17:36:36.795 [main] INFO - Initial driver: {
  "name" : "RocketMQ",
  "driverClass" : "io.openchaos.driver.rocketmq.RocketMQDriver",
  "nodes" : [ "rocketmq1", "rocketmq2" ],
  "metaNodes" : [ "rocketmq1", "rocketmq2" ],
  "isOrderTest" : false,
  "endToEndLatencyCheck" : true,
  "pull" : false,
  "metaNodesParticipateInFault" : false,
  "isUploadImage" : false,
  "ossEndPoint" : null,
  "ossAccessKeyId" : null,
  "ossAccessKeySecret" : null,
  "bucketName" : null
}
2022-05-12 17:36:36.810 [main] INFO - Node rocketmq1 download rocketmq for nameserver ...
2022-05-12 17:36:43.532 [main] ERROR - Node rocketmq1 setup nameserver node failed
java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.rmi.ServerException: 1
	... 8 more
2022-05-12 17:36:43.537 [main] ERROR - Queue model setupCluster fail
java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:62) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	... 4 more
Caused by: java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	... 4 more
2022-05-12 17:36:43.538 [main] ERROR - Failed to ready chaos test.
java.lang.RuntimeException: java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:164) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:62) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
Caused by: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
Caused by: java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
2022-05-12 17:36:43.539 [main] INFO - Teardown client
2022-05-12 17:36:43.541 [main] ERROR - Failed to run chaos test.
java.lang.NullPointerException: null
	at io.openchaos.driver.rocketmq.RocketMQDriver.shutdown(RocketMQDriver.java:185) ~[ChaosControl.jar:?]
	at io.openchaos.model.QueueModel.shutdown(QueueModel.java:223) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.clearAfterException(ChaosControl.java:483) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:371) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
2022-05-12 17:36:43.542 [main] INFO - Teardown client
2022-05-12 17:36:43.542 [main] INFO - Test stop
2022-05-12 17:36:43.544 [main] INFO - Teardown client

rhelbrent · 2022-05-13T07:35:07Z

rhelbrent
May 13, 2022
Author

Try to change "installDir: rocketmq-chaos-test" in rocketmq.yaml, same errors reported again.
The authentication from control node to cluster node is correct, I can find directory /root/rocketmq-chaos-test created.

[root@rocketmq1 rocketmq-4.9.3]# pwd
/root/rocketmq-chaos-test/rocketmq-4.9.3
[root@rocketmq1 rocketmq-4.9.3]# ls -l
total 40
drwxr-xr-x 2 root root   126 Feb 22 01:25 benchmark
drwxr-xr-x 3 root root  4096 Feb 22 00:25 bin
drwxr-xr-x 7 root root   201 Feb 22 00:25 conf
drwxr-xr-x 2 root root  4096 Feb 22 01:25 lib
-rw-r--r-- 1 root root 17327 Feb 22 00:25 LICENSE
-rw-r--r-- 1 root root  1338 Feb 22 00:25 NOTICE
-rw-r--r-- 1 root root  6069 Feb 22 00:25 README.md

BTW, I've used "installDir: /opt/rocketmq-4.9.3/" in rocketmq.yaml, which is the RocketMQ home directory in node rocketmq1, after running chaos.sh, it removed all original files and directories then create its own without any operation pre check, I think it is a very significant defects and will cause customer's data loss!

0 replies

rhelbrent · 2022-05-13T07:37:46Z

rhelbrent
May 13, 2022
Author

Content of "rocketmq.yaml"

[root@client1 driver-rocketmq]# cat /opt/openchaos-openchaos-0.6.4/driver-rocketmq/rocketmq.yaml
name: RocketMQ
driverClass: io.openchaos.driver.rocketmq.RocketMQDriver
endToEndLatencyCheck: true

nodes:
- rocketmq1
- rocketmq2

metaNodes:
- rocketmq1
- rocketmq2

rocketmqVersion: 4.9.3
installDir: rocketmq-chaos-test
nameServerPort: 9876

clusterName: DefaultCluster # same value as the config in broker.properties
vipChannelEnabled: false

brokerClusterName: DefaultCluster
brokerName: c1-m1
storePathRootDir: /store/c1-m1
storePathCommitLog: /commitlog/c1-m1
enableDLegerCommitLog: true
dLegerGroup: c1-m1
sendMessageThreadPoolNums: 32

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Got ERROR when running test for rocketmq #55

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Got ERROR when running test for rocketmq #55

rhelbrent May 13, 2022

Replies: 2 comments

rhelbrent May 13, 2022 Author

rhelbrent May 13, 2022 Author

rhelbrent
May 13, 2022

rhelbrent
May 13, 2022
Author

rhelbrent
May 13, 2022
Author