cannot delete all docker instances through the cli

[chavafg]

Launched around 50 docker instances and then tried to delete them. 1 of them failed to be deleted, listed instances to be sure it was still there and it appears listed, but in the compute nodes there is no docker instance running.

root@sn-controller ~/bin # for i in $(./ciao-cli -password hello -username csr -identity https://sn-keystone.zpn.intel.com:35357 -tenant 92834d7fd3494ef994e421fdb3eff3a9 -list-instances | grep -A1 'Instance' | awk '/UUID/ {print $2}'); do ./ciao-cli -password hello -username csr -identity https://sn-keystone.zpn.intel.com:35357 -tenant 92834d7fd3494ef994e421fdb3eff3a9 -delete-instance -instance $i; done
Deleted instance: 93b0f44b-253b-492d-92ca-47cea6bdf1d6
Deleted instance: b9957b8c-0489-41fe-99cf-e2da361f3f6f
Deleted instance: 51c8ab26-f8e7-48ae-a5e8-ba210aa391d1
....
Deleted instance: 10e096de-28f7-4f7b-a1e4-ac550818dff6
Deleted instance: d2727311-66db-4f35-8025-9a3d11b6d018

• failure

F0415 21:12:32.270183 1798 ciao-cli.go:109] ciao-cli FATAL: HTTP Error [500] for [DELETE https://localhost:8774/v2.1/92834d7fd3494ef994e421fdb3eff3a9/servers/6eef8f8d-a6a8-48d0-9e93-2701c2a49ac4]: Instance not available
goroutine 1 [running]:
github.com/golang/glog.stacks(0xa64a00, 0x0, 0x0, 0x0)
/home/fuentess/go/src/github.com/golang/glog/glog.go:769 +0xb8
github.com/golang/glog.(_loggingT).output(0xa487c0, 0xc800000003, 0xc82000a0c0, 0xa2a264, 0xb, 0x6d, 0x0)
/home/fuentess/go/src/github.com/golang/glog/glog.go:720 +0x2ce
github.com/golang/glog.(_loggingT).printf(0xa487c0, 0x3, 0xc820112d10, 0xb0, 0x0, 0x0, 0x0)
/home/fuentess/go/src/github.com/golang/glog/glog.go:655 +0x1d4
github.com/golang/glog.Fatalf(0xc820112d10, 0xb0, 0x0, 0x0, 0x0)
/home/fuentess/go/src/github.com/golang/glog/glog.go:1148 +0x5d
main.fatalf(0xc820104b40, 0xa0, 0x0, 0x0, 0x0)
/home/fuentess/go/src/github.com/01org/ciao/ciao-cli/ciao-cli.go:109 +0x89
main.deleteTenantInstance(0x7ffffff4372f, 0x20, 0x7ffffff4376b, 0x24)
/home/fuentess/go/src/github.com/01org/ciao/ciao-cli/ciao-cli.go:625 +0x27b
main.main()
/home/fuentess/go/src/github.com/01org/ciao/ciao-cli/ciao-cli.go:907 +0x7c9
Deleted instance: 715c7858-0f4b-451c-8cff-d903b16b4f8d
Deleted instance: c3ff8985-e9c5-4912-9331-ceda5740404b
Deleted instance: 74c84ae2-c81c-4e6a-a592-263b7fc9f1ad

• Instance still shown

root@sn-controller ~/bin # ./ciao-cli -password hello -username csr -identity https://sn-keystone.zpn.intel.com:35357 -tenant 92834d7fd3494ef994e421fdb3eff3a9 -list-instances
Instance #1
UUID: 4d4ff4f3-c3fc-41b0-950f-173362273f37
Status: running
Private IP: 172.16.0.15
MAC Address: 02:00:ac:10:00:0f
CN UUID: 3534b9eb-b2ad-47b5-9248-4f329aa59f41
Image UUID: fa7d86d8-fa46-11e5-8493-38607786d9ec
Tenant UUID: 92834d7fd3494ef994e421fdb3eff3a9
SSH IP: 192.168.10.169
SSH Port: 33015

• tried to delete again

root@sn-controller ~/bin # ./ciao-cli -password hello -username csr -identity https://sn-keystone.zpn.intel.com:35357 -tenant 92834d7fd3494ef994e421fdb3eff3a9 -delete-instance -instance 4d4ff4f3-c3fc-41b0-950f-173362273f37
F0415 21:14:24.212779 1880 ciao-cli.go:109] ciao-cli FATAL: HTTP Error [500] for [DELETE https://localhost:8774/v2.1/92834d7fd3494ef994e421fdb3eff3a9/servers/4d4ff4f3-c3fc-41b0-950f-173362273f37]: Instance not available
goroutine 1 [running]:
github.com/golang/glog.stacks(0xa64a00, 0x0, 0x0, 0x0)
/home/fuentess/go/src/github.com/golang/glog/glog.go:769 +0xb8
github.com/golang/glog.(_loggingT).output(0xa487c0, 0xc800000003, 0xc820224180, 0xa2a264, 0xb, 0x6d, 0x0)
/home/fuentess/go/src/github.com/golang/glog/glog.go:720 +0x2ce
github.com/golang/glog.(_loggingT).printf(0xa487c0, 0x3, 0xc8202086e0, 0xb0, 0x0, 0x0, 0x0)
/home/fuentess/go/src/github.com/golang/glog/glog.go:655 +0x1d4
github.com/golang/glog.Fatalf(0xc8202086e0, 0xb0, 0x0, 0x0, 0x0)
/home/fuentess/go/src/github.com/golang/glog/glog.go:1148 +0x5d
main.fatalf(0xc8202021e0, 0xa0, 0x0, 0x0, 0x0)
/home/fuentess/go/src/github.com/01org/ciao/ciao-cli/ciao-cli.go:109 +0x89
main.deleteTenantInstance(0x7ffe45f5b72f, 0x20, 0x7ffe45f5b76b, 0x24)
/home/fuentess/go/src/github.com/01org/ciao/ciao-cli/ciao-cli.go:625 +0x27b
main.main()
/home/fuentess/go/src/github.com/01org/ciao/ciao-cli/ciao-cli.go:907 +0x7c9

• on compute nodes, there is no docker instance running:

root@sn-compute01 /var/lib/ciao/logs/launcher # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

root@sn-compute02 /var/lib/ciao/logs/launcher # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@sn-compute02 /var/lib/ciao/logs/launcher #

[sameo]

Could you please try to reproduce with -delete-instance -all-instances ?

[rbradford]

Closing this bug as no response from reporter on this old issue and no evidence that this is a valid issue any longer.