A problem with a demo #111

martynia · 2023-09-27T13:13:33Z

martynia
Sep 27, 2023

Hi,

I follow the instruction and got stuck with a demo:
The output is below:

What could be a reason for this ?

cheers, JM

🦄 Installing DiracX...
W0927 13:03:44.631471 16664 warnings.go:70] unknown field "spec.template.spec.volumes[0].sizeLimit"
W0927 13:03:44.631573 16664 warnings.go:70] unknown field "spec.template.spec.volumes[2].empty-dir"
W0927 13:03:44.631593 16664 warnings.go:70] unknown field "spec.template.spec.volumes[2].sizeLimit"
W0927 13:03:44.631608 16664 warnings.go:70] unknown field "spec.template.spec.volumes[4].sizeLimit"
NAME: diracx-demo
LAST DEPLOYED: Wed Sep 27 13:03:35 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:

Get the application URL by running these commands:
http:///
🦄 Waiting for installation to finish...
error: timed out waiting for the condition on pods/diracx-demo-cd88485cd-bwhmc
⚠️ Installation did not start sucessfully!

ℹ️ Press Ctrl+C to clean up and exit

chrisburr · 2023-09-28T07:47:45Z

chrisburr
Sep 28, 2023
Maintainer

You'll probably need to poke around with kubectl to figure out what's going on. See the README for a starting point:

https://github.com/DIRACGrid/diracx-charts#interacting-with-the-demo

At the moment the demo doesn't print the variables you need to set if something fails but if you're in the chart's repo they will be:

export KUBECONFIG="$PWD/.demo/kube.conf"
export HELM_DATA_HOME="$PWD/.demo/helm_data"
export PATH=${PATH}:$PWD/.demo

0 replies

martynia · 2023-09-29T09:42:42Z

martynia
Sep 29, 2023
Author

1 reply

chrisburr Oct 2, 2023
Maintainer

The errors above were caused by an incorrect version (apparently) of docker.

Why do you think this? If that is the case we should test the docker version put a more helpful error message.

The fact that I needed sudo was a bit worrying (docker is configured in such a way that docker commands don't require sudo).

There is no need to use sudo and the script has been working without it on Ubuntu/Fedora/Arch/macOS.

Since everything is cleaned up no much can be done with kubectl. Nothing is running in docker. Timeout (highlighted above) seems to be a problem.

Your original run of the demo seems to have been much more successful. I'd suggest trying to get back to that state and sharing the output of running kubectl get pods after set the three environment variables I mentioned. (It should now print them to the terminal if you update the repositories.

martynia · 2023-10-03T11:50:17Z

martynia
Oct 3, 2023
Author

OK,
I realised that the installation of Kubernetes was a mistake. I went the original route:

sudo: this is caused by wrong permission in the cloned repository, i.e.
[rocky@diracx-r9 diracx-charts]$ ls -l /home/rocky/diracx-charts/.demo/demo_cluster_conf.yaml
-rw-r--r--. 1 root root 1040 Oct 3 10:16 /home/rocky/diracx-charts/.demo/demo_cluster_conf.yaml

(and I did not clone with sudo ...)

So, so far but only installing docker I'm reaching the same stage as in my first attempt:

`validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
🦄 Waiting for ingress controller to be created...
pod/ingress-nginx-controller-77f58bf6cd-x6lqp condition met
🦄 Installing DiracX...
W1003 10:21:26.735598 114666 warnings.go:70] unknown field "spec.template.spec.volumes[0].sizeLimit"
W1003 10:21:26.735686 114666 warnings.go:70] unknown field "spec.template.spec.volumes[2].empty-dir"
W1003 10:21:26.735711 114666 warnings.go:70] unknown field "spec.template.spec.volumes[2].sizeLimit"
W1003 10:21:26.735723 114666 warnings.go:70] unknown field "spec.template.spec.volumes[4].sizeLimit"
NAME: diracx-demo
LAST DEPLOYED: Tue Oct 3 10:21:17 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:

Get the application URL by running these commands:
http:///
🦄 Waiting for installation to finish...
error: timed out waiting for the condition on pods/diracx-demo-f4b4dbf55-w62mf
⚠️ Installation did not start sucessfully!:`

From another terminal window I can explore the installation:
I've set the env. vars and:

(sorry for that sudo, it makes things look awful)

cluster info

[rocky@diracx-r9 diracx-charts]$ sudo /home/rocky/diracx-charts/.demo/kubectl cluster-info  --context kind-diracx-demo --kubeconfig /home/rocky/diracx-charts/.demo/kube.conf
Kubernetes control plane is running at https://127.0.0.1:43689
CoreDNS is running at https://127.0.0.1:43689/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

(I have done the dump and looked for some problematic places, but it is quite a lot of it, not sure what to search for.

Although there is a hint when describing pods, see below)

pods
sudo /home/rocky/diracx-charts/.demo/kubectl get pods --context kind-diracx-demo --kubeconfig /home/rocky/diracx-charts/.demo/kube.conf NAME READY STATUS RESTARTS AGE diracx-demo-dex-64856f8d5b-485cr 1/1 Running 0 64m diracx-demo-f4b4dbf55-w62mf 0/1 Init:CrashLoopBackOff 14 (4m37s ago) 64m diracx-demo-minio-5d87f67c7d-dl58t 1/1 Running 0 64m diracx-demo-mysql-0 0/1 Running 16 (90s ago) 64m diracx-demo-rabbitmq-0 1/1 Running 0 64m opensearch-cluster-master-0 1/1 Running 0 64m

Describe a problematic pod

sudo /home/rocky/diracx-charts/.demo/kubectl describe pod/diracx-demo-f4b4dbf55-w62mf --context kind-diracx-demo --kubeconfig /home/rocky/diracx-charts/.demo/kube.conf

(tail of the listing)

Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Normal   Pulling  59m                    kubelet  Pulling image "ghcr.io/diracgrid/diracx/server:latest"
  Normal   Pulled   59m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 1.204049421s (13.839721729s including waiting)
  Normal   Created  59m                    kubelet  Created container init-cs-user
  Normal   Started  59m                    kubelet  Started container init-cs-user
  Normal   Pulling  58m                    kubelet  Pulling image "ghcr.io/diracgrid/diracx/server:latest"
  Normal   Created  58m                    kubelet  Created container init-signing-key
  Normal   Pulled   58m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 777.47691ms (777.533585ms including waiting)
  Normal   Started  58m                    kubelet  Started container init-signing-key
  Normal   Pulled   58m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 766.966703ms (766.984454ms including waiting)
  Normal   Pulled   57m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 975.937695ms (975.99232ms including waiting)
  Normal   Created  57m (x3 over 58m)      kubelet  Created container create-sql-db-schema
  Normal   Pulled   57m                    kubelet  Successfully pulled image "ghcr.io/diracgrid/diracx/server:latest" in 743.452172ms (743.522113ms including waiting)
  Normal   Started  57m (x3 over 58m)      kubelet  Started container create-sql-db-schema
  Normal   Pulling  39m (x9 over 58m)      kubelet  Pulling image "ghcr.io/diracgrid/diracx/server:latest"
  Warning  BackOff  4m24s (x220 over 57m)  kubelet  Back-off restarting failed container create-sql-db-schema in pod diracx-demo-f4b4dbf55-w62mf_default(9be18492-c677-4fba-8c69-eec98a347548)

Is it something wrong with the DB schema ?

cheers, JM

1 reply

chrisburr Oct 6, 2023
Maintainer

From that point you can do kubectl logs diracx-demo-f4b4dbf55-w62mf_default -c create-sql-db-schema to see the logs and hopefully get some idea about the problem.

martynia · 2023-10-03T13:14:55Z

martynia
Oct 3, 2023
Author

I went further w/o sudo, still a timeout, though.:

`ℹ️ To interact with the cluster directly using kubectl/helm you'll need to set the
first following environment variables:

export KUBECONFIG=/home/rocky/diracx-charts/.demo/kube.conf
export HELM_DATA_HOME=/home/rocky/diracx-charts/.demo/helm_data
export PATH=${PATH}:/home/rocky/diracx-charts/.demo

Then see the chart README for more information on how to use kubectl/helm.

ℹ️ To use the demo with the DiracX client set:

export DIRACX_URL=http://172.18.0.1:8000

ℹ️ To access the interactive API documentation visit:

http://172.18.0.1:8000/docs

ℹ️ When prompted to login the credentials are:

Username: [email protected]
Password: password
🦄 Waiting for installation to finish...

error: timed out waiting for the condition on pods/diracx-demo-866c9cf6f-cvpv6
⚠️ Installation did not start sucessfully!`

Still timeout !

2 replies

chrisburr Oct 6, 2023
Maintainer

The path for debugging this is the same:

Find which pod is crashing (kubectl get pods)
Look at the logs (kubectl logs $POD_NAME)
If you don't get anything from 2 because the main container hasn't started, do kubectl logs $POD_NAME -c $CONTAINER_NAME (the available container names are printed when running kubectl logs $POD_NAME).

chrisburr Oct 6, 2023
Maintainer

(Adding a centralised way of collecting the logs is being worked on, but if it's a problem starting up you still might need to fall back on looking at the pods explicitly.)

martynia · 2023-10-03T13:15:30Z

martynia
Oct 3, 2023
Author

Running on my Mac:

ℹ️ When prompted to login the credentials are:

Username: [email protected]
Password: password
🦄 Waiting for installation to finish...
pod/diracx-demo-79ddcb598c-9c8fx condition met
🎉 🎉 🎉 Pods are ready! 🎉 🎉 🎉

2 replies

martynia Oct 3, 2023
Author

I looked around http://192.168.1.21:8000/docs

Will a client work on my Mac ?

chrisburr Oct 6, 2023
Maintainer

Will a client work on my Mac ?

Yes, though be aware that everything is rapidly evolving and still very incomplete. We're still more focused on the foundations than any actual functionality.

martynia · 2023-11-13T15:40:54Z

martynia
Nov 13, 2023
Author

I tried a demo again today. The installation stops at (Centos 7):

`networkpolicy.networking.k8s.io/ingress-nginx-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
🦄 Waiting for ingress controller to be created...
⚠️ Cluster is low on space (35GB free, 13%)
pod/ingress-nginx-controller-d9b6d7bf5-sxfsb condition met
🦄 Installing DiracX...
⚠️ Cluster is low on space (35GB free, 14%)
⚠️ Cluster is low on space (34GB free, 16%)
⚠️ Cluster is low on space (33GB free, 19%)
⚠️ Cluster is low on space (33GB free, 19%)
W1113 12:39:47.413773 109637 warnings.go:70] path /.well-known cannot be used with pathType Prefix
⚠️ Cluster is low on space (33GB free, 20%)
⚠️ Cluster is low on space (31GB free, 24%)
⚠️ Cluster is low on space (31GB free, 25%)
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
* timed out waiting for the condition

⚠️ Error using helm DiracX

ℹ️ Press Ctrl+C to clean up and exit
⚠️ Cluster is low on space (31GB free, 25%)
⚠️ Cluster is low on space (31GB free, 25%)
⚠️ Cluster is low on space (31GB free, 25%)`

The "low on space" message is new, but the installation still stops with an error. Do I need any ports open or any other special settings on the node? (I doubt it since it works on my Mac w/o any adjustments)

1 reply

chaen Nov 14, 2023
Maintainer

you do not need anything special. Just re-run the script. Sometime it timesout (as per the message) for slowish network

martynia · 2023-11-16T12:53:24Z

martynia
Nov 16, 2023
Author

I tried to reinstall the demo, the problem persists..

1 reply

martynia Nov 17, 2023
Author

It installs on ubuntu 22.04.3 LTS . This was an OS problem, not the network. I tried an older Ubuntu and Centos7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A problem with a demo #111

{{title}}

Replies: 7 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

A problem with a demo #111

martynia Sep 27, 2023

Replies: 7 comments · 8 replies

chrisburr Sep 28, 2023 Maintainer

martynia Sep 29, 2023 Author

chrisburr Oct 2, 2023 Maintainer

martynia Oct 3, 2023 Author

chrisburr Oct 6, 2023 Maintainer

martynia Oct 3, 2023 Author

chrisburr Oct 6, 2023 Maintainer

chrisburr Oct 6, 2023 Maintainer

martynia Oct 3, 2023 Author

martynia Oct 3, 2023 Author

chrisburr Oct 6, 2023 Maintainer

martynia Nov 13, 2023 Author

chaen Nov 14, 2023 Maintainer

martynia Nov 16, 2023 Author

martynia Nov 17, 2023 Author

martynia
Sep 27, 2023

Replies: 7 comments 8 replies

chrisburr
Sep 28, 2023
Maintainer

martynia
Sep 29, 2023
Author

chrisburr Oct 2, 2023
Maintainer

martynia
Oct 3, 2023
Author

chrisburr Oct 6, 2023
Maintainer

martynia
Oct 3, 2023
Author

chrisburr Oct 6, 2023
Maintainer

chrisburr Oct 6, 2023
Maintainer

martynia
Oct 3, 2023
Author

martynia Oct 3, 2023
Author

chrisburr Oct 6, 2023
Maintainer

martynia
Nov 13, 2023
Author

chaen Nov 14, 2023
Maintainer

martynia
Nov 16, 2023
Author

martynia Nov 17, 2023
Author