Abnormally Long Pod Deployment Time

[alyhasansakr]

The original issue was raised here. In short, deploying any new pods takes a very long time (up to 30 mins).

The problem was believed to be from Kubernetes and Docker's side, but no combination of versions worked for us.

We normally use KVM for virtualization on our server infrastructure, but we also find the same issue running Ubuntu on bare metal on laptops.

By the way, it used to work well before version 1.6, but somehow it doesn't now, even if we go back to 1.4 (that is why we assume it is a dependency problem).

Any help would be appreciated.

[dilallkx]

Hi @alyhasansakr,

Thank you for reaching out!

To help debug your issue, please provide the following details about your host setup:

• Hardware info:
  • System info: sudo dmidecode | grep -A 9 "System Information"
  • CPU info: lscpu
  • Memory info: free -m
  • Disk technology/model/size:
    • sudo lshw -class disk -class storage
    • sudo nvme list
  • Disk usage: df
• Network info:
  • Network device list: nmcli device status
  • Network card info: lspci | awk '/[Nn]et/ {print $1}' | xargs -i% lspci -ks %
  • Internet connection speed uplink & downlink ( use speedtest.net )
• Software info:
  • OS/kernel version: uname -a
  • Distribution version: lsb_release -a
  • AdvantEDGE versions: meepctl version all
  • Docker version: docker version

If using a VM -- also provide VM info/configuration in addition to host info

Also, please provide the following logs & debug info:

• K8s node info: kubectl describe nodes
• k8s pods info: kubectl get pods --all-namespaces
• Failing pod information: kubectl describe pod <pod-name>
• Failing pod logs (if any)
• Persistent volume disk usage: sudo du -hs ~/.meep/*

NOTE: Please use the latest supported versions according to the AdvantEDGE documentation:
https://interdigitalinc.github.io/AdvantEDGE/docs/setup/env-runtime/

Regards!

[alyhasansakr]

I did some more trials today on another server (Ubuntu 18.04). Here is what I tried:

1- Build with ansible: got this error

2- Use prebuilt binaries: got this error

I also got another error after rebooting

But it seems that I need to turn memory swap off every time I reboot the machine.

3- Build it myself: got this error

This one I was able to go around by installing nvm v12.19.0. I think you should remove this note.
Now I have a version that is working. When I deploy the sandbox it is finished in a few seconds. This is good news, however, I am still not sure about the source of the problem.

Now I have two different machines, one is working correctly and one takes forever to start a sandbox (mentioned in @anjumm comment). There is a lot of differences between the two of them (VM vs bare, 1.7 vs 1.8, ubuntu 20 vs 18, different hardware). I will try to reduce the differences until I find the condition that causes the problem.

[dilallkx]

Hi @anjumm, @alyhasansakr,

Thank you for providing additional information. We took a look and found that your VM hardware CPU & memory meets the recommended minimum HW requirements, however your disk does not. AdvantEDGE uses persistent volumes that map to host filesystem paths for persistent data storage. If the disk size is too small or the disk speed & access is slow this could cause issues similar to what you are observing. We recommend an SSD of 500G. Would it be possible to scale up your VM disk size? The logs did not specify if the underlying disk is an HDD or SSD, could you please confirm the disk type and whether it could be upgraded to SSD?

It would also be interesting to compare with the HW specifications from @alyhasansakr bare metal setup. Could you please provide the same information (HW, SW, etc.) for the bare metal system?

Finally, please provide the following k8s logs when the hanging issue occurs: kubectl describe all --all-namespaces

On my side I have run tests with AdvantEDGE v1.8.1 in a Virtualbox VM running Ubuntu 20.04.4 with a 128G & 64G SSD disk and I was able to install and run AdvantEDGE successfully in both cases. Although I was successful in running on a 64G disk, I would still recommend trying it using a larger & SSD disk on your side. Another consideration that we should compare between setups is the virtual filesystem configuration. I tried with dynamically allocated normal VDI & QED storage (with an underlying SSD disk); is your VM configuration similar?

In response to what @alyhasansakr tried:

1- Ansible installation is a beta feature and know to have some issues on specific setups. It may be worth investigating further for your specific deployments if you plan on installing several instances of AdvantEDGE. Otherwise I recommend the manual installation procedure.

2- Prebuilt images worked well on my setup. If you take the released tarball, make sure you unzip it in the ~/AdvantEDGE folder. You can then run the dockerize command without building first. Note that if you subsequently run the build command it will remove the prebuilt apps and rebuild them. If this build fails (e.g. the frontend build fails) then your bin folder will no longer have the prebuilt apps and the dockerize will fail.

3- Thank you for pointing out the npm issue. The reason for this failure is that I did not set the npm version to use to remain compatible with node v10.16.3. I will update the documentation to install npm version 6.14.8 using the command npm install -g [email protected]. I will leave the note for now because I have not yet confirmed that newer versions will work with demo1.

It is great to hear you have a working setup with the latest AdvantEDGE versions. I agree that we need to compare the working setup with @anjumm VM setup. Let's start by comparing the systems and checking the the disk concerns I mentioned above.

Thanks & Regards!

[anjumm]

Hi @dilallkx, Hope everything is good at your end.

I deployed advantedge on a new virtual machine with the recommended hardware settings (VM Disk Size increased). I used the setup of ./install.sh script and was able to run it properly. However during the installation, NVM and NPM versions caused conflict which I solved with changing nvm to 12,19.0 as @alyhasansakr did. The deployment as suspected did created the issues of long deployment time. For example here:

This docker registry was not ready even for more than one hour. Then after multiple attempts, I was able to deploy it for the first time which took roughly 10mins.

This week, I will look into the scenarios and see if I get any further issues or not.

[roymx]

Hello @anjumm,

When it happens, please perform the following command.
kubectl describe pod <pod-name>

for example, in the above example:
kubectl describe pod meep-docker-registry-65b77797cp-jplvr

In the Events section, you will see that K8s is doing (e.g. what resource the pod is waiting for)
On first deployments, it is normal that it takes a bit longer since the container image is not present locally and has to be downloaded, but 1 hour is abnormal unless you have a super slow network (which is not the case based on previous inputs)

Thank you.

[anjumm]

Hello @roymx I did some further testing on existing advantedge demos and our own scenarios.
In terms of configuring and deploying scenarios, the setup is running smoothly and takes less time than before. The same goes with termination of pods and redeploying the core. I will continue to further investigate if it stays the same way or slow down like the old setup. For now, the issue seems to be resolved but will request your advice in case we face issue again.