No data available in network traffic for first scenario

[nordanz]

Hello again,

Still I try to make AdvantEdge works in my PC without success. This time I reinstalled everything exactly as it's mentioned in the wiki page(same version for runtime and development), the dashboard is up with a green light and sandbox creation is successful as well.
Next I tried to create and deploy the first scenario as explained in here. Everything seems to work correctly and the pods are up and logs look good. for example output of iperf-cloud-client pod log:

------------------------------------------------------------
Client connecting to 10.100.123.169, UDP port 80
Sending 1470 byte datagrams, IPG target: 235.20 us (kalman adjust)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 10.32.0.35 port 51391 connected with 10.100.123.169 port 80

and exposed service for iperf-cloud-server is: iperf-cloud-server ClusterIP 10.100.123.169
and logs :

Server listening on UDP port 80
Receiving 1470 byte datagrams
UDP buffer size:  208 KByte (default)

[  3] local 10.32.0.33 port 80 connected with 10.32.0.35 port 51391
[  4] local 10.32.0.33 port 80 connected with 10.32.0.36 port 50075

When I go to monitoring field to check the traffic, Network Traffic Aggregation and Point to point only show N/A data in all charts (latency and etc.. )
What can be wrong? I checked all the pods created by this demo and demo3 and meep-tc-sidecar container is not running. I assumed this is the main container collecting and filtering network traffic? Can you explain the relation btw meep-tc-engine and meep-tc-sidecar? I should say the image is build successfully and i can see the meep-tc-sidecar image in my docker images.

The demo1 as well still cannot be deployed and I get same error in meep-virt-engine pod
{"level":"error","meep.component":"meep-virt-engine","meep.from":"install.go:83","msg":"Failed to install Release [meep-demo1-ue2-svc] at /charts/edge-apps/scenario/demo1/ue2-svc","time":"2022-03-14T12:40:27Z"} {"level":"error","meep.component":"meep-virt-engine","meep.from":"install.go:84","msg":"Error(exit status 1): Error: Deployment.apps \"ue2-svc\" is invalid: spec.template.spec.containers: Required value\n","time":"2022-03-14T12:40:27Z"}

Additional info and logs:

the AdvantEdge is the master repository: f203f7a

kubectl version info
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:26:42Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

meepctl version all
Using repo config file: /home/dan/AdvantEDGE/.meepctl-repocfg.yaml Using meepctl config file: /home/dan/.meepctl.yaml { "name": "meepctl", "version": "1.8.1" } { "name": ".meepctl-repocfg.yaml", "version": "1.8.1" } { "name": "meep-cert-manager", "version": "NA" } { "name": "couchdb", "version": "couchdb:3.1.0", "id": "b604d056d8024f10346eab768de7aea06bc0a1b7c55d6087e1b1cd4328c8061c" } { "name": "docker-registry", "version": "registry:2.7.1", "id": "169211e20e2f2d5d115674681eb79d21a217b296b43374b8e39f97fcf866b375" } { "name": "grafana", "version": "grafana/grafana:7.3.5", "id": "511bc20bfcd1b79f3947bb1c33d152f7484e7a91418883fb4dddf71274227321" } { "name": "meep-influxdb", "version": "influxdb:1.8.0-alpine", "id": "5eca9dfe9930a3325323cef801827eb1b0940070465f8f215447b8e732c72b34" } { "name": "meep-ingress", "version": "NA" } { "name": "kube-state-metrics", "version": "quay.io/coreos/kube-state-metrics:v1.9.7", "id": "2f82f0da199c60a7699c43c63a295c44e673242de0b7ee1b17c2d5a23bec34cb" } { "name": "meep-minio", "version": "NA" } { "name": "meep-open-map-tiles", "version": "NA" } { "name": "meep-postgis", "version": "postgis/postgis:12-3.0", "id": "71acda16357f2973034483a4a8363cc9499061120b592bcc3b7f2fbed82da621" } { "name": "meep-prometheus", "version": "NA" } { "name": "meep-redis", "version": "NA" } { "name": "meep-thanos", "version": "NA" } { "name": "meep-thanos-archive", "version": "NA" } { "name": "helm", "version": "v3.3.4", "id": "a61ce5633af99708171414353ed49547cf05013d" } { "name": "docker client", "version": "20.10.7", "id": "f0df350" } { "name": "docker server", "id": "go1.13.15" } { "name": "k8s client", "version": "v1.19.1", "id": "206bcadf021e76c27513500ca24182692aabd17e" } { "name": "k8s server", "version": "v1.19.16", "id": "e37e4ab4cc8dcda84f1344dda47a97bb1927d074" } { "name": "weave", "version": "ghcr.io/weaveworks/launcher/weave-kube:2.8.1", "id": "d797338e7beb17222e10757b71400d8471bdbd9be13b5da38ce2ebf597fb4e63" } { "name": "meep-auth-svc", "version": "meep-docker-registry:30001/meep-auth-svc:latest", "id": "be76c3409fb56138715842ff2b4f1cd4cdc31d6f8dce20248a9d9d1c2ee51926", "build": "sha256sum:" } { "name": "meep-ingress-certs", "version": "NA" } { "name": "meep-mon-engine", "version": "meep-docker-registry:30001/meep-mon-engine:latest", "id": "e5461c6d97743e481f7806cd54edcb55e85ca66526e1f604ee96798d4c93d49b", "build": "sha256sum:" } { "name": "meep-platform-ctrl", "version": "meep-docker-registry:30001/meep-platform-ctrl:latest", "id": "a8942219c644c93b1269a65c5e2c8d3f407e6de2ef911147960ee5674f4aed7c", "build": "sha256sum:" } { "name": "meep-virt-engine", "version": "meep-docker-registry:30001/meep-virt-engine:latest", "id": "b4b8ec905550de4d357cd984a8c0d6321049366d632235320a33ae6128426547", "build": "34d7cd836a63366b6883638c5c7cd155a00279f0622884fc69e365fef2e4d85c" } { "name": "meep-webhook", "version": "meep-docker-registry:30001/meep-webhook:latest", "id": "0d1e82f5a5e16c6c369321f7204f9666781fb5026eb86a26a8d8719a8888bf1f", "build": "1a89ba61fb8923351aeee07f10781c13d98f81e499511b0949754b565a5cda6d" }

[dilallkx]

For reference, this discussion was initially raised here.

[dilallkx]

Hi @nordanz,

Sorry for the delayed response, and thank you for providing more details on your testing & observed issues!

Based on your observations, it seems the tc-sidecar never gets injected in scenario pods on your setup, both for internal or external applications. Scenarios with external nodes (e.g. demo1) with fail during deployment because external pods only contain the tc-sidecar container, whereas scenarios with internal applications only will deploy successfully (e.g. my-first-scenario), however if the tc-sidecar is not injected none of the network characteristics will be applied. We need to determine why the tc-sidecar is not being injected.

To answer your questions, the TC Engine reads the network characteristics from the deployed scenario, calculates the required latency, jitter, packet loss & throughput values to be applied, and finally informs the TC sidecar containers about which application specific rules must be applied. The TC sidecars then enforce the required network characteristics by setting traffic control rules in each pod, and finally collect traffic metrics.

To debug this issue, could you please deploy my-first-scenario and provide the pod details for the iperf cloud server to confirm that the tc-sidecar was never injected and to see if any errors were logged:
kubectl describe pod -n <sandbox name> <iperf-cloud-server pod name>

Also, please provide the meep-virt-engine & meep-webhook logs starting just before scenario deployment until the scenario pods are up & running.

Thanks!

[nordanz]

Hello Kevin,

Thanks for prompt reply, this is the iperf-cloud-server pod logs

Name:         iperf-cloud-server-748ff8fbf5-9jqd6
Namespace:    myapps
Priority:     0
Node:         fe2z0380/10.14.56.42
Start Time:   Mon, 14 Mar 2022 15:34:43 +0100
Labels:       meepAppId=a90fadca-17bf-44c1-a848-e6245d41d5b0
              meepSvc=iperf-cloud-server
              pod-template-hash=748ff8fbf5
              release=meep-first-scenario-iperf-cloud-server
Annotations:  <none>
Status:       Running
IP:           10.32.0.33
IPs:
  IP:           10.32.0.33
Controlled By:  ReplicaSet/iperf-cloud-server-748ff8fbf5
Containers:
  iperf-cloud-server:
    Container ID:  docker://2f1b4b8b1e7508bbb2e2f75904b2dea9537b0cb74b421eea54c4c7b6aab94422
    Image:         gophernet/iperf-server
    Image ID:      docker-pullable://gophernet/iperf-server@sha256:85e7799eb3482216568f654048f8402036037e1ffdc42c38443ddec227f92db9
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
    Args:
      -c
      iperf -u -s -p $IPERF_CLOUD_SERVER_SERVICE_PORT
    State:          Running
      Started:      Mon, 14 Mar 2022 15:34:47 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-bzf4s (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-bzf4s:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-bzf4s
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  73s                      Successfully assigned myapps/iperf-cloud-server-748ff8fbf5-9jqd6 to fe2z0380
  Normal  Pulling    73s   kubelet, fe2z0380  Pulling image "gophernet/iperf-server"
  Normal  Pulled     70s   kubelet, fe2z0380  Successfully pulled image "gophernet/iperf-server" in 2.025697934s
  Normal  Created    70s   kubelet, fe2z0380  Created container iperf-cloud-server
  Normal  Started    70s   kubelet, fe2z0380  Started container iperf-cloud-server

and here meep-virt-engine logs

{"level":"debug","meep.component":"meep-virt-engine","meep.from":"virt-engine.go:187","msg":"RX MSG: Message[SCENARIO-ACTIVATE] Src[myapps:meep-sandbox-ctrl] Dst[all:all] Scope[mq:global] Payload[map[sandbox-name:myapps scenario-name:first-scenario]]","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:319","msg":"Processing virt-engine chart for element[iperf-cloud-server]","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:655","msg":"command /bin/bash","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:544","msg":"Creation of the output chart path: /charts/myapps/scenario/first-scenario/iperf-cloud-server","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:457","msg":"chart added 1","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:319","msg":"Processing virt-engine chart for element[iperf-fog-server]","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:655","msg":"command /bin/bash","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:544","msg":"Creation of the output chart path: /charts/myapps/scenario/first-scenario/iperf-fog-server","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:457","msg":"chart added 2","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:319","msg":"Processing virt-engine chart for element[iperf-cloud-client]","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:655","msg":"command /bin/bash","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:544","msg":"Creation of the output chart path: /charts/myapps/scenario/first-scenario/iperf-cloud-client","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:457","msg":"chart added 3","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:319","msg":"Processing virt-engine chart for element[ue2-term]","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:655","msg":"command /bin/bash","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:544","msg":"Creation of the output chart path: /charts/myapps/scenario/first-scenario/ue2-term","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:457","msg":"chart added 4","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"chart-template.go:145","msg":"Created 4 scenario charts","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"worker.go:49","msg":"Installing 4 Charts...","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"install.go:62","msg":"Installing chart: meep-first-scenario-iperf-cloud-server","time":"2022-03-14T14:34:43Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"install.go:62","msg":"Installing chart: meep-first-scenario-iperf-fog-server","time":"2022-03-14T14:34:43Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"install.go:62","msg":"Installing chart: meep-first-scenario-iperf-cloud-client","time":"2022-03-14T14:34:44Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"install.go:62","msg":"Installing chart: meep-first-scenario-ue2-term","time":"2022-03-14T14:34:45Z"}
{"level":"debug","meep.component":"meep-virt-engine","meep.from":"worker.go:51","msg":"Charts installed (4)","time":"2022-03-14T14:34:45Z"}

and meep-webhook

{"level":"debug","meep.component":"meep-webhook","meep.from":"webhook.go:88","msg":"RX MSG: Message[SCENARIO-ACTIVATE] Src[myapps:meep-sandbox-ctrl] Dst[all:all] Scope[mq:global] Payload[map[sandbox-name:myapps scenario-name:first-scenario]]","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-webhook","meep.from":"db.go:69","msg":"Redis Connector connecting to meep-redis-master.default.svc.cluster.local:6379","time":"2022-03-14T14:34:42Z"}
{"level":"info","meep.component":"meep-webhook","msg":"Redis Connector connected to meep-redis-master.default.svc.cluster.local:6379","time":"2022-03-14T14:34:42Z"}
{"level":"info","meep.component":"meep-webhook","msg":"Successfully connected to DB","time":"2022-03-14T14:34:42Z"}
{"level":"debug","meep.component":"meep-webhook","meep.from":"model.go:160","msg":"[meep-webhook] Model created myapps","time":"2022-03-14T14:34:42Z"}

Then in first place who is responsible to inject the tc-sidecar to the containers? webhook? I saw in the code something like this sidecarCfgFile string // path to sidecar injector configuration file where is this config file, maybe it's a permission issue.

Thank you again.

[dilallkx]

Hi @nordanz,

meep-webhook uses a mutating admission webhook to inject TC sidecar containers in scenario pods. However, your meep-webhook logs indicate that the mutating webhook is not being triggered during deployment & statefulset creation.

k8s documentation on admission control:

• https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
• https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

A few things we can check:

• Do you see any errors in your meep-webhook logs (e.g. Failed to listen and serve webhook server) ?
• Are you running a single node k8s deployment or a cluster?
• Please provide output from: ps -ef | grep kube-apiserver

Regards!

[nordanz]

Hello Kevin,

Many tanks for explanation. based on the provided info I checked the kube-apiserver logs and I see this failed request repeated with same error
E0314 14:34:45.812179 1 dispatcher.go:171] failed calling webhook "meep-webhook.idcc.com": Post "https://meep-webhook.default.svc:443/mutate?timeout=30s": x509: certificate signed by unknown authority

and regarding your questions:

• no, there is no error, only same logs
• yes, it's single node
• root 446198 446074 7 Mar14 ? 01:47:24 kube-apiserver --advertise-address=10.14.56.42 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

Thanks in advance.

[nordanz]

Hello Again,

Found the issue, I added the default.svc to no_proxy env in /etc/kubernetes/manifests/kube-apiserver.yaml and it start working for both demos. Thank you so much for your support and great software :).

[dilallkx]

Great! Glad to hear your issue is resolved and the platform is now working for you.
Thanks!