Unit tests failing on pkg: meep-metrics

[fehmi8]

Hello there,

I'm trying to pass all tests after a fresh AdvantEDGE installation, starting with the unit tests.

./start-ut-env.sh ran fine and my pods look like this after that:

default               meep-auth-svc-6bd7b55dd8-9przh                   1/1     Running            0                15h   10.32.0.16   main     <none>           <none>
default               meep-couchdb-couchdb-0                           1/1     Running            0                15h   10.32.0.4    main     <none>           <none>
default               meep-docker-registry-86b6b6b849-fkg29            1/1     Running            0                15h   10.32.0.5    main     <none>           <none>
default               meep-grafana-66fff596d-6jzcn                     1/1     Running            0                15h   10.32.0.6    main     <none>           <none>
default               meep-influxdb-0                                  1/1     Running            0                15h   10.32.0.7    main     <none>           <none>
default               meep-ingress-controller-wvgmf                    1/1     Running            0                15h   10.50.2.72   main     <none>           <none>
default               meep-ingress-defaultbackend-5d8778d54f-t8gjq     1/1     Running            0                15h   10.32.0.8    main     <none>           <none>
default               meep-kube-state-metrics-6c75f9d88c-hcpwv         1/1     Running            0                15h   10.32.0.9    main     <none>           <none>
default               meep-mon-engine-7b9d647b49-m6f2g                 1/1     Running            0                15h   10.32.0.11   main     <none>           <none>
default               meep-open-map-tiles-8698cccbfb-rqfj7             0/1     Pending            0                15h   <none>       <none>   <none>           <none>
default               meep-platform-ctrl-bc6488fbc-s77mf               1/1     Running            0                15h   10.32.0.15   main     <none>           <none>
default               meep-postgis-0                                   2/2     Running            0                15h   10.32.0.10   main     <none>           <none>
default               meep-prometheus-operator-7d979b99b-wspcx         1/1     Running            0                15h   10.32.0.12   main     <none>           <none>
default               meep-prometheus-prometheus-node-exporter-cfhb2   1/1     Running            0                15h   10.50.2.72   main     <none>           <none>
default               meep-redis-master-0                              2/2     Running            0                15h   10.32.0.19   main     <none>           <none>
default               meep-redis-slave-0                               2/2     Running            0                15h   10.32.0.14   main     <none>           <none>
default               meep-ut-couchdb-couchdb-0                        1/1     Running            0                16m   10.32.0.33   main     <none>           <none>
default               meep-ut-influxdb-0                               1/1     Running            0                16m   10.32.0.34   main     <none>           <none>
default               meep-ut-postgis-postgis-0                        2/2     Running            0                16m   10.32.0.35   main     <none>           <none>
default               meep-ut-redis-master-0                           2/2     Running            0                16m   10.32.0.31   main     <none>           <none>
default               meep-ut-redis-slave-0                            2/2     Running            0                16m   10.32.0.32   main     <none>           <none>
default               meep-virt-engine-7b4cbbcd7f-ws49n                1/1     Running            0                15h   10.32.0.17   main     <none>           <none>
default               meep-webhook-f97d4599b-8zvzn                     1/1     Running            0                15h   10.32.0.18   main     <none>           <none>
default               prometheus-meep-prometheus-prometheus-0          2/2     Running            0                15h   10.32.0.13   main     <none>           <none>
sandbox-system-test   meep-ams-5486fd765-nrp4v                         1/1     Running            0                14h   10.32.0.21   main     <none>           <none>
sandbox-system-test   meep-app-enablement-74f6d6cbd5-m7lhp             1/1     Running            0                14h   10.32.0.22   main     <none>           <none>
sandbox-system-test   meep-gis-engine-845bd9446c-fph6h                 1/1     Running            0                14h   10.32.0.30   main     <none>           <none>
sandbox-system-test   meep-loc-serv-69c45f8d8f-wshcd                   1/1     Running            0                14h   10.32.0.23   main     <none>           <none>
sandbox-system-test   meep-metrics-engine-d76d4546d-5977w              1/1     Running            0                14h   10.32.0.24   main     <none>           <none>
sandbox-system-test   meep-mg-manager-5f8989c949-hzmdt                 1/1     Running            0                14h   10.32.0.20   main     <none>           <none>
sandbox-system-test   meep-rnis-5cb75c66c4-fp95z                       1/1     Running            0                14h   10.32.0.25   main     <none>           <none>
sandbox-system-test   meep-sandbox-ctrl-566559c9db-244mz               1/1     Running            0                14h   10.32.0.27   main     <none>           <none>
sandbox-system-test   meep-tc-engine-6878b996c4-d5q2m                  1/1     Running            0                14h   10.32.0.28   main     <none>           <none>
sandbox-system-test   meep-vis-5496f8fdcf-r88nk                        1/1     Running            0                14h   10.32.0.29   main     <none>           <none>
sandbox-system-test   meep-wais-7cfc9987cf-nzjkz                       1/1     Running            0                14h   10.32.0.26   main     <none>           <none>

I assume the pending open-map-tiles service deployment is not relevant (?) so I went ahead and launched ./run-ut.sh.

All good at the beginning:

[...]
+ pkg: meep-couch
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-couch    3.109s  coverage: 79.3% of statements
+ pkg: meep-gis-asset-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-gis-asset-mgr    12.008s coverage: 59.0% of statements
+ pkg: meep-vis-traffic-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-vis-traffic-mgr  1.400s  coverage: 54.8% of statements

But then it throws an error at the TestIncrementalInfluxQuery test:

+ pkg: meep-metrics
---  TestEventsMetricsGetSet
Create valid Metric Store
[...]
---  TestHttpMetricsGetSet
Create valid Metric Store
[...]
---  TestMetricStoreNew
Invalid Metric Store address
[...]
---  TestMetricStoreGetSetInflux
Create valid Metric Store
[...]
---  TestMetricStoreCopyInflux
Create valid Metric Store
[...]
---  TestIncrementalInfluxQuery
Create valid Metric Store
[...]
time="2023-01-10T18:05:28.067Z" level=debug msg="QUERY: SELECT field1,field2,field3,field4 FROM metric1  WHERE (tag1='tag1') AND (tag2='tag2') ORDER BY desc LIMIT 50" meep.component=TestIncrementalInfluxQuery meep.from="metric-store.go:349"
time="2023-01-10T18:05:28.07Z" level=debug msg="QUERY: SELECT field1,field2,field3,field4 FROM metric1  WHERE (tag1='tag1') AND (tag2='tag2') AND time < 1673373926988167019 ORDER BY desc LIMIT 50" meep.component=TestIncrementalInfluxQuery meep.from="metric-store.go:349"
Length of results:  100
Get N = log count = maxlimit
time="2023-01-10T18:05:28.075Z" level=debug msg="QUERY: SELECT field1,field2,field3,field4 FROM metric1  WHERE (tag1='tag1') AND (tag2='tag2') ORDER BY desc LIMIT 100" meep.component=TestIncrementalInfluxQuery meep.from="metric-store.go:349"
Length of results:  100
Check Duration
time="2023-01-10T18:05:28.078Z" level=debug msg="QUERY: SELECT field1,field2,field3,field4 FROM metric1  WHERE (tag1='tag1') AND (tag2='tag2') AND time > 1673373928078589882 - 1s ORDER BY desc LIMIT 100" meep.component=TestIncrementalInfluxQuery meep.from="metric-store.go:349"
Length of results:  46
--- FAIL: TestIncrementalInfluxQuery (2.48s)
    metric-store_test.go:485: Failed to get metric

Then it continues without errors until the TestSessionMetricsGetSet test:

---  TestNetworkMetricGetSet
Create valid Metric Store
[...]
---  TestNetworkMetricSnapshot
Create valid Metric Store
[...]
---  TestSandboxMetricsGetSet
Create valid Metric Store
[...]
---  TestSessionMetricsGetSet
Create valid Metric Store
time="2023-01-10T18:05:28.958Z" level=debug msg="InfluxDB Connector connecting to http://localhost:30986" meep.component=TestSessionMetricsGetSet meep.from="metric-store.go:119"
time="2023-01-10T18:05:28.959Z" level=info msg="InfluxDB Connector connected to http://localhost:30986 version: 1.8.0" meep.component=TestSessionMetricsGetSet
time="2023-01-10T18:05:28.959Z" level=info msg="Connected to Metrics Influx DB" meep.component=TestSessionMetricsGetSet
time="2023-01-10T18:05:28.961Z" level=info msg="Store name set to: common_session_store" meep.component=TestSessionMetricsGetSet
time="2023-01-10T18:05:28.961Z" level=info msg="Successfully create Metric Store" meep.component=TestSessionMetricsGetSet
Flush store metrics
time="2023-01-10T18:05:28.992Z" level=info msg="[{0 [] [] }]" meep.component=TestSessionMetricsGetSet
Set session metric
Get session metrics
time="2023-01-10T18:05:29.422Z" level=debug msg="QUERY: SELECT provider,userid,sid,sbox,errtype,description FROM session  WHERE (type='login') AND time > 1673373929422336766 - 1ms ORDER BY desc LIMIT 100" meep.component=TestSessionMetricsGetSet meep.from="metric-store.go:349"
time="2023-01-10T18:05:29.424Z" level=debug msg="QUERY: SELECT provider,userid,sid,sbox,errtype,description FROM session  WHERE (type='login') ORDER BY desc LIMIT 1" meep.component=TestSessionMetricsGetSet meep.from="metric-store.go:349"
time="2023-01-10T18:05:29.427Z" level=debug msg="QUERY: SELECT provider,userid,sid,sbox,errtype,description FROM session  WHERE (type='login') ORDER BY desc LIMIT 100" meep.component=TestSessionMetricsGetSet meep.from="metric-store.go:349"
time="2023-01-10T18:05:29.429Z" level=debug msg="QUERY: SELECT provider,userid,sid,sbox,errtype,description FROM session  WHERE (type='logout') ORDER BY desc LIMIT 100" meep.component=TestSessionMetricsGetSet meep.from="metric-store.go:349"
time="2023-01-10T18:05:29.432Z" level=debug msg="QUERY: SELECT provider,userid,sid,sbox,errtype,description FROM session  WHERE (type='error') ORDER BY desc LIMIT 100" meep.component=TestSessionMetricsGetSet meep.from="metric-store.go:349"
FAIL
coverage: 68.1% of statements
FAIL    github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-metrics  5.543s
FAIL

and the rest runs fine:

+ pkg: meep-model
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-model    0.166s  coverage: 66.2% of statements
+ pkg: meep-mq
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-mq       3.734s  coverage: 82.1% of statements
+ pkg: meep-net-char-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-net-char 0.527s  coverage: 61.3% of statements
+ pkg: meep-pdu-session-store
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-pdu-session-store        0.029s  coverage: 89.2% of statements
+ pkg: meep-subscriptions
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-subscriptions    4.764s  coverage: 68.2% of statements
+ pkg: meep-users
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-users    2.048s  coverage: 73.3% of statements
+ pkg: meep-watchdog
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-watchdog 7.326s  coverage: 86.5% of statements
+ app: meep-loc-serv/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-loc-serv/server      12.301s coverage: 48.5% of statements
+ app: meep-rnis/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-rnis/server  13.252s coverage: 48.5% of statements
+ app: meep-wais/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-wais/server  4.034s  coverage: 42.9% of statements
+ app: meep-ams/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-ams/server   5.540s  coverage: 46.7% of statements

Looks like something is wrong with meep-metrics, maybe, here's the kubectl describe:

Name:             meep-metrics-engine-d76d4546d-5977w
Namespace:        sandbox-system-test
Priority:         0
Service Account:  meep-metrics-engine
Node:             main/10.50.2.72
Start Time:       Tue, 10 Jan 2023 03:18:29 +0000
Labels:           app=meep-metrics-engine
                  meepOrigin=core
                  pod-template-hash=d76d4546d
                  release=meep-metrics-engine
Annotations:      <none>
Status:           Running
IP:               10.32.0.24
IPs:
  IP:           10.32.0.24
Controlled By:  ReplicaSet/meep-metrics-engine-d76d4546d
Init Containers:
  init-system-kube-dns:
    Container ID:  containerd://003707751fa032122e63f52cc1d628626a68fbd37c956c6a3aebf02b06d4596f
    Image:         busybox:1.28
    Image ID:      docker.io/library/busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      until nslookup kube-dns.kube-system ; do echo waiting for kube-dns; sleep 0.25; done;
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 10 Jan 2023 03:18:47 +0000
      Finished:     Tue, 10 Jan 2023 03:18:47 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gwqwr (ro)
Containers:
  meep-metrics-engine:
    Container ID:   containerd://e770b67098fd97b99c676d96d234f2b3f4ba315cb68d77ad8ba38e30b7952e90
    Image:          meep-docker-registry:30001/meep-metrics-engine:latest
    Image ID:       meep-docker-registry:30001/meep-metrics-engine@sha256:e6d4c7a824a153af4d5bc48258218c327c48fa2f7c2909802b9cfa08f2a6ecd3
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 10 Jan 2023 03:20:47 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      MEEP_CODECOV:       
      MEEP_HOST_URL:      http://my-platform-fqdn
      MEEP_SANDBOX_NAME:  sandbox-system-test
      MEEP_SVC_PATH:      /metrics/v2
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gwqwr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-gwqwr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

And the logs:

-:~$ kubectl logs  meep-metrics-engine-d76d4546d-5977w -n sandbox-system-test
MEEP_HOST_URL: http://my-platform-fqdn
MEEP_SANDBOX_NAME: sandbox-system-test
MEEP_MEP_NAME: 
MEEP_CODECOV: 
Replacing [localhost] with my-platform-fqdn to url in: '/api/AdvantEDGE Metrics Service Notification REST API'
Replacing [sandboxname] with sandbox-system-test to basepath or url in: '/api/AdvantEDGE Metrics Service Notification REST API'
Replacing [localhost] with my-platform-fqdn to url in: '/api/AdvantEDGE Metrics Service REST API'
Replacing [sandboxname] with sandbox-system-test to basepath or url in: '/api/AdvantEDGE Metrics Service REST API'
{"level":"info","meep.component":"meep-metrics-engine","msg":"[/meep-metrics-engine]","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Starting Metric Engine Service","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"MEEP_SANDBOX_NAME: sandbox-system-test","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"resource URL: http://my-platform-fqdn","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Creating new MsgQueue","time":"2023-01-10T03:20:47Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"db.go:69","msg":"Redis Connector connecting to meep-redis-master.default.svc.cluster.local:6379","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Redis Connector connected to meep-redis-master.default.svc.cluster.local:6379","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Successfully connected to DB","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Connected to Message Queue Redis DB","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Message Queue created","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"MEEP_HOST_URL: http://my-platform-fqdn","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"MEEP_SVC_PATH: /metrics/v2","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Service URL: http://my-platform-fqdn/sandbox-system-test/metrics/v2","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Created Swagger API Manager","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Swagger API Manager created","time":"2023-01-10T03:20:47Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"db.go:69","msg":"Redis Connector connecting to meep-redis-master.default.svc.cluster.local:6379","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Redis Connector connected to meep-redis-master.default.svc.cluster.local:6379","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Successfully connected to DB","time":"2023-01-10T03:20:47Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"model.go:161","msg":"[meep-metrics-engine] Model created activeScenario","time":"2023-01-10T03:20:47Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"db.go:69","msg":"Redis Connector connecting to meep-redis-master.default.svc.cluster.local:6379","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Redis Connector connected to meep-redis-master.default.svc.cluster.local:6379","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Successfully connected to DB","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Connected to Metrics Redis DB","time":"2023-01-10T03:20:47Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"metric-store.go:119","msg":"InfluxDB Connector connecting to http://meep-influxdb.default.svc.cluster.local:8086","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"InfluxDB Connector connected to http://meep-influxdb.default.svc.cluster.local:8086 version: 1.8.0","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Connected to Metrics Influx DB","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Store name set to: ","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Successfully create Metric Store","time":"2023-01-10T03:20:47Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"db.go:69","msg":"Redis Connector connecting to meep-redis-master.default.svc.cluster.local:6379","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Redis Connector connected to meep-redis-master.default.svc.cluster.local:6379","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Successfully connected to DB","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Connected to Redis DB","time":"2023-01-10T03:20:47Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"model.go:1482","msg":"Scenario was deleted","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Subscription Message: subscribe to channel mq:sandbox-system-test. Total subscriptions: 1","time":"2023-01-10T03:20:47Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Swagger API Manager started","time":"2023-01-10T03:20:48Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"swagger-api-mgr.go:221","msg":"TX MSG: Message[API-UPDATE] Src[sandbox-system-test:meep-metrics-engine] Dst[sandbox-system-test:meep-sandbox-ctrl] Scope[mq:sandbox-system-test] Payload[map[apilist:{\"Apis\":[{\"Name\":\"AdvantEDGE Metrics Service Notification REST API\",\"Url\":\"http://my-platform-fqdn/sandbox-system-test/metrics/v2/api/AdvantEDGE Metrics Service Notification REST API\"},{\"Name\":\"AdvantEDGE Metrics Service REST API\",\"Url\":\"http://my-platform-fqdn/sandbox-system-test/metrics/v2/api/AdvantEDGE Metrics Service REST API\"}],\"UserApis\":null} mep: module:meep-metrics-engine]]","time":"2023-01-10T03:20:48Z"}
{"level":"info","meep.component":"meep-metrics-engine","msg":"Swagger APIs successfully added","time":"2023-01-10T03:20:48Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"swagger-api-mgr.go:263","msg":"RX MSG: Message[API-REQUEST] Src[sandbox-system-test:meep-sandbox-ctrl] Dst[sandbox-system-test:all] Scope[mq:sandbox-system-test] Payload[map[]]","time":"2023-01-10T03:21:54Z"}
{"level":"debug","meep.component":"meep-metrics-engine","meep.from":"swagger-api-mgr.go:221","msg":"TX MSG: Message[API-UPDATE] Src[sandbox-system-test:meep-metrics-engine] Dst[sandbox-system-test:meep-sandbox-ctrl] Scope[mq:sandbox-system-test] Payload[map[apilist:{\"Apis\":[{\"Name\":\"AdvantEDGE Metrics Service Notification REST API\",\"Url\":\"http://my-platform-fqdn/sandbox-system-test/metrics/v2/api/AdvantEDGE Metrics Service Notification REST API\"},{\"Name\":\"AdvantEDGE Metrics Service REST API\",\"Url\":\"http://my-platform-fqdn/sandbox-system-test/metrics/v2/api/AdvantEDGE Metrics Service REST API\"}],\"UserApis\":null} mep: module:meep-metrics-engine]]","time":"2023-01-10T03:21:54Z"}

Any clue what went wrong?

Thanks

[dilallkx]

Hi @fehmi8,

Thank you for reaching out!

The unit tests pass on my setup:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>> Running Unit Tests
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

+ pkg: meep-couch
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-couch    0.301s  coverage: 82.9% of statements

+ pkg: meep-gis-asset-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-gis-asset-mgr    2.707s  coverage: 59.1% of statements

+ pkg: meep-vis-traffic-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-vis-traffic-mgr  0.255s  coverage: 54.8% of statements

+ pkg: meep-metrics
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-metrics  15.865s coverage: 68.4% of statements

+ pkg: meep-model
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-model    0.163s  coverage: 66.2% of statements

+ pkg: meep-mq
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-mq       3.726s  coverage: 82.1% of statements

+ pkg: meep-net-char-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-net-char 0.444s  coverage: 61.3% of statements

+ pkg: meep-pdu-session-store
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-pdu-session-store        0.015s  coverage: 89.2% of statements

+ pkg: meep-subscriptions
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-subscriptions    4.747s  coverage: 68.2% of statements

+ pkg: meep-users
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-users    0.836s  coverage: 74.3% of statements

+ pkg: meep-watchdog
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-watchdog 7.317s  coverage: 86.5% of statements

+ app: meep-loc-serv/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-loc-serv/server      11.908s coverage: 48.5% of statements

+ app: meep-rnis/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-rnis/server  12.998s coverage: 48.5% of statements

+ app: meep-wais/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-wais/server  3.616s  coverage: 42.9% of statements

+ app: meep-ams/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-ams/server   5.401s  coverage: 46.7% of statements

For some reason your initial unit tests take much more time than mine. Did you immediately run the UT after deploying the test databases (i.e. start-ut-env.sh)? You need to allow the database pods to start completely using start-ut-env.sh before running the unit tests. When you run them again afterwards do you get the same error?

WRT your error, it seems the number of metrics retrieved from the Influx DB was less than expected. This may be due to a timing issue on your setup. Could you please check the timestamp just before the Set metrics log in the TestSessionMetricsGetSet unit test and compare it with the failure log timestamp? The Influx query retrieves all metrics stored within the last 1 second, however it may not get all of them if there is a delay between the moment the metrics were written and fetched. If the time difference in the logs is greater than 1s then this is likely your issue. If this is the case you can try editing the unit test to increase the duration to 10s instead.

Other notes:

• The coverage results are different than mine. Are you using the latest v1.9.0 release on the master branch? Make sure your meepctl configuration points to the latest repo.
• meep-open-map-tiles pod should eventually start; it may take more time on the first install but it should start. However, the unit tests should pass even when this pod is not started.
• The meep-metrics-engine pod in the sandbox-system-test sandbox (i.e. k8s namespace) is used for the system tests and has no impact on the unit tests. Only the meep-metrics package is unit tested.

I hope this helps find your issue.
Regards!

[fehmi8]

Hi Kevin,

Thanks for your answer.

Did you immediately run the UT after deploying the test databases (i.e. start-ut-env.sh)?

I waited until everything was deployed (except the open map tiles).

the timestamp just before the Set metrics log in the TestSessionMetricsGetSet unit test

When running ./run-ut.sh I see the timestamp 2023-01-10T18:05:28.992Z right before the Set session metric of the TestSessionMetricsGetSet unit test (I don't see any Set metrics). Is that the timestamp we are looking for?

the failure log timestamp

As for the failure log timestamp, the last timestamp I see in the output I posted of ./run-ut.sh before the FAIL is 2023-01-10T18:05:29.432Z. Is that the timestamp we are looking for?

The coverage results are different than mine. Are you using the latest v1.9.0 release on the master branch? Make sure your meepctl configuration points to the latest repo.

Looks like it, yes!

-:~$ meepctl version
Using repo config file: /home/fehmi/AdvantEDGE/.meepctl-repocfg.yaml
Using meepctl config file: /home/fehmi/.meepctl.yaml
{
  "name": "meepctl",
  "version": "1.9.0"
}
{
  "name": ".meepctl-repocfg.yaml",
  "version": "1.9.0"
}

meep-open-map-tiles pod should eventually start; it may take more time on the first install but it should start. However, the unit tests should pass even when this pod is not started.

Still pending after 13 days, I guess something went wrong... 🤔

I now ran ./stop-ut-env.sh and then ./start-ut-env.sh again. Everything seems to be running:

-:~$ kubectl get pods --all-namespaces -o wide | grep meep
default               meep-auth-svc-6bd7b55dd8-9przh                   1/1     Running            0                13d    10.32.0.16   main     <none>           <none>
default               meep-couchdb-couchdb-0                           1/1     Running            0                13d    10.32.0.4    main     <none>           <none>
default               meep-docker-registry-86b6b6b849-fkg29            1/1     Running            0                13d    10.32.0.5    main     <none>           <none>
default               meep-grafana-66fff596d-6jzcn                     1/1     Running            0                13d    10.32.0.6    main     <none>           <none>
default               meep-influxdb-0                                  1/1     Running            0                13d    10.32.0.7    main     <none>           <none>
default               meep-ingress-controller-wvgmf                    1/1     Running            0                13d    10.50.2.72   main     <none>           <none>
default               meep-ingress-defaultbackend-5d8778d54f-t8gjq     1/1     Running            0                13d    10.32.0.8    main     <none>           <none>
default               meep-kube-state-metrics-6c75f9d88c-hcpwv         1/1     Running            0                13d    10.32.0.9    main     <none>           <none>
default               meep-mon-engine-7b9d647b49-m6f2g                 1/1     Running            0                13d    10.32.0.11   main     <none>           <none>
default               meep-open-map-tiles-8698cccbfb-rqfj7             0/1     Pending            0                13d    <none>       <none>   <none>           <none>
default               meep-platform-ctrl-bc6488fbc-s77mf               1/1     Running            0                13d    10.32.0.15   main     <none>           <none>
default               meep-postgis-0                                   2/2     Running            0                13d    10.32.0.10   main     <none>           <none>
default               meep-prometheus-operator-7d979b99b-wspcx         1/1     Running            0                13d    10.32.0.12   main     <none>           <none>
default               meep-prometheus-prometheus-node-exporter-cfhb2   1/1     Running            0                13d    10.50.2.72   main     <none>           <none>
default               meep-redis-master-0                              2/2     Running            0                13d    10.32.0.19   main     <none>           <none>
default               meep-redis-slave-0                               2/2     Running            0                13d    10.32.0.14   main     <none>           <none>
default               meep-ut-couchdb-couchdb-0                        1/1     Running            0                96s    10.32.0.33   main     <none>           <none>
default               meep-ut-influxdb-0                               1/1     Running            0                88s    10.32.0.34   main     <none>           <none>
default               meep-ut-postgis-postgis-0                        2/2     Running            0                83s    10.32.0.35   main     <none>           <none>
default               meep-ut-redis-master-0                           2/2     Running            0                101s   10.32.0.31   main     <none>           <none>
default               meep-ut-redis-slave-0                            2/2     Running            0                101s   10.32.0.32   main     <none>           <none>
default               meep-virt-engine-7b4cbbcd7f-ws49n                1/1     Running            0                13d    10.32.0.17   main     <none>           <none>
default               meep-webhook-f97d4599b-8zvzn                     1/1     Running            0                13d    10.32.0.18   main     <none>           <none>
default               prometheus-meep-prometheus-prometheus-0          2/2     Running            0                13d    10.32.0.13   main     <none>           <none>
sandbox-system-test   meep-ams-5486fd765-nrp4v                         1/1     Running            0                13d    10.32.0.21   main     <none>           <none>
sandbox-system-test   meep-app-enablement-74f6d6cbd5-m7lhp             1/1     Running            0                13d    10.32.0.22   main     <none>           <none>
sandbox-system-test   meep-gis-engine-845bd9446c-fph6h                 1/1     Running            0                13d    10.32.0.30   main     <none>           <none>
sandbox-system-test   meep-loc-serv-69c45f8d8f-wshcd                   1/1     Running            0                13d    10.32.0.23   main     <none>           <none>
sandbox-system-test   meep-metrics-engine-d76d4546d-5977w              1/1     Running            0                13d    10.32.0.24   main     <none>           <none>
sandbox-system-test   meep-mg-manager-5f8989c949-hzmdt                 1/1     Running            0                13d    10.32.0.20   main     <none>           <none>
sandbox-system-test   meep-rnis-5cb75c66c4-fp95z                       1/1     Running            0                13d    10.32.0.25   main     <none>           <none>
sandbox-system-test   meep-sandbox-ctrl-566559c9db-244mz               1/1     Running            0                13d    10.32.0.27   main     <none>           <none>
sandbox-system-test   meep-tc-engine-6878b996c4-d5q2m                  1/1     Running            0                13d    10.32.0.28   main     <none>           <none>
sandbox-system-test   meep-vis-5496f8fdcf-r88nk                        1/1     Running            0                13d    10.32.0.29   main     <none>           <none>
sandbox-system-test   meep-wais-7cfc9987cf-nzjkz                       1/1     Running            0                13d    10.32.0.26   main     <none>           <none>

So I try again ./run-ut.sh (3 minutes later) and now I get the same coverage results as you do!

+ pkg: meep-couch
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-couch    3.222s  coverage: 82.9% of statements
+ pkg: meep-gis-asset-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-gis-asset-mgr    42.598s coverage: 59.1% of statements
+ pkg: meep-vis-traffic-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-vis-traffic-mgr  1.562s  coverage: 54.8% of statements

Then errors with meep-metrics

+ pkg: meep-model
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-model    0.168s  coverage: 66.2% of statements
+ pkg: meep-mq
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-mq       3.734s  coverage: 82.1% of statements
+ pkg: meep-net-char-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-net-char 0.461s  coverage: 61.3% of statements
+ pkg: meep-pdu-session-store
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-pdu-session-store        0.034s  coverage: 89.2% of statements
+ pkg: meep-subscriptions
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-subscriptions    4.767s  coverage: 68.2% of statements
+ pkg: meep-users
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-users    34.126s coverage: 74.3% of statements
+ pkg: meep-watchdog
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-watchdog 7.328s  coverage: 86.5% of statements

Errors with meep-loc-serv/server

+ app: meep-rnis/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-rnis/server  13.224s coverage: 48.5% of statements

Errors with meep-wais/server

+ app: meep-ams/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-ams/server   5.538s  coverage: 46.7% of statements

I ran again ./run-ut.sh and this time I get different coverage values aagin, and errors in meep-metrics and meep-loc-serv/server (errors in meep-wais/server disappeared)

+ pkg: meep-couch
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-couch    0.974s  coverage: 79.3% of statements
+ pkg: meep-gis-asset-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-gis-asset-mgr    10.379s coverage: 59.0% of statements
+ pkg: meep-vis-traffic-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-vis-traffic-mgr  0.914s  coverage: 54.8% of statements
+ pkg: meep-metrics
[...]
--- FAIL: TestIncrementalInfluxQuery (5.26s)
    metric-store_test.go:485: Failed to get metric
[...]
FAIL
coverage: 68.1% of statements
FAIL    github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-metrics  8.950s
FAIL
+ pkg: meep-model
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-model    0.161s  coverage: 66.2% of statements
+ pkg: meep-mq
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-mq       3.736s  coverage: 82.1% of statements
+ pkg: meep-net-char-mgr
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-net-char 0.474s  coverage: 61.3% of statements
+ pkg: meep-pdu-session-store
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-pdu-session-store        0.030s  coverage: 89.2% of statements
+ pkg: meep-subscriptions
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-subscriptions    4.766s  coverage: 68.2% of statements
+ pkg: meep-users
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-users    0.414s  coverage: 73.3% of statements
+ pkg: meep-watchdog
ok      github.com/InterDigitalInc/AdvantEDGE/go-packages/meep-watchdog 7.325s  coverage: 86.5% of statements
+ app: meep-loc-serv/server
[...]
--- FAIL: TestZoneSubscriptionNotification (0.65s)
    loc-serv_test.go:3064: Failed to get expected response
FAIL
coverage: 46.8% of statements
FAIL    github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-loc-serv/server      12.001s
FAIL
+ app: meep-rnis/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-rnis/server  13.284s coverage: 48.5% of statements
+ app: meep-wais/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-wais/server  3.733s  coverage: 42.9% of statements
+ app: meep-ams/server
ok      github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-ams/server   5.536s  coverage: 46.7% of statements

Thanks for helping. But I guess I should just start trying to use AdvantEdge despite those tests failing...

Fehmi

[dilallkx]

Hi @fehmi8,

Thank you for your additional feedback!

First, I mistakenly gave you the wrong test name to get the Set metrics log from... I meant for you to look at the TestIncrementalInfluxQuery test, which is the one that fails. Could you compare the timestamps for this test between the Set metrics log and when the failure occurs? You can then follow the steps in my previous response.

WRT the coverage discrepancies, it is possible some timing related differences are occurring that are not cause UT failures. Ideally this should not occur and every operation should be properly validated. Could you please provide logs for test runs where you obtain different coverage results? You will need to enable verbose unit test execution by modifying test execution command in run-ut.sh as follows: go test -v -count=1 ./... -cover

Also, your test execution time is much longer than my test run. What hardware/software setup are you using to run AdvantEDGE? Do you meet the minimum/recommended requirements?

Finally, for open map tiles, try redeploying the platform dependencies to see if it starts successfully. To do so make sure you meepctl delete core and meepctl delete dep and that all pods are terminated before you start them again. Also note that the system test sandbox pods should no longer be present after running the system tests. They may be present due to a system test execution failure and can be removed by deleting the sandbox in the AdvantEDGE frontend (note: do this before uninstalling the platform). If the open map tiles issue persists, please provide pod logs to help pinpoint the issue. You can use kubectl describe pod <pod name> to get this information.

I hope this helps!
Regards.