Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Resource manager] Invalid memory address or nil pointer dereference #14

Open
TommyLike opened this issue Jun 18, 2019 · 6 comments
Open

[Resource manager] Invalid memory address or nil pointer dereference #14

TommyLike opened this issue Jun 18, 2019 · 6 comments

Comments

@TommyLike
Copy link

Operation:

  1. run up minicluster with command:
PELOTON=app make minicluster
  1. check container status
➜  peloton git:(master) PELOTON=app make minicluster
PELOTON=app ./scripts/minicluster.sh setup
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
tools/minicluster/main.py:61: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(f)
started container peloton-cassandra
failed to create cassandra store, retrying...
failed to create cassandra store, retrying...
cassandra store is created
started container peloton-zk
sleep 20 secs for zk to come up
started container peloton-mesos-master
started container peloton-mesos-agent0
started container peloton-mesos-agent1
started container peloton-mesos-agent2
docker image "uber/peloton" has to be built first locally by running IMAGE=uber/peloton make docker
app peloton-hostmgr0 is not up yet, retrying...
app peloton-hostmgr0 is not up yet, retrying...
app peloton-hostmgr0 is not up yet, retrying...
started peloton-hostmgr0
app peloton-resmgr0 is not up yet, retrying...
started peloton-resmgr0
app peloton-placement0 is not up yet, retrying...
started peloton-placement0
app peloton-placement1 is not up yet, retrying...
started peloton-placement1
app peloton-jobmgr0 is not up yet, retrying...
started peloton-jobmgr0
app peloton-aurorabridge0 is not up yet, retrying...
started peloton-aurorabridge0
➜  peloton git:(master) docker ps
CONTAINER ID        IMAGE                               COMMAND                  CREATED              STATUS              PORTS                                                                     NAMES
9409deaf5e49        uber/peloton                        "/bin/entrypoint.sh"     25 seconds ago       Up 24 seconds       0.0.0.0:5396->5396/tcp, 5290-5293/tcp, 0.0.0.0:8282->8282/tcp             peloton-aurorabridge0
dfa10c61cfeb        uber/peloton                        "/bin/entrypoint.sh"     30 seconds ago       Up 29 seconds       5290-5291/tcp, 0.0.0.0:5292->5292/tcp, 5293/tcp, 0.0.0.0:5392->5392/tcp   peloton-jobmgr0
f07c49345fc4        uber/peloton                        "/bin/entrypoint.sh"     36 seconds ago       Up 35 seconds       0.0.0.0:5303->5303/tcp, 5290-5293/tcp, 0.0.0.0:5403->5403/tcp             peloton-placement1
b0962774e39c        uber/peloton                        "/bin/entrypoint.sh"     42 seconds ago       Up 41 seconds       0.0.0.0:5293->5293/tcp, 5290-5292/tcp, 0.0.0.0:5393->5393/tcp             peloton-placement0
7c4ab56fe895        uber/peloton                        "/bin/entrypoint.sh"     48 seconds ago       Up 46 seconds       0.0.0.0:5290->5290/tcp, 0.0.0.0:5394->5394/tcp, 5291-5293/tcp             peloton-resmgr0
ddffb662e05b        uber/peloton                        "/bin/entrypoint.sh"     About a minute ago   Up About a minute   5290/tcp, 0.0.0.0:5291->5291/tcp, 5292-5293/tcp, 0.0.0.0:5391->5391/tcp   peloton-hostmgr0
b312e9107c1f        mesosphere/mesos-slave:1.6.0-rc1    "bash /files/run_mes…"   About a minute ago   Up About a minute   0.0.0.0:5053->5051/tcp                                                    peloton-mesos-agent2
c84a01d652f7        mesosphere/mesos-slave:1.6.0-rc1    "bash /files/run_mes…"   About a minute ago   Up About a minute   0.0.0.0:5052->5051/tcp                                                    peloton-mesos-agent1
6f321ccfb488        mesosphere/mesos-slave:1.6.0-rc1    "bash /files/run_mes…"   About a minute ago   Up About a minute   0.0.0.0:5051->5051/tcp                                                    peloton-mesos-agent0
354ae443d9e5        mesosphere/mesos-master:1.6.0-rc1   "bash /files/run_mes…"   About a minute ago   Up About a minute   0.0.0.0:5050->5050/tcp                                                    peloton-mesos-master
177c941daeb4        zookeeper:3.4                       "/docker-entrypoint.…"   About a minute ago   Up About a minute   2888/tcp, 3888/tcp, 0.0.0.0:8192->2181/tcp                                peloton-zk
1ca9583ad784        cassandra:3.0.14                    "bash /files/run_cas…"   About a minute ago   Up About a minute   7000-7001/tcp, 0.0.0.0:9042->9042/tcp, 7199/tcp, 0.0.0.0:9160->9160/tcp   peloton-cassandra
  1. create resource pool according to [tutorial] with yaml (https://github.com/uber/peloton/blob/master/docs/tutorial.md)
name: HelloWorldPool
owningteam: MyTeam
ldapgroups:
- MyGroup
description: "My first resource pool on Peloton"
resources:
- kind: cpu
  reservation: 3
  limit: 6
  share: 1
- kind: memory
  reservation: 2048
  limit: 4096
  share: 1
  1. resource manager container exited with logs
    Related logs:
{"level":"info","msg":"LookupResourcePoolID called","request":{"path":{"value":"/"}},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"CreateResourcePool called","request":{"config":{"name":"HelloWorldPool","owningTeam":"MyTeam","ldapGroups":["MyGroup"],"description":"My first resource pool on Peloton","resources":[{"kind":"cpu","reservation":3,"limit":6,"share":1},{"kind":"memory","reservation":2048,"limit":4096,"share":1}],"parent":{"value":"root"}}},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Scheduling policy is not set, use default PriorityFIFO","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"siblings to check","name":"HelloWorldPool","siblingNames":{},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"a8006272-9181-4529-b4b0-feccf339f654","slack_limit":{"CPU":0,"MEMORY":409.6,"DISK":0,"GPU":0},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":3,"MEMORY":2048,"DISK":0,"GPU":0},"respool_id":"a8006272-9181-4529-b4b0-feccf339f654","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-06-18T08:39:35Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-06-18T08:39:35Z"}
{"demand_cap_by_limit":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_cap_by_reservation_assignment":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_not_satisfied":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"First pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":3,"limit":6,"share":1},"memory":{"kind":"memory","reservation":2048,"limit":4096,"share":1}},"slack_entitlement":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-06-18T08:39:35Z","total_share":{}}
{"final_assignment":{"CPU":6,"MEMORY":0,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":3,"limit":6,"share":1},"memory":{"kind":"memory","reservation":2048,"limit":4096,"share":1}},"time":"2019-06-18T08:39:35Z"}
{"final_assignment":{"CPU":6,"MEMORY":4096,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":3,"limit":6,"share":1},"memory":{"kind":"memory","reservation":2048,"limit":4096,"share":1}},"time":"2019-06-18T08:39:35Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xbac011]

goroutine 611 [running]:
github.com/uber/peloton/resmgr/entitlement.(*Calculator).getChildShare(0xc000683b60, 0x1011aa0, 0xc0005f2360, 0xe90cf3, 0x4, 0x3ff0000000000000)
	/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:182 +0xd1
github.com/uber/peloton/resmgr/entitlement.(*Calculator).distributeUnclaimedResources(0xc000683b60, 0x1011aa0, 0xc0005f2360, 0xc001cc38e8, 0xc001cc3a30)
	/go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:320 +0x189
github.com/uber/peloton/resmgr/entitlement.(*Calculator).setEntitlementForChildren(0xc000683b60, 0x1011aa0, 0xc0005f2360)
	/go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:75 +0x572
github.com/uber/peloton/resmgr/entitlement.(*Calculator).calculateEntitlement(0xc000683b60, 0x1002b00, 0xc0000b8070, 0x0, 0x0)
	/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:162 +0x20e
github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start.func1(0xc000683b60, 0xc0004821c0)
	/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:106 +0x138
created by github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start
	/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:96 +0x133
@talaniz
Copy link

talaniz commented Jun 26, 2019

Currently, I'm unable to reproduce this. What does your setup look like (OS, go version, python version)?

@talaniz
Copy link

talaniz commented Jun 26, 2019

@TommyLike Can you pull the latest version and see if you're still reproducing this issue? I checked with our team and it looks like there was a fix put in for this, but would like to get some verification.

@talaniz
Copy link

talaniz commented Jul 1, 2019

@TommyLike Just wanted to follow up to see if you were able to confirm that this issue has been resolved?

@TommyLike
Copy link
Author

@talaniz sorry for late response, I will recheck this today:)

@TommyLike
Copy link
Author

TommyLike commented Jul 4, 2019 •
edited
Loading

Hey @talaniz , I paste some of the configurations and logs here in case needed.

Environment:：Ubuntu 16.04
Docker version:

Client:
 Version:           18.09.7
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        2d0083d
 Built:             Thu Jun 27 17:56:17 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.7
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       2d0083d
  Built:            Thu Jun 27 17:23:02 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Python Version:

Python 2.7.12

Images information:

uber/peloton                  latest              sha256:36e4d089ba273cd6c3c88ec213e80a37b88aac16308c9bb12e3f017967886188   4c1d9f5761be        3 months ago        1.8GB
mesosphere/mesos-slave        1.6.0-rc1           sha256:b44439db7b5bb2e7742d8e83efaa82d6b276ab7287f0d99107ef6bdb6502599b   69562200cd36        14 months ago       1.02GB
mesosphere/mesos-master       1.6.0-rc1           sha256:e63cfdbd7265ad4a73c4e01bb8ff1393c3fcf49c68ce330b0c56fd3977f63d9f   b2bbd52d0b2b        14 months ago       1.02GB
cassandra                     3.0.14              sha256:616454e966db82cbea0fdf0b579fa827445de952214edbab5a23b63bc92a5aa0   3bdcdfbd60a0        21 months ago       380MB

Node info:

⚡ root@husheng-test  ~/go_projects/src/github.com/uber/peloton/bin   master  ./peloton hostmgr hosts
              Hostname|   CPU|   GPU|         MEM|         Disk|  State| Task Hold
  peloton-mesos-agent0|  4.00|  0.00|  2048.00 MB|  20000.00 MB|  ready|
  peloton-mesos-agent1|  4.00|  0.00|  2048.00 MB|  20000.00 MB|  ready|
  peloton-mesos-agent2|  4.00|  0.00|  2048.00 MB|  20000.00 MB|  ready|

Resource pool yaml:

name: HelloWorldPool
owningteam: MyTeam
ldapgroups:
- MyGroup
description: "My first resource pool on Peloton"
resources:
- kind: cpu
  reservation: 1
  limit: 2
  share: 1
- kind: memory
  reservation: 1024
  limit: 2048
  share: 1

Parts of the log out put

{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"GetActiveTasks called","req":{"states":["LAUNCHING","PENDING","PLACED","PLACING","PREEMPTING","READY"]},"time":"2019-07-04T09:41:02Z"}
{"level":"info","msg":"GetActiveTasks returned","time":"2019-07-04T09:41:02Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:02Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:05Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:05Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:06Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:10Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:12Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:15Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:15Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:16Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:20Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:22Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:25Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:25Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:26Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:30Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:32Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:35Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:35Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"LookupResourcePoolID called","request":{"path":{"value":"/"}},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"CreateResourcePool called","request":{"config":{"name":"HelloWorldPool","owningTeam":"MyTeam","ldapGroups":["MyGroup"],"description":"My first resource pool on Peloton","resources":[{"kind":"cpu","reservation":1,"limit":2,"share":1},{"kind":"memory","reservation":1024,"limit":2048,"share":1}],"parent":{"value":"root"}}},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Scheduling policy is not set, use default PriorityFIFO","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"siblings to check","name":"HelloWorldPool","siblingNames":{},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"41b5f815-89ec-4a59-906b-c0623b2728c7","slack_limit":{"CPU":0,"MEMORY":204.8,"DISK":0,"GPU":0},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":1,"MEMORY":1024,"DISK":0,"GPU":0},"respool_id":"41b5f815-89ec-4a59-906b-c0623b2728c7","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:36Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:36Z"}
{"demand_cap_by_limit":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_cap_by_reservation_assignment":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_not_satisfied":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"First pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":1,"limit":2,"share":1},"memory":{"kind":"memory","reservation":1024,"limit":2048,"share":1}},"slack_entitlement":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:36Z","total_share":{}}
{"final_assignment":{"CPU":2,"MEMORY":0,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":1,"limit":2,"share":1},"memory":{"kind":"memory","reservation":1024,"limit":2048,"share":1}},"time":"2019-07-04T09:41:36Z"}
{"final_assignment":{"CPU":2,"MEMORY":2048,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":1,"limit":2,"share":1},"memory":{"kind":"memory","reservation":1024,"limit":2048,"share":1}},"time":"2019-07-04T09:41:36Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xbac011]

goroutine 79 [running]:
github.com/uber/peloton/resmgr/entitlement.(*Calculator).getChildShare(0xc00067fa40, 0x1011aa0, 0xc001632480, 0xe90cf3, 0x4, 0x3ff0000000000000)
	/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:182 +0xd1
github.com/uber/peloton/resmgr/entitlement.(*Calculator).distributeUnclaimedResources(0xc00067fa40, 0x1011aa0, 0xc001632480, 0xc001ddb8e8, 0xc001ddba30)
	/go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:320 +0x189
github.com/uber/peloton/resmgr/entitlement.(*Calculator).setEntitlementForChildren(0xc00067fa40, 0x1011aa0, 0xc001632480)
	/go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:75 +0x572
github.com/uber/peloton/resmgr/entitlement.(*Calculator).calculateEntitlement(0xc00067fa40, 0x1002b00, 0xc0000400f0, 0x0, 0x0)
	/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:162 +0x20e
github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start.func1(0xc00067fa40, 0xc0005584d0)
	/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:106 +0x138
created by github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start
	/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:96 +0x133

@talaniz
Copy link

talaniz commented Jul 25, 2019

@TommyLike thanks for this and apologies for the late reply. I'll check this with the team.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@talaniz @TommyLike