Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid memory address or nil pointer dereference #12

Open
eric-engberg opened this issue Jun 27, 2023 · 18 comments
Open

Invalid memory address or nil pointer dereference #12

eric-engberg opened this issue Jun 27, 2023 · 18 comments

Comments

@eric-engberg
Copy link

Just tried installing and I'm getting this error
`time="2023-06-27T01:28:54Z" level=info msg="Starting EKS Cost Exporter. [log-level=info]"
time="2023-06-27T01:28:56Z" level=info msg="Retrieving EC2 Instance Types took 1.443560543s"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x21d3f68]

goroutine 1 [running]:
github.com/AndreZiviani/eks-cost-exporter/exporter.Metrics.getNormalizedCost(...)
/app/exporter/ec2.go:70
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).GetOnDemandPricing(0xc000170380, {0x2afd590, 0xc000056038})
/app/exporter/ec2.go:136 +0x928
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).GetInstances(0xc00005a00b?, {0x2afd590, 0xc000056038})
/app/exporter/ec2.go:35 +0x9b
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).init(0xc000170380, {0x2afd590, 0xc000056038})
/app/exporter/metrics.go:53 +0x205
github.com/AndreZiviani/eks-cost-exporter/exporter.NewMetrics({0x2afd590, 0xc000056038}, 0x27a78b8?, {0x3acec80, 0x0, 0x0}, {0x3acec80, 0x0, 0x0})
/app/exporter/metrics.go:28 +0x1a9
main.main()
/app/main.go:50 +0x249 `
@AndreZiviani
Copy link
Owner

I'm not sure whats wrong with just that, please try this version (docker image also available) and run it with debug log (-log-level debug)

@eric-engberg
Copy link
Author

Same results and no new log info

time="2023-06-27T16:56:20Z" level=info msg="Starting EKS Cost Exporter. [log-level=debug]"
time="2023-06-27T16:56:21Z" level=info msg="Retrieving EC2 Instance Types took 1.591354516s"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x21d3f68]

goroutine 1 [running]:
github.com/AndreZiviani/eks-cost-exporter/exporter.Metrics.getNormalizedCost(...)
	/app/exporter/ec2.go:70
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).GetOnDemandPricing(0xc0000f4000, {0x2afd590, 0xc000126008})
	/app/exporter/ec2.go:136 +0x928
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).GetInstances(0x26efde0?, {0x2afd590, 0xc000126008})
	/app/exporter/ec2.go:35 +0x9b
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).init(0xc0000f4000, {0x2afd590, 0xc000126008})
	/app/exporter/metrics.go:53 +0x205
github.com/AndreZiviani/eks-cost-exporter/exporter.NewMetrics({0x2afd590, 0xc000126008}, 0x27a78b8?, {0x3acec80, 0x0, 0x0}, {0x3acec80, 0x0, 0x0})
	/app/exporter/metrics.go:28 +0x1a9
main.main()
	/app/main.go:50 +0x249

@AndreZiviani
Copy link
Owner

looks like you are running the old version, please make sure to run the version I mentioned before, you should see the following log "Collected %d instance types" after "Retrieving EC2 Instance Types took 1.591354516s"

@eric-engberg
Copy link
Author

My bad. We vendor charts and I accidently change the tag in the default values yaml and not the one we provide. Still blowing up but some more info possible

time="2023-06-27T17:06:34Z" level=debug msg="Set log level to debug"
time="2023-06-27T17:06:34Z" level=info msg="Starting EKS Cost Exporter. [log-level=debug]"
time="2023-06-27T17:06:35Z" level=info msg="Collected 651 instance types"
time="2023-06-27T17:06:35Z" level=panic msg="could not find instance type %scc2.8xlarge"
time="2023-06-27T17:06:35Z" level=info msg="Retrieving EC2 Instance Types took 1.463842607s"
panic: (*logrus.Entry) 0xc000606a10

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0006069a0, 0x0, {0xc00059cf30, 0x2a})
	/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:260 +0x4d6
github.com/sirupsen/logrus.(*Entry).Log(0xc0006069a0, 0x0, {0xc0002a5520?, 0x10?, 0x2345c60?})
	/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:304 +0x4f
github.com/sirupsen/logrus.(*Logger).Log(0xc0000f4f00, 0x0, {0xc0002a5520, 0x2, 0x2})
	/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:204 +0x65
github.com/sirupsen/logrus.(*Logger).Panic(...)
	/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:253
github.com/sirupsen/logrus.Panic(...)
	/go/pkg/mod/github.com/sirupsen/[email protected]/exported.go:129
github.com/AndreZiviani/eks-cost-exporter/exporter.Metrics.getNormalizedCost({0xc000492660, 0xc000492690, 0xc0004926c0, 0x0, {{0xc00005a00b, 0x9}, {0x2ad5080, 0xc0001455c0}, {0x0, 0x0}, ...}, ...}, ...)
	/app/exporter/ec2.go:83 +0x165
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).GetOnDemandPricing(0xc0001cc540, {0x2b02830, 0xc000056038})
	/app/exporter/ec2.go:144 +0x92d
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).GetInstances(0xc00005a00b?, {0x2b02830, 0xc000056038})
	/app/exporter/ec2.go:36 +0x9b
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).init(0xc0001cc540, {0x2b02830, 0xc000056038})
	/app/exporter/metrics.go:53 +0x205
github.com/AndreZiviani/eks-cost-exporter/exporter.NewMetrics({0x2b02830, 0xc000056038}, 0x27acbeb?, {0x3ad1ca0, 0x0, 0x0}, {0x3ad1ca0, 0x0, 0x0})
	/app/exporter/metrics.go:28 +0x1a9
main.main()
	/app/main.go:50 +0x249

I'm guessing the issue is this? time="2023-06-27T17:06:35Z" level=panic msg="could not find instance type %scc2.8xlarge"

@AndreZiviani
Copy link
Owner

AndreZiviani commented Jun 27, 2023 •
edited
Loading

I'm guessing the issue is this?

correct, what is your ec2 instance type? is it cc2.8xlarge? never heard of it

@eric-engberg
Copy link
Author

should just be c2. Trying to view what all is in the struct but I don't know go so fumbling my way through

@eric-engberg
Copy link
Author

eric-engberg commented Jun 27, 2023 •
edited
Loading

some debut output I added. Everything looks correct except the last one.. but i guess it probably didn't get through everything

INFO[0000] Starting EKS Cost Exporter. [log-level=info]
INFO[0002] Collected 651 instance types
INFO[0002] Checking instance: m5d.12xlarge
INFO[0002] Checking instance: m5a.12xlarge
INFO[0002] Checking instance: m6in.xlarge
INFO[0002] Checking instance: r6idn.24xlarge
INFO[0002] Checking instance: r7g.metal
INFO[0002] Checking instance: r5n.metal
INFO[0002] Checking instance: m1.small
INFO[0002] Checking instance: c6in.xlarge
INFO[0002] Checking instance: t4g.large
INFO[0002] Checking instance: r5a.xlarge
INFO[0002] Checking instance: r5.2xlarge
INFO[0002] Checking instance: r6id.24xlarge
INFO[0002] Checking instance: r5b.large
INFO[0002] Checking instance: m6idn.12xlarge
INFO[0002] Checking instance: t2.2xlarge
INFO[0002] Checking instance: c6a.32xlarge
INFO[0002] Checking instance: m6i.xlarge
INFO[0002] Checking instance: m6i.2xlarge
INFO[0002] Checking instance: m6idn.large
INFO[0002] Checking instance: m6in.12xlarge
INFO[0002] Checking instance: c5.xlarge
INFO[0002] Checking instance: m5ad.12xlarge
INFO[0002] Checking instance: c4.2xlarge
INFO[0002] Checking instance: cc2.8xlarge

@AndreZiviani
Copy link
Owner

the only info I could find on cc2.8xlarge is that it was removed in 2018, I will see what I can do about it

@eric-engberg
Copy link
Author

Well we aren't using an cc2 instance that I know of.. Especially if they've been removed. Any ideas why you aren't having the problem?

@AndreZiviani
Copy link
Owner

Any ideas why you aren't having the problem?

because cc2.8xlarge is not available on my region.
when I run in us-east-1 it is still not returned by ec2 instance types list but IT IS by pricing api, will patch it to ignore instance types not returned by instance types api

@eric-engberg
Copy link
Author

Ah that makes sense. Though doesn't make sense why aws returns it at all.

@AndreZiviani
Copy link
Owner

please try this version
https://github.com/AndreZiviani/eks-cost-exporter/releases/tag/v0.8.3-rc1
https://github.com/AndreZiviani/eks-cost-exporter/pkgs/container/eks-cost-exporter/105059372?tag=v0.8.3-rc1

@eric-engberg
Copy link
Author

Well I saw the ignores but ended up with this still

time="2023-06-27T19:33:09Z" level=info msg="Retrieving current pod list took 600.837551ms"
time="2023-06-27T19:33:10Z" level=debug msg="Refreshing pod usage and cost"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x21dd61a]

goroutine 168 [running]:
github.com/AndreZiviani/eks-cost-exporter/exporter.(*Metrics).Collect(0xc0000ce000, 0x0?)
	/app/exporter/metrics.go:78 +0x21a
github.com/prometheus/client_golang/prometheus.DescribeByCollect.func1()
	/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/collector.go:90 +0x2b
created by github.com/prometheus/client_golang/prometheus.DescribeByCollect
	/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/collector.go:89 +0x9c

@AndreZiviani
Copy link
Owner

sorry for the issues and thank you for testing and reporting it back!

that is the crash I mentioned on the other PR that I still haven't figured out but it should work mostly fine (should only crash eventually), I've added some checks to try to make it not crash but I can't invest more time on it today
https://github.com/AndreZiviani/eks-cost-exporter/releases/tag/v0.8.3-rc2
https://github.com/AndreZiviani/eks-cost-exporter/pkgs/container/eks-cost-exporter/105063927?tag=v0.8.3-rc2

@eric-engberg
Copy link
Author

My day is pretty much spent.. I'll poke around and see if I can magically figure the issue out. I put it at about a 0.000000001% chance considering I don't know go at all

@eric-engberg
Copy link
Author

Just loaded rc2 and I see this time="2023-06-27T20:12:12Z" level=error msg="%!w(*errors.errorString=&{pod monitoring/zabbix-nodesclean-28121820-lxwq6 does not have a node associated!})" so maybe that's the cause? I could try removing that check and see

@eric-engberg
Copy link
Author

Just going to add on here. I can make another ticket if you want. We have a z1d.6xlarge instance right now but that is not on the metrics page. Been running for 38m so definitely should have updated..

Ok I just restarted it and it appears it loaded that instance now.. I'm guessing it's not updating it's instance list after the initial start.

@AndreZiviani
Copy link
Owner

hey @eric-engberg , I've just released a new version with a few improvements, feel free to check it
https://github.com/AndreZiviani/eks-cost-exporter/releases/tag/v0.8.3-rc3
https://github.com/users/AndreZiviani/packages/container/eks-cost-exporter/106579181?tag=v0.8.3-rc3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AndreZiviani @eric-engberg