Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak in wiregrid encoder agent #784

Open
BrianJKoopman opened this issue Oct 25, 2024 · 0 comments
Open

Memory leak in wiregrid encoder agent #784

BrianJKoopman opened this issue Oct 25, 2024 · 0 comments
Assignees
Labels
bug Something isn't working needs triage Cause of bug still unknown, needs investigation.

Comments

@BrianJKoopman
Copy link
Member

It seems under certain conditions there is a memory leak in the wiregrid encoder agent. We ran into this on satp1 last night:
image

Right around that time the logs go from filled with "Received IRIG timeout packet" to "while calling from thread":

2024-10-24 15:39:08.363 2024-10-24T15:39:08+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:18.819 2024-10-24T15:39:18+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:28.838 2024-10-24T15:39:28+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:38.862 2024-10-24T15:39:38+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:48.894 2024-10-24T15:39:48+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:58.904 2024-10-24T15:39:58+0000 Recieved IRIG timeout packet.
2024-10-24 15:40:08.922 2024-10-24T15:40:08+0000 Recieved IRIG timeout packet.
2024-10-24 15:40:21.938 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.939 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.940 2024-10-24T15:40:21+0000 Recieved IRIG timeout packet.
2024-10-24 15:40:21.940 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.941 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.942 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.944 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.944 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.946 2024-10-24T15:40:21+0000 while calling from thread

I'm not sure what's generating that "while calling from thread" message. I don't see anything with that string in it in the agent or Encoder beaglebone code, but maybe I'm missing something.

My immediate suspicion is in the nested while True: loops in grab_and_parse_data(). That's a good place to start for a deeper investigation. (And this should probably be improved, even if it doesn't end up being the problem.)

@BrianJKoopman BrianJKoopman added bug Something isn't working needs triage Cause of bug still unknown, needs investigation. labels Oct 25, 2024
@sadachi5 sadachi5 self-assigned this Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage Cause of bug still unknown, needs investigation.
Projects
None yet
Development

No branches or pull requests

3 participants