eBPF plugin crashed on update to v1.28.0-143-nightly

Ubuntu 20.04.1, kernel 5.4.0-62-generic. After the update to the above nightly this morning I noticed a crash report generated. Here is a link to the generated crash report eBPF Plugin crash 01172021 and the error log Netdata error log 01172021

1 Like

Thank you @chris.pollock for reporting this.

Pinging @Thiago_Marques_0 who is the main driver behind our ebpf functinality. As always you help us tons in improving and fixing our product in a rapid manner. We thank you :man_bowing:

Best,
Odysseas

Hello @chris.pollock ,

I hope everything is ok with your fingers.

I tried to replicate the problem on two different VMs and this time I could not recreate them. On the other hand, we discovered that we have a problem with our installer and probably this can affect the update, I will take a look tomorrow and let you know.

best regards!

Good evening @Thiago_Marques_0, i had the same crash this morning. The update went fine for a few days with no issues.

Hello @chris.pollock ,

Firstly I really apologize for the big delay, but the pandemic has been hard with my family. And this was affecting me so much, unfortunately I only discovered this last Thursday.
I hope your fingers are fine now.
I detected a problem on CentOS that I also think is also affecting Ubuntu, because both distributions are changing their file /proc/sys/kernel/osrelease.
The PR that can fix the problem is this.

Again I apologize!

Best regards!

Good afternoon Thiago, I also have to apologize for taking so long to reply. I had an arterial bypass done on 24 Feb to try and correct the blood clot in my hand and my left hand was in a large brace so I couldn’t use it at all. Finally got it off yesterday so I can type again. I’m still getting the eBPF crashes almost daily and there are days when I have to manually start Netdata from the command line. The crashes I’m pretty much ignoring as Netdata seems to be running fine and the plugin seems to be working correctly as far as I can see. If you’d like the most current crash info which was yesterday I can upload to pastebin and send you the link.

1 Like

Ooof buddy, I hope that you are better now. :v:

That pastebin should be useful for @Thiago_Marques_0 , so please do upload it!

Hello @chris.pollock ,

Firstly I hope that your fingers are good nowadays.
We merged a new eBPF release today an we also have another PR eBPF plugin (fixes 10727) by thiagoftsm · Pull Request #10756 · netdata/netdata · GitHub that is expected to fix these problems. When Ubuntu is running the kernel 4.18 we observed different problems on it, on the other hand the kernels 4.15 and 5.4 are more stable. Somehow the modifications the Ubuntu team is doing for 4.18 is creating some problems.

If you can upload to pastebin it will be great for me to debug.

Best regards!

@Thiago_Marques_0 I’m uploading three files
03-15-21 crash report
updater-log-03152021
error-log-03152021

I’m currently running kernel 5.4.0-67-generic

1 Like

thank you @chris.pollock ,

I took a look in the log files, but the crash report is saying that it is private, please, can you send it for my email thiago@netdata.cloud? Thanks.

Another thing, please, can you run the command ls -l /usr/libexec/netdata/plugins.d/ebpf.d? I wanna confirm that you are using the latest version of ebpf.plugin, if this directory is not present, it is possible there is a problem with the updater.

Finally I discovered the motive I cannot recreate the reported problem :confused: , I run every week the commands apt-get update; apt-get upgrade, but somehow on my VM the kernel was not upgraded, I was using 5.4.0-65. I imagined that Ubuntu followed the same pattern of Slackware and CentOS. I will try to recreate the problem right now.

Best Regards,

Thiago

apt-get dist-upgrade

Thank you, I never run this command :frowning:

Hello @chris.pollock ,

Thank you to send me the core dump, after to study it, I tried to recreate the problem and I could not, I gave to you some commands for you compile Netdata with debug flags that will help us to understand the problem with socket thread.

Best regards,

Thiago