Netdata Installation Causing Issues For Jenkins Service

Help
, ,
1

After setting up our jenkins cluster (controller + agents) as netdata agents using the curl command provided by our War room in our netdata cloud account, our jenkins service crashed the next day.

The VMs that comprise our jenkins cluster are all running debian buster.

Our investigation revealed that the jenkins:amd64 package was apt-get upgraded from version 2.289.1 to 2.332.1. This was a strange finding as there was no apt-get upgrade of the jenkins:amd64 package in the apt history log of our machines. What was even more strange was that the apt history log showed that an apt-get upgrade of the netdata-repo-edge package was happening on a daily basis. The first time at apt-get upgrade was run, a whole bunch of packages were upgraded, including jenkins (which was upgraded from 2.289.1 to 2.332.1). The second time the apt-get upgrade was run, it updated jenkins to 2.332.2.

Further digging then revealed that the apt-get upgrades happening on a daily basis was a result of a cronjob in /etc/cron.daily/netdata-updater, which was set up as part of the netdata agent installation.

I have deleted that cron job for now, to ensure that the jenkins package doesn’t get upgraded again. But made sense to notify you guys about this issue as it would be a problem for all users who are running their jenkins controller machines as netdata agents. Honestly, I’m not sure I understand why the jenkins package is a dependency of the netdata-repo-edge package, i.e., why does jenkins get upgraded when apt-get upgrade -y netdata-repo-edge is run.

Output of netdata -W buildinfo:
Version: netdata 1.34.0-1-nightly
Configure options: ‘–build=x86_64-linux-gnu’ ‘–includedir=${prefix}/include’ ‘–mandir=${prefix}/share/man’ ‘–infodir=${prefix}/share/info’ ‘–disable-silent-rules’ ‘–libdir=${prefix}/lib/x86_64-linux-gnu’ ‘–libexecdir=${prefix}/lib/x86_64-linux-gnu’ ‘–disable-maintainer-mode’ ‘–disable-dependency-tracking’ ‘–prefix=/usr’ ‘–sysconfdir=/etc’ ‘–localstatedir=/var’ ‘–libdir=/usr/lib’ ‘–libexecdir=/usr/libexec’ ‘–with-user=netdata’ ‘–with-math’ ‘–with-zlib’ ‘–with-webdir=/var/lib/netdata/www’ ‘build_alias=x86_64-linux-gnu’ ‘CFLAGS=-g -O2 -fdebug-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security’ ‘LDFLAGS=-Wl,-z,relro’ ‘CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2’ ‘CXXFLAGS=-g -O2 -fdebug-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security’
Install type: binpkg-deb
Binary architecture: x86_64
Packaging distro:
Features:
dbengine: YES
Native HTTPS: YES
Netdata Cloud: YES
ACLK Next Generation: YES
ACLK-NG New Cloud Protocol: YES
ACLK Legacy: NO
TLS Host Verification: YES
Machine Learning: YES
Stream Compression: NO
Libraries:
protobuf: YES (system)
jemalloc: NO
JSON-C: YES
libcap: NO
libcrypto: YES
libm: YES
tcalloc: NO
zlib: YES
Plugins:
apps: YES
cgroup Network Tracking: YES
CUPS: YES
EBPF: YES
IPMI: YES
NFACCT: YES
perf: YES
slabinfo: YES
Xen: NO
Xen VBD Error Tracking: NO
Exporters:
AWS Kinesis: NO
GCP PubSub: NO
MongoDB: NO
Prometheus Remote Write: YES

2

Hi, @njpau. There is a problem, indeed. I created a bug report.

TL;DR

The problem is using apt-get upgrade PKG for upgrading netdata/netdate-repo. We should use apt-get install (or apt-get install --only-upgrade).

1 Like
3

@njpau if you want to keep the version, you can pin it. To do it you need to create /etc/apt/preferences.d/jenkins.pref with the following content:

Package: jenkins
Pin: version 2.289.1
Pin-Priority: 999
1 Like
4

Sweets. Thanks folks

5

Fixed in fix upgrading all currently installed packages when updating Netdata on Debian by iigorkarpov · Pull Request #12716 · netdata/netdata · GitHub.

1 Like