docker stats broken on v1.38.0-153-nightly; On v1.37.0-91-g6954b6f88 it works

Help
1

I can’t see any docker charts on v1.38.0-153-nightly
before it worked OK

2

Hi @ben is this still an issue for you?

Can you provide a little bit more info about your system and installation.

# Linux
uname -a; grep -HvE "^#|URL" /etc/*release
# BSD
uname -a; uname -K
# macOS
uname -a; sw_vers

netdata -W buildinfo
# If get "netdata: command not found", try (required running Netdata)
$(ps aux | grep -m1 -E -o "[a-zA-Z/]+netdata ") -W buildinfo
3

Yes still broken.

Linux wololo 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
/etc/lsb-release:DISTRIB_ID=Ubuntu
/etc/lsb-release:DISTRIB_RELEASE=22.04
/etc/lsb-release:DISTRIB_CODENAME=jammy
/etc/lsb-release:DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
/etc/os-release:PRETTY_NAME="Ubuntu 22.04.1 LTS"
/etc/os-release:NAME="Ubuntu"
/etc/os-release:VERSION_ID="22.04"
/etc/os-release:VERSION="22.04.1 LTS (Jammy Jellyfish)"
/etc/os-release:VERSION_CODENAME=jammy
/etc/os-release:ID=ubuntu
/etc/os-release:ID_LIKE=debian
/etc/os-release:UBUNTU_CODENAME=jammy

netdata -W buildinfo
Version: netdata v1.38.0-159-nightly
Configure options:  '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--with-user=netdata' '--with-math' '--with-zlib' '--with-webdir=/var/lib/netdata/www' '--disable-dependency-tracking' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security'
Install type: binpkg-deb
    Binary architecture: x86_64
    Packaging distro:  
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES 
    ACLK:                       YES
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    YES
    EBPF:                    YES
    IPMI:                    YES
    NFACCT:                  YES
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES
Debug/Developer Features:
    Trace Allocations:       NO
4

Hi, @ben. What are docker stats/charts? What charts are you referring to?

5

Well like ~10 days or so ago I could see:

  • docker container count (both running and offline)
  • docker images count
    e.t.c
6

image
image2212×779 73.6 KB

This is screenshot from node which is still using
v1.37.0-91-g6954b6f88
And it works here.
v1.38* no docker

7

Ah, I see. Yes, there was a bug (GH Issue). Fixed in 14618. You need to wait for the next update, fixed in v1.38.0-163+).

2 Likes
8

Still doesn’t work. I have auto-updates on.
Netdata Agent:
v1.38.0-224-nightly

9

Try to run the plugin in debug mode.

10 
[ DEBUG ] build[manager] build.go:164 received config group ('/usr/lib/netdata/conf.d/go.d/docker_engine.conf'): 1 jobs (added: 1, removed: 0)
[ DEBUG ] build[manager] build.go:313 building docker_engine[local] job, config: map[__provider__:file reader __source__:/usr/lib/netdata/conf.d/go.d/docker_engine.conf autodetection_retry:0 module:docker_engine name:local priority:70000 update_every:1 url:http://127.0.0.1:9323/metrics]
[ ERROR ] docker_engine[local] docker_engine.go:107 Get "http://127.0.0.1:9323/metrics": dial tcp 127.0.0.1:9323: connect: connection refused
[ ERROR ] docker_engine[local] job.go:205 check failed
11

These are docker_engine log lines, not docker. Can you try again, please?

12

[ ERROR ] docker[local] docker.go:73 Error response from daemon: client version 1.42 is too new. Maximum supported API version is 1.41
[ ERROR ] docker[local] job.go:205 check failed

13

Ok, thanks. I think I see what is happening, fixed in docker: add api version negotiation. The fix will be in the nightly release early next week.