Netdata Community

How to look at data of docker from before restart?

Hi there,

I would like to look at the data of docker containers that restarted. Unfortunately, netdata is only reporting data from after the restart. (non-docker container-related metrics go further back).

I’m trying to find out the actual memory and CPU usage of said containers before they were restarted.

I can’t find any container data from before the restart. What I don’t know is if this data is just not showing or if it stored under a different id that is not showing in the navigation/chart list.

If you run docker restart, you should still see the old data. If the container was removed and a new one was created, it has a new id, so its charts are new and the old ones are made obsolete and aren’t displayed in the dashboard after a page refresh.

1 Like

After restarting docker they are all gone.

Can we somehow see/show them?

1 Like

If you have restarted docker, then the containers do not start automatically. Should you start again the containers (and not destry/create new ones), then you should be able to see the charts.

I think Netdata doesn’t show charts where the data sources have not sent any data for some time, is that correct @vlvkobal ?

Our containers are configured to automatically start on system reboot as well as docker restart.

1 Like

If you are not seeing any info about your containers, can you please try to restart Netdata and check the error.log, perhaps it can’t access the socket for some reason. One idea is that because it can’t access the containers as a data source for some time, it ceases to collect them. Another idea could be that the containers are ephemeral, and are deleted and created anew, meaning that while the image is the same, the containers are different (different IDs). This would result in Netdata to show new charts, since it can’t tie (for now) the old container to the new ones. Netdata is not configured (yet) to be able to handle ephemeral situations.

Pinging @ilyam8 who is our integrations maestro. Perhaps he can provide more context.

Cheers :slight_smile:

The container data is available but only from when they last started. The data from before is not available.

Can you please verify that the containers have a different ID than from those before the restart? I think that you destroyed the previous containers and rebuilt new ones from the same images, thus the containers are actually different.

1 Like

It depends on a collector. As for the cgroups plugin, yes, charts for stoppend containers aren’t shown.

Ok, so for swarm the services persists while the container is re-created even if it is only a restart.

Service:
image

Task:

So every time we change a config option, env var or update the image the data gets lost.

Is there any chance that this could be changed?

If you do any of these operations you are deleting an old container and creating a new one - it’s not restarting. As you can see from your screenshots, the id has been changed.

Please provide the output of docker ps before you apply changes and after that.

cgroups collector is unaware of docker swarm services, from its point of view it is a new container(s).

if it is only a restart

Do you mean a restart triggers recreatation of containers? (delete old one/creata a new one)

I assume it should be the case only if you make any changes (add a new env var, update the image, etc.)

When it is controlled by a swarm it creates a new container/task every time.

In any case, for us, it is critical to have data available from all containers that existed during the selected time period. Is this somehow possible?

No, it’s not possible at the moment and it’s not easy to implement. @dim08 do you think we should create a feature request? How is this supposed to work?