How to look at data of docker from before restart?

Help
1

Hi there,

I would like to look at the data of docker containers that restarted. Unfortunately, netdata is only reporting data from after the restart. (non-docker container-related metrics go further back).

I’m trying to find out the actual memory and CPU usage of said containers before they were restarted.

I can’t find any container data from before the restart. What I don’t know is if this data is just not showing or if it stored under a different id that is not showing in the navigation/chart list.

2

If you run docker restart, you should still see the old data. If the container was removed and a new one was created, it has a new id, so its charts are new and the old ones are made obsolete and aren’t displayed in the dashboard after a page refresh.

1 Like
3

After restarting docker they are all gone.

Can we somehow see/show them?

1 Like
4

If you have restarted docker, then the containers do not start automatically. Should you start again the containers (and not destry/create new ones), then you should be able to see the charts.

I think Netdata doesn’t show charts where the data sources have not sent any data for some time, is that correct @vlvkobal ?

5

Our containers are configured to automatically start on system reboot as well as docker restart.

1 Like
6

If you are not seeing any info about your containers, can you please try to restart Netdata and check the error.log, perhaps it can’t access the socket for some reason. One idea is that because it can’t access the containers as a data source for some time, it ceases to collect them. Another idea could be that the containers are ephemeral, and are deleted and created anew, meaning that while the image is the same, the containers are different (different IDs). This would result in Netdata to show new charts, since it can’t tie (for now) the old container to the new ones. Netdata is not configured (yet) to be able to handle ephemeral situations.

Pinging @ilyam8 who is our integrations maestro. Perhaps he can provide more context.

Cheers :slight_smile:

7

The container data is available but only from when they last started. The data from before is not available.

8

Can you please verify that the containers have a different ID than from those before the restart? I think that you destroyed the previous containers and rebuilt new ones from the same images, thus the containers are actually different.

1 Like
9

It depends on a collector. As for the cgroups plugin, yes, charts for stoppend containers aren’t shown.

10

Ok, so for swarm the services persists while the container is re-created even if it is only a restart.

Service:
image

Task:

image
image932×483 22.3 KB

So every time we change a config option, env var or update the image the data gets lost.

Is there any chance that this could be changed?

11

If you do any of these operations you are deleting an old container and creating a new one - it’s not restarting. As you can see from your screenshots, the id has been changed.

Please provide the output of docker ps before you apply changes and after that.

12

cgroups collector is unaware of docker swarm services, from its point of view it is a new container(s).

if it is only a restart

Do you mean a restart triggers recreatation of containers? (delete old one/creata a new one)

I assume it should be the case only if you make any changes (add a new env var, update the image, etc.)

13

When it is controlled by a swarm it creates a new container/task every time.

In any case, for us, it is critical to have data available from all containers that existed during the selected time period. Is this somehow possible?

14

No, it’s not possible at the moment and it’s not easy to implement. @User23 do you think we should create a feature request? How is this supposed to work?

16

ahh, that is a pity.

Unfortunately this a big pain for us. On every new deployment and restart, the data is disappearing.

17

Same problem here. System crashed because of memory outage and I was very interested to know which container exhausted it. But since the containers are managed by portainer, they got recreated and all data from old containers was lost. IMHO, enabling viewing the history of old containers is ESSENTIAL feature of diagnostic system. The data was collected so why it is not displayed? I can understand that when you look the present live view, it shows only current containers. But when I go back in slot of history were only the old ones exited, netdata should show the diagnostics of them.
Is the data still there and not displayed or is it removed from database?