Netdata Community

Is stream compression possible

Environment

Problem/Question

For data streaming is it possible to perform compression?

What I expected to happen

Streaming protocol is clear-text, verbose and redundant. It appears compression could significantly reduce network bandwidth.

Hello @sdo ,

As you described, we are not compressing the stream protocol right now, but we have a PR open that will change our protocol.

You are giving a good suggestion for possible next steps of the new protocol we are workin. @Stelios_Fragkakis I am not sure if this was already planned for the change that we are doing now, if we are already bringing this, please, let us know.

Finally Netdata uses HTTP compression when libz is installed, so it is possible that we need to do an extension for STREAM.

Best regards!

Thanks, I will follow the PR. Although it seems the PR takes a different approach, and it appears that it is in late stages already.

The compression is not planned as part of the work in the PR you mentioned, but we certainly need to look into it to reduce bandwidth usage.

2 Likes

This is a great discussion. @sdo it would be very helpful if you shared any ideas and your use-case that demands lower bandwidth. It allows the contributors to better understand the problem space so that they can design a better solution.

I am moving this topic to the #feature-requests:agent-fr category.

Thanks for opportunity to explain further.

The use case for us, is that we have a couple hundred servers spread geographically in satellite installations. There is no space for storing data at each location, so it’s streamed to a central location. Streaming is saturating WAN links, causing loss of data and impacting business applications. High resolution data is required for detecting and troubleshooting usage spikes. We don’t require real-time data - near realtime would be sufficient. For example transmission every 15 seconds of 1 second resolution data. It’s essential that the total network bandwidth usage is reduced significantly.

Being able to look at clear text transmissions is helpful in debugging. And clear-text transmission in production is excessive. Preferably stream compression should be an optional feature.

1 Like

Really interesting discussion. A while ago when we were streaming from some community members into our research ml parent nodes i noticed a huge network ingress and egress costs as some of them had to either stream from or to GCP <-> AWS. Seems like this discussion relevant to what i was struggling a bit with then.

1 Like