Exporting to JSON should write individual records

Help
,
1

We have configured JSON exporter and the problem is that this writes many records at once separated by line breaks. So each line written is valid JSON, but the full data string isn’t. This is a problem for certain receivers of that data as they are usually built to receive valid JSON only.

Could we get Netdata to write single records only to the exporter?

2

In fact Netdata is writing the string “\n” but not a real newline.

1 Like
3

Hey @jurgenhaas,

I was just about to answer this, but I see that @vlvkobal has replied to you over GitHub. For posterity, I link the GH issue here:

https://github.com/netdata/netdata/issues/10643

It would be best to continue the discussion here :man_bowing:

4

Thanks @OdysLam for linking this. I’ve just finished my tests and I can confirm, that the original problem continues to exist if you export over UDP. And I guess that’s a bug.

When switching to TCP, the export works as expected.

1 Like
5

Thanks for the update. I changed category for this topic to support. Once we create a bug issue to track this (or re-open yours), we will let you know through this thread.

cc @vlvkobal @ilyam8

6

@jurgenhaas yes, it looks like a bug.

@vlvkobal is on vacation till Monday. We will check the issue next week and open a ticket if needed.

1 Like
7

We made another test with Ilya.

Configured the exporting connector for UDP

[json:my_json_instance]
    enabled = yes
    destination = udp:127.0.0.1:2115
    send charts matching = system.processes

Ran the following Python script

import random
import socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket.bind(('', 2115))
while True:
    message, address = server_socket.recvfrom(1024)
    print(message.decode("utf-8"), end="")

The output

{"prefix":"netdata","hostname":"arch-esxi","chart_id":"system.processes","chart_name":"system.processes","chart_family":"processes","chart_context": "system.processes","chart_type":"system","units": "processes","id":"running","name":"running","value":5.6666667,"timestamp": 1614081221}
{"prefix":"netdata","hostname":"arch-esxi","chart_id":"system.processes","chart_name":"system.processes","chart_family":"processes","chart_context": "system.processes","chart_type":"system","units": "processes","id":"blocked","name":"blocked","value":0.0000000,"timestamp": 1614081221}

Thus, exporting works well using both TCP and UDP. @jurgenhaas, please check your UDP client.

8

Thanks @vlvkobal for the extra detail. We’re now happy ourselves with the TCP protocol instead of UDP, but we reported the issue to FluentD, which is the receiving client in this case. See Remove_newline default must be "false" in in_udp plugin · Issue #1749 · fluent/fluentd · GitHub

Putting this here just in case somebody else comes along with a similar issue.