Custom file collector

Help
,
1

I am looking for some collector magic:

I have an application (restic backup service) which writes log files in json format. Is there a collector out there that I can use to parse this log file and extract the status to be sent to netdata? I want to keep track of the status of my last backup, whether it took place within a reasonable time period and whether its status was successful.

2

Hey @davosian,

That’s cool! I hope everything is alright regarding the pandemic, here the lockdown is again on the horizon, so I might get my hands dirty myself. (or even more dirty for that matter).

Cheers :stuck_out_tongue:

3

Hi @odyslam,

I have reviewed the python plugin info already. Looks straight forward enough :wink:
Now I just have to find a rainy day to get going…

4

Hey @davosian,

How is the collector going? Did you manage to get it started? You can read some initial documentation here: python.d.plugin | Learn Netdata.

If you need any help, just post it here and we will help you out in any way we can :slight_smile:

5

Thanks a lot for your offer, @OdysLam. Unfortunately, Go is not my cup of tea. Every time I look at some code I do not find it intuitive.
That being said, I am happy to look into a Python based solution with some pointers from @andrewm4894 (thanks a lot)! I am currently working on some final pieces for my back-up concept and will then try to dive into this connector to get the monitoring going.

6

I am by no means an expert here but maybe you could look at building a LogService collector in Python

I see fail2ban as maybe an example you could build from that seems to be based off a LogService collector: netdata/collectors/python.d.plugin/fail2ban at master · netdata/netdata · GitHub

Of course if you have the time and inclination to implement something in Go then i’d deo take up the offer from @OdysLam :slight_smile:

7

Hey @davosian,

Since we are migrating to Go, for a number of reasons, it would be ideal if you could write the collector in Go. Me and @ilyam8 we will support you along the way, so you are certain that you will succeed (and you will get comfortable with Go along the way). What do you think?

8

@davosian I think you are right, I do not remember any collector that monitors backups.

Is this not a good use case?

It is a good case for sure. We do not have a collector yet, because we did not have time to bring one, but this is something that we would like to monitor. Our goal is to monitor everything possible to help people like you.

I was hoping to find a good example (ideally in Python) that I could use for inspiration.

Netdata has more than 50 python collectors written at this directory https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin , you can use them as example, and of course, be free to ask us any doubt you have.

Best regards!

9

I just checked the repository of available connectors at Supported collectors list | Learn Netdata but it looks like there is none for monitoring backups. Am I missing something? Is this not a good use case?

I was hoping to find a good example (ideally in Python) that I could use for inspiration.

10

Hello @davosian ,

Unfortunatelly Netdata does not have a restic parser at this moment, but it is possible to create one using any programming language, because Netdata has an external language External plugins overview | Learn Netdata used to communicate with collectors.

Best regards!