Creating dynamic charts within plugin get_data() method

Help
,
1

Is it possible to create dynamic charts within the get_data() method of a python plugin? Using the below example plugin I’m able to create the dynamic charts at runtime:

# -*- coding: utf-8 -*-
# Description: example netdata python.d module
# Author: Put your name here (your github login)
# SPDX-License-Identifier: GPL-3.0-or-later

from random import SystemRandom

from bases.FrameworkServices.SimpleService import SimpleService

priority = 90000

ORDER = []
CHARTS = {}

class Service(SimpleService):
    def __init__(self, configuration=None, name=None):
        SimpleService.__init__(self, configuration=configuration, name=name)
        self.order = ORDER
        self.definitions = CHARTS
        self.random = SystemRandom()

    def check(self):
        self.create_charts()
        return True

    def generate_data(self):
        dummy_data = {
            "my_chart_family0": ["my_chart_name0", "my_chart_name1"],
            "my_chart_familyB": ["my_chart_nameA", "my_chart_nameB"]
        }
        return dummy_data

    def get_data(self):
        data = dict()

        dummy_data = self.generate_data()
        for cf in dummy_data:
            # cf is the chart family
            for cn in dummy_data.get(cf):
                # cn is the chart name

                for i in range(1, 4):
                    dimension_id = "{}.{}.{}".format(cf, cn, i)

                    if dimension_id not in self.charts[cn]:
                        self.charts[cn].add_dimension([dimension_id])

                    data[dimension_id] = self.random.randint(0, 100)

        return data

    def create_charts(self):
        dummy_data = self.generate_data()

        # Loop through the dummy data.  Construct a chart and tie it to the chart family for grouping
        for cf in dummy_data:
            # cf is the chart family
            for cn in dummy_data.get(cf):
                # cn is the chart name
                CHARTS.update({cn: {
                    "options": [],
                    'lines': []
                }})

                self.order.insert(0, cn)
                self.definitions[cn] = {
                    'options': [None, 'Title', 'Unit', cf, cf, 'line'],
                    'lines': []
                }

However, if I shift self.create_charts() from the check() method into get_data() I get errors when trying to update the ORDER:

  File "/usr/libexec/netdata/python.d/python_modules/bases/FrameworkServices/SimpleService.py", line 197, in run
    updated = self.update(interval=since)
  File "/usr/libexec/netdata/python.d/python_modules/bases/FrameworkServices/SimpleService.py", line 222, in update
    data = self.get_data()
  File "/usr/libexec/netdata/python.d/random.chart.py", line 33, in get_data
    self.create_charts()
  File "/usr/libexec/netdata/python.d/random.chart.py", line 65, in create_charts
    self.order.insert(0, cn)
AttributeError: 'Service' object has no attribute 'order'

The purpose of me doing it within get_data() is that I need to create charts as consumer_groups (kafka) come and go and by restricting it to check() I’ll need to restart netdata every time a consumer_group is created in order to create the chart.

2

Hello!

It sure is possible, I actually had a need to do something similar recently and did it like this

Basically the charts object has add_chart() and add_dimension() methods you can use.

Let me know if anything I can do to help.

3

You can also remove dimensions as needed in case useful, here is an example of that netdata/alarms.chart.py at 292dea4e23fb381059fcc87da07b9c840ce2cb4e · netdata/netdata · GitHub

Your kafka use case sounds quite interesting, I’d love to hear more about it.

4

Adding link to this issue too as seem like a collector for kafka of some sort is something the community would find very useful.

5

@mjtice does it work if you just comment out the line that says self.order.insert()? Just wondering if it would work and then just order them itself behind the scenes as the charts get added?

6

Hi Andrew, it then fails on the next line:

self.definitions[cn] = {
                    'options': [None, 'Title', 'Unit', cf, cf, 'line'],
                    'lines': []
                }

AttributeError: 'Service' object has no attribute 'definitions'
7

self.order and self.definitions attributes are being deleted during SimpleService.create()

If there are any dynamic/charts dimensions we need to use self.charts (populated with charts in SimpleService.create())

8

Think of ORDER and CHARTS (self.order and self.definitions) as raw data that is being used to create charts/chart objects during init. Later we don’t use them.

It is a bit confusing because python.d was refactored (new behaviour was added on top of existing, not instead of) at some point, but we had maintain backward compatibility to not break any custom collector.

Lack of good documentation is another problem which needs to be fixed.

1 Like
9

I see. Thanks, @ilyam8

10

Thanks @andrewm4894 . I took this snippet of yours and integrated it into my script.

11

Essentially I’m just trying to track consumer_lag (nothing more at this point).

I started writing the plugin using the kafka-python lib and creating admin/consumer connections. This ended up being a little too slow and cumbersome because I had to reconnect the consumers on each iteration.

I pivoted to just using the REST interface bundled with GitHub - yahoo/CMAK: CMAK is a tool for managing Apache Kafka clusters. It makes for much shorter and easier-to-read code.

12

Cool - let us know how you get on - would be great to make a PR for whatever collector you build as i’m sure other kafka users (who are using CMAK) might find it very useful.

I’d be happy to try help any way i can.

1 Like
13

That sounds awesome @mjtice! Welcome to our community :slight_smile:

Thanks @ilyam8 and @andrewm4894 for the tips. A new python collector guide is underway, hopefully, it will be out shortly!

In the meantime, @mjtice or any other user has any questions about creating a custom python collector, we are all ears!

Interestingly, although we have questions every now and then about our python collector, we don’t hear that often about go.d @ilyam8 . Lack of evidence is of-course not evidence of lack, but it’s an interesting observation.

Cheers!