Hello,
I’m developing a pipeline for research and am exited to use Netdata to monitor the resources that it is taking. I’m very new to Netdata, and I can’t seem to find the monitoring I’m looking for in the documentation, hence my post here.
The pipeline I have will run multiple scripts and tools, each of which could spawn sub processes. Here is an example of a moment in time where my pipeline is running using the pstree command
|-sshd-+-sshd---sshd---bash---seq2mgs.sh---R---sh---reformat.sh---java-+-pigz---3*[{pigz}]
| | |-pigz---5*[{pigz}]
| | `-20*[{java}]
| |-2*[sshd---sshd---sftp-server]
| `-sshd---sshd---bash---pstree
The seq2mgs.sh script is the main script, which invokes R scripts that then start other processes / scripts. What I’d like to do is to monitor / export each of the sub-processes’ CPU and memory usage at set intervals (e.g. every 5 sec) and be able to calculate the afterwards the parts of the pipeline which high / low resource usage.
I have figured out how to add processes to the apps.groups.conf (like R or my own script), but I’m not sure how I can keep track of the process tree at any point in time so I know how to aggregate the data in the end.
Ideally I’d like a table / json with the CPU / Memory per process, and the parent process that it is linked to (if any) with seq2mgs.sh being the root.
I’m looking forward to any suggestions
PJ