Hi @Morne_Supra thanks for trying it out!
-
It will just drop and retrain a new model based on last 4 hours every 30 minutes. So if your system has very strong time patterns each day then this could be a limiting factor in how well the model could ever do (since only knows about last 4 hours). For some systems you might actually want this - you want the model to evolve and adapt as the system changes and new workloads come onto it etc. But for others, maybe web servers perhaps, you want to try take more advantage of the strong ‘time of day’ type effect. So it’s hard to really know up front what the best middle ground here is and how to make it easy for users to tweek for either setting/scenario. One thing you could do is increase
train_n_secsto be 24 hours and then i’d recommend usingtrain_max_nto cap the size of the training data. So instead the data would be samples from last 24 hours. This could help avoid the training step being too resource expensive on the agent. But again this can depend on the agent itself and number of charts/models you have it set up for, so can be a bit of trial and error and exploration involved here. -
At restart of the agent it will all start from scratch.
It could be cool to have a collector that just incrementally trains models over time in small batches - so it could be that every hour it somehow incrementally trains the model and maybe even pushes the new model as a new version so you could even have some model management type functionality. Some deep learning models and approaches as sometimes easier to do in this regard when you are just updating the latest weights for the model in some way. Or also some traditional ML models where you could ensemble more recent models with the ones you have already trained on a longer history of data.
The main idea to begin with was to try keep it as simple as we can get away with and see if the community finds any use in it.
Am super curious to hear how you get on with default settings after a couple of days of it running.