[Feature]: filter for labels like severity / silences are ignored

[mattttv]

Describe the feature request

Hi Guys!

first of all - thanks netways for this nice piece of monitoring software!

silences

I was wondering if there is any way to respect silences inside the alertmanager that is used by openshift.

include/exclude labels

Overmore we are trying to replace signalilo and therefore we are wondering if there could be way to monitor for labels like severity.

Although I have seen the parameter --group it does not seem that labels are used for grouping alerts.

By having a new parameter we could create different icinga-services that would filter for different labels (e.g. based on severity). It would be best if there could be a white and blacklist. Given the fact that prometheusrules could have arbitrary labels I propose two new repeatable parameters that can take labels and values

for instance --include-flag --exclude-flag
check_prometheus --include-flag severity=warning .. only show checks with severity = warning
check_prometheus --exclude-flag mylabel=foobar .. do not show checks with label mylabel=foobar

Thanks in advance,
Matt

[martialblog]

Hi, thanks for the feedback.

Regarding "silences". The plugin only talks to the Prometheus API, not to the Alertmanager since it's a separate tool with its own API.

Rather than this plugin talk to two different APIs, I think having something like a "check_alertmanager" would be nicer.

Regarding "include/exclude labels". That it something we could probably add here. I think all required data and functions should already be in the alerts.go.

Regards,
Markus

[martialblog]

Been thinking about the "silences" thing a bit. We could introduce an alertmanager subcommand that talk to the Alertmanager API.

[martialblog]

Did a little prototyping on filtering by labels here #114

Still need to work out a bit of the logic I think. I any one has thoughts on that let me know

[martialblog]

include/exclude labels will be in the next release.

We discussed adding Alertmanager support here #117

• but we're not sure how or where to add that... yet.