Skip to content

procedures: Update monitoring procedures for automated Prometheus resources #3105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tolusha wants to merge 1 commit into from
Closed

procedures: Update monitoring procedures for automated Prometheus resources #3105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions modules/administration-guide/pages/monitoring-che.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
:_content-type: ASSEMBLY
:description: Monitoring {prod-short} Server
:keywords: administration-guide, monitoring-che
:keywords: administration-guide, monitoring-che, prometheus, metrics
:navtitle: Monitoring {prod-short} Server
:page-aliases: .:monitoring-che.adoc, .:tracing-che.adoc, tracing-che.adoc

[id="monitoring-{prod-id-short}"]
= Monitoring {prod-short} Server

You can configure {prod-short} to expose JVM metrics such as JVM memory and class loading for {prod-short} Server.
You can configure {prod-short} to expose JVM metrics such as JVM memory and class loading for {prod-short} Server. The {prod-short} Operator automatically configures Prometheus monitoring resources when metrics are enabled.

include::partial$proc_enabling-and-exposing-che-metrics.adoc[leveloffset=+1]

Expand Down
4 changes: 2 additions & 2 deletions modules/administration-guide/pages/monitoring-the-dev-workspace-operator.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
:_content-type: ASSEMBLY
:description: Monitoring the {devworkspace} Operator
:keywords: administration-guide, monitoring-the-dev-workspace-operator
:keywords: administration-guide, monitoring-the-dev-workspace-operator, prometheus, metrics
:navtitle: Monitoring the {devworkspace} Operator
:page-aliases: .:monitoring-the-dev-workspace-operator.adoc


[id="monitoring-the-dev-workspace-operator"]
= Monitoring the {devworkspace} Operator

You can configure the OpenShift in-cluster monitoring stack to scrape metrics exposed by the {devworkspace} Operator.
The {prod-short} Operator automatically configures the OpenShift in-cluster monitoring stack to scrape metrics exposed by the {devworkspace} Operator.

include::partial$proc_collecting-dev-workspace-operator-metrics-with-prometheus.adoc[leveloffset=+1]

Expand Down
90 changes: 12 additions & 78 deletions .../administration-guide/partials/proc_collecting-che-metrics-with-prometheus.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,100 +3,34 @@
[id="collecting-{prod-id-short}-metrics-with-prometheus"]
= Collecting {prod-short} Server metrics with Prometheus

To use the in-cluster Prometheus instance to collect, store, and query JVM metrics for {prod-short} Server:
The {prod-short} Operator automatically configures Prometheus monitoring when {prod-short} Server metrics are enabled. The operator creates a ServiceMonitor, RBAC resources, and applies the necessary namespace labels for the in-cluster Prometheus instance to collect, store, and query JVM metrics.

Check failure on line 6 in modules/administration-guide/partials/proc_collecting-che-metrics-with-prometheus.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[CheDocs.Attributes] Use the AsciiDoc attribute {orch-namespace}, {platforms-namespace}, or {namespace} rather than ' namespace .'

Raw Output:
{"message": "[CheDocs.Attributes] Use the AsciiDoc attribute {orch-namespace}, {platforms-namespace}, or {namespace} rather than ' namespace .'", "location": {"path": "modules/administration-guide/partials/proc_collecting-che-metrics-with-prometheus.adoc", "range": {"start": {"line": 6, "column": 200}}}, "severity": "ERROR"}

.Prerequisites

* Your organization's instance of {prod-short} is installed and running in Red Hat OpenShift.

* An active `oc` session with administrative permissions to the destination OpenShift cluster. See link:https://docs.openshift.com/container-platform/{ocp4-ver}/cli_reference/openshift_cli/getting-started-cli.html[Getting started with the CLI].
* An active `{orch-cli}` session with administrative permissions to the destination {orch-name} cluster. See {orch-cli-link}.

* {prod-short} is exposing metrics on port `8087`. See xref:enabling-and-exposing-{prod-id-short}-metrics[Enabling and exposing {prod-short} server JVM metrics].

.Procedure

. Create the ServiceMonitor for detecting the {prod-short} JVM metrics Service.
. Verify that metrics are enabled in the `CheCluster` Custom Resource:
+
.ServiceMonitor
====
[source,yaml,subs="+quotes,+attributes,+macros"]
[source,yaml,subs="+attributes,+quotes"]
----
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: che-host
namespace: {prod-namespace} <1>
spec:
endpoints:
- interval: 10s <2>
port: metrics
scheme: http
namespaceSelector:
matchNames:
- {prod-namespace} <1>
selector:
matchLabels:
app.kubernetes.io/name: {prod-deployment}
components:
metrics:
enable: true
----
<1> The {prod-short} namespace. The default is `{prod-namespace}`.
<2> The rate at which a target is scraped.
====

. Create a Role and RoleBinding to allow Prometheus to view the metrics.

+
.Role
====
[source,yaml,subs="+quotes,+attributes,+macros"]
----
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: prometheus-k8s
namespace: {prod-namespace} <1>
rules:
- verbs:
- get
- list
- watch
apiGroups:
- ''
resources:
- services
- endpoints
- pods
----
<1> The {prod-short} namespace. The default is `{prod-namespace}`.
====

The {prod-short} Operator automatically creates the following resources in the `{prod-namespace}` namespace:
+
.RoleBinding
====
[source,yaml,subs="+quotes,+attributes,+macros"]
----
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: view-{prod-id-short}-openshift-monitoring-prometheus-k8s
namespace: {prod-namespace} <1>
subjects:
- kind: ServiceAccount
name: prometheus-k8s
namespace: openshift-monitoring
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: prometheus-k8s
----
<1> The {prod-short} namespace. The default is `{prod-namespace}`.
====

. Allow the in-cluster Prometheus instance to detect the ServiceMonitor in the {prod-short} namespace. The default {prod-short} namespace is `{prod-namespace}`.
+
[source,terminal,subs="+attributes,quotes"]
----
$ oc label namespace {prod-namespace} openshift.io/cluster-monitoring=true
----
* ServiceMonitor named `{prod-host}` for scraping {prod-short} Server metrics
* Role named `{prod-id-short}-prometheus` granting permissions to read services, endpoints, and pods
* RoleBinding named `{prod-id-short}-prometheus` binding the `prometheus-k8s` service account from the `openshift-monitoring` namespace
* Namespace label `openshift.io/cluster-monitoring=true` to enable cluster monitoring

.Verification

Expand Down
46 changes: 7 additions & 39 deletions ...de/partials/proc_collecting-dev-workspace-operator-metrics-with-prometheus.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,56 +2,24 @@
= Collecting {devworkspace} Operator metrics

[role="_abstract"]
To use the in-cluster Prometheus instance to collect, store, and query metrics about the {devworkspace} Operator:
The {prod-short} Operator automatically configures Prometheus monitoring for the {devworkspace} Operator. The operator creates a ServiceMonitor, RBAC resources, and applies the necessary namespace labels for the in-cluster Prometheus instance to collect, store, and query metrics.

Check failure on line 5 in modules/administration-guide/partials/proc_collecting-dev-workspace-operator-metrics-with-prometheus.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[CheDocs.Attributes] Use the AsciiDoc attribute {orch-namespace}, {platforms-namespace}, or {namespace} rather than ' namespace .'

Raw Output:
{"message": "[CheDocs.Attributes] Use the AsciiDoc attribute {orch-namespace}, {platforms-namespace}, or {namespace} rather than ' namespace .'", "location": {"path": "modules/administration-guide/partials/proc_collecting-dev-workspace-operator-metrics-with-prometheus.adoc", "range": {"start": {"line": 5, "column": 187}}}, "severity": "ERROR"}

.Prerequisites

* Your organization's instance of {prod-short} is installed and running in Red Hat OpenShift.

* An active `oc` session with administrative permissions to the destination OpenShift cluster. See link:https://docs.openshift.com/container-platform/{ocp4-ver}/cli_reference/openshift_cli/getting-started-cli.html[Getting started with the CLI].
* An active `{orch-cli}` session with administrative permissions to the destination {orch-name} cluster. See {orch-cli-link}.

* The `devworkspace-controller-metrics` Service is exposing metrics on port `8443`. This is preconfigured by default.

.Procedure

. Create the ServiceMonitor for detecting the Dev Workspace Operator metrics Service.
. Verify that {prod-short} is installed. The {prod-short} Operator automatically creates the following resources in the `{prod-namespace}` namespace:
+
.ServiceMonitor
====
[source,yaml,subs="+quotes,+attributes,+macros"]
----
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: devworkspace-controller
namespace: {prod-namespace} <1>
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 10s <2>
port: metrics
scheme: https
tlsConfig:
insecureSkipVerify: true
namespaceSelector:
matchNames:
- openshift-operators
selector:
matchLabels:
app.kubernetes.io/name: devworkspace-controller
----
<1> The {prod-short} namespace. The default is `{prod-namespace}`.
<2> The rate at which a target is scraped.
====

include::example$snip_{project-context}-create-a-role-and-rolebinding-for-prometheus-to-view-metrics.adoc[]

. Allow the in-cluster Prometheus instance to detect the ServiceMonitor in the {prod-short} namespace. The default {prod-short} namespace is `{prod-namespace}`.
+
[source,subs="+attributes"]
----
$ oc label namespace {prod-namespace} openshift.io/cluster-monitoring=true
----
* ServiceMonitor named `devworkspace` for scraping {devworkspace} Operator metrics from the `openshift-operators` namespace
* Role named `{prod-id-short}-prometheus` granting permissions to read services, endpoints, and pods in the `openshift-operators` namespace
* RoleBinding named `{prod-id-short}-prometheus` binding the `prometheus-k8s` service account from the `openshift-monitoring` namespace
* Namespace label `openshift.io/cluster-monitoring=true` to enable cluster monitoring

.Verification

Expand Down
Loading