Page tree
Skip to end of metadata
Go to start of metadata

Overview

Observability is a must in complex platforms made up of multiple components. Also Cloud Native platforms should be aware of resource usage to scale up or down each component and to adapt to dynamic loads without incur in overcharges.

We have selected Prometheus as the monitoring technology for the Cloud Native PNDA platform, since it became the monitoring reference in the cloud space.

Getting Started

We will setup a local deployment of PNDA to test how observability works. Follow the Testing locally with microk8s' instructions, using the next yaml profile:

## ----------------------------
## PNDA Components Configuration
## ----------------------------
monitoring:
  prometheusOperator:
    enabled: true

 To install prometheus Operator in microk8s cluster, enable the prometheus addons:

microk8s.status
sudo microk8s.enable prometheus 

Prometheus Operator components are deployed by default in the 'monitoring' namespace.

You can get the microk8s prometheus addon url with:

kubectl get endpoints -o wide -n monitoring prometheus-k8s

Accessing the web url, PNDA targets should be listed in Status > Targets: 

The grafana UI of the microk8s prometheus addon url is:

kubectl get endpoints -o wide -n monitoring grafana

 You can import PNDA example dashboards provided in grafana-dashboards folder.

Developer Information

Exporters

Each component deploys one or several exporters to provide metrics to prometheus. A default configuration for each exporter is provided and it can be customized through a profile yaml. Detailled information of each component exporter is provided.

HDFS Component

To monitoring hdfs we have added the following files to the hdfs chart:

  • hdfs-nn-exporter-configmap.yaml (new), provides the configuration file for the  "Prometheus Hadoop HDFS FSImage Exporter".
  • hdfs-nn-exporter-service.yaml (new), provides a k8s service pointing to the exporter's endpoints.
  • hdfs-nn-statefulset.yaml (modified).

The hdfs-nn-statefulset.yaml modification is to add another container to the hdfs-nn-statefulset.yaml template, thus creating a sidecar container. The sidecar container runs a "Prometheus Hadoop HDFS FSImage Exporter" and It is responsible for collecting metrics given by the hadoop 'fsimage' and serving them at the 5556 port. To enable the sidecar container access to the fsimage, it should share the 'dfs' volume with the namenode container. The sidecar container must also mount the hdfs-nn-exporter-configmap to access FSImage Exporter configuration file.

Kafka

To monitoring kafka we have added the following files to the cp-kafka chart:

  • exporter-configmap.yaml (new),  provides the configuration file for the  "JMX Exporter".
  • kafka-exporter-service.yaml (new) provides a k8s service pointing to the exporter's endpoints.
  • statefulset.yaml (modified).

We use the jmx_exporter to get metrics from kafka. We have two options with the jmx_exporter:

  • Run as JavaAgent  exposing a HTTP server and serving metrics of the local JVM.
  • Run as an independent HTTP server scraping remote JMX targets.

jmx_exporter documentation strongly encourages the use of the exporter as JavaAgent. In this case the JavaAgent must be set in the JAVA_OPTS when running kafka. On the other hand, running the jmx_exporter as an independent Http server allow us to run it as a sidecar container close to the kafka container.

We have follow jmx_exporter documentation advice and we use jmx_exporter as a JavaAgent. To do so, we inject the jmx_exporter.jar through a init container and set the following -javaagent flag when launching the java application:

-javaagent:/jmx-exporter/jmx_prometheus_javaagent.jar={{ .Values.prometheus.port }}:/etc/exporter/jmx-kafka-prometheus.yml

In this case we add the javaagent flag through the KAFKA_OPTS environment variable in the kafka container (see cp-kafka/templates/statefulset.yaml) if .Values.prometheus.enabled is set in the kafka chart (default behavior).

HBase

To monitor hbase we have added the following files to the hbase chart:

  • exporter-configmap.yaml (new), provides the configuration file for the  "JMX Exporter".
  • hbase-master-exporter-service.yaml (new), provides a k8s service pointing to the hbase-master's exporter endpoints.
  • hbase-region-exporter-service.yaml (new), provides a k8s service pointing to the hbase-region's exporter endpoints.
  • hbase-master-statefulset.yaml (modified).
  • hbase-region-statefulset.yaml (modified).

We use the jmx_exporter to get metrics from hbase-master-statefulset and hbase-region-statefulset. We use jmx_exporter as JavaAgent, the same we did for kafka.

The modifications in hbase-master-statefulset and hbase-region-statefulset are similar to what was done in kafka. Create an init container to inject the jmx_exporter jar and set the following -javaagent flag when launching the java application: 

-javaagent:/jmx-exporter/jmx_prometheus_javaagent.jar={{ .Values.prometheus.port }}:/etc/exporter/jmx-kafka-prometheus.yml

In this case we add the -javaagent flag to the HBASE_OPTS environment variable in bootstrap.sh script (see hbase/templates/hbase-configmap.yaml)  if .'Values.prometeus.enable' is set in the hbase chart (default behavior).

Spark

To carry out the monitoring of Spark with prometheus there are several problems. Among them, two could be highlighted:

  • Spark issues its metrics using "Push" → Prometheus collects the metrics using "Pull".
  • Spark has metrics based on a library called "Dropwizard" → Prometheus is not compatible with Dropwizard.

To solve the first problem, prometheus could be configured to use the pushgateway. However, the pushgateway does not support receiving data with timestamp and, therefore, we would have to add this when the prometheus made a pull to the pushgateway. 

As for the second problem, prometheus could solve it using DropwizardExports.

Reference: https://banzaicloud.com/blog/spark-monitoring/.

Solution used in PNDA

A possible solution to previous problems is to use Spark support of Graphite. We could export Spark metrics to Graphite and, in turn, from Graphite to Prometheus. All this will be possible thanks to prometheus / graphite_exporter. The idea is tell Spark to export its metrics in Graphite and use the graphite exporter to export them to Prometheus.

We follow these steps:

  1. Configure Spark to export your metrics to Graphite. → We will configure a volume, based on a configmap called "pnda-spark-exporter", which contains the file 'metrics.properties'. The latter configures Spark to export the metrics to Graphite, the host, the port where they will be exported (9109), etc. This file will be mounted in the / opt / spark / conf directory (default location of the Spark configuration) of the Spark itself.
  2. Create container with graphite exporter. → After configuring the Spark, we will create the second container with the exporter. In the image of this exporter, the Graphite metric listening port is specified, which is just 9109, that is, the exporter listens to the port to which Spark sends the metrics. Once this is done, Spark metrics can be viewed at the address <ip-pod>: 9108.
  3. Map metrics to Prometheus. → Finally, Graphite metrics must be transformed to Prometheus format in order to create queries and analyze the data. To do this, a volume based on the configmap called "pnda-spark-graphite-mapping" has been created. This configmap contains the file 'graphite_mapping.conf', which defines the "translations" to be made from Graphite to Prometheus. The file will be mounted in / tmp, and it will only have to be passed to the container through the argument "--graphite.mapping-config".

With this, and connecting the ServiceMonitor with the service of the exporter we can obtain the metrics with Prometheus.

Monitoring with Prometheus Operator

Prometheus Operator provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus instances. 

A diagram of Prometheus Operator architecture:


Each component of the PNDA provides a Service for metrics scraping with a label: "monitoring: prometheus". PNDA-helm-chart enables all these services by default, but it can be customized by component in the profile yaml (component_name.prometheus).

The ServiceMonitor Resource to autodiscover component services is also enabled by default ("monitoring.prometheusOperator.enabled=true) in pnda-helm-chart.



  • No labels