Monitor Consul Cluster With Azure Monitor

Daniel Merchant 6

I am successfully using Azure Monitor container insights Prometheus integration to monitor services deployed on our Azure Kubernetes Service (AKS) instances (e.g. RabbitMQ) and I want to monitor our Hashicorp Consul cluster in the same manner.
ref:
https://zcusa.951200.xyz/en-us/azure/azure-monitor/insights/container-insights-prometheus-integration
I have added annotations to the pod in the same way as working other k8s services, but I am not seeing the data get scraped.
"prometheus.io/scrape": "true"
"prometheus.io/port": "80"
"prometheus.io/path": "/v1/agent/metrics?format=prometheus"

The metrics are being exposed on the path and port specified and I can access them fine manually via the external IP. I have configured the config map to read the annotations and other services are correctly getting their metrics scraped.

Using native Prometheus monitoring, I would use the base URL "/v1/agent/metrics" and have to add a?format=prometheus URL parameter in the YAML config as a parameter.
Do I need add a something similar here with Azure Monitor? There is nothing I can find in the current documentation.
Consul requires the passing of this URL parameter to expose Prometheus style metrics.

Looking through the logs in the OMS we can see errors due to the URL encoding the URL parameter, which is not returning metrics.

2020-08-10T15:04:00Z E! [inputs.prometheus]: Error in plugin: error making HTTP request to http://10.x.x.x:80/v1/agent/metrics%3Fformat=prometheus: Get http://10.x.x.x:80/v1/agent/metrics%3Fformat=prometheus: dial tcp 10.x.x.x:80: getsockopt: connection refused

tbgangav-MSFT 10,416 Reputation points

2020-08-11T15:07:49.447+00:00
Hi @Daniel Merchant , Thanks for reaching out. I see two questions here.

Reason for data not getting scraped

Feedback to enhance this Azure document with details on what needs to be added in your case

Regarding #1, I am checking on it and will get back to you at the earliest.
Regarding #2, I recommend you to share the document enhancement feedback by clicking on "This page" option (shown in below screenshot) that's available at the bottom of this referred Azure document so that related document's author would review the feedback and consider enhancing the document as appropriate.
tbgangav-MSFT 10,416 Reputation points

2020-08-14T06:19:43.007+00:00

Hi @Daniel Merchant , I believe your Hashicorp Consul cluster is self-managed k8s cluster environment. May I know what version you are using here? Also, are you running this on k8s Azure Stack?
Vishwanath 96 Reputation points Microsoft Employee

2020-08-14T16:59:12.397+00:00

Based on the error (connection refused) it looks like config is incorrect, due to which its hitting an incorrect endpoint ?
When you say it works with ‘external ip’ , are you using the same ip as seen in the error (10.x.x.80) in the working case?
Can you share you config map if possible ?
tbgangav-MSFT 10,416 Reputation points

2020-08-17T07:16:17.573+00:00

Hi @Daniel Merchant , Did you get chance to review our earlier comments? Can you please provide the requested additional information so that we can assist you accordingly?
Daniel Merchant 6 Reputation points

2020-08-18T16:25:01.243+00:00
@tbgangav-MSFT apologies for the late reply, I have been out of the office.
Our Hashicorp Consul cluster in hosted in an Azure AKS instance.

The pod annotations on the pod are definitely correct and running kubectl describe, I can see the metrics path is reporting correctly:
"prometheus.io/path": "/v1/agent/metrics?format=prometheus"

I can browse to http://10.x.x.x:80/v1/agent/metrics?format=prometheus directly and see the metrics.
The error message suggests whatever is interpreting the pod annotations does not like the question mark.

Our config map is pretty straightforward and we are scraping metrics fine for our RabbitMQ cluster:

prometheus-data-collection-settings: |-
# Custom Prometheus metrics data collection settings
[prometheus_data_collection_settings.cluster]
#various comments
interval = "1m"

#various comments
Daniel Merchant 6 Reputation points

2020-08-18T16:27:29.2+00:00

We have implemented a successful workaround as adding each URL for each Consul instance in the ConfigMap yaml file for each AKS instance does successfully pick up the metrics. It looks like just the pod annotation method does not work.
e.g. this config map is working and scraping the metrics:

prometheus-data-collection-settings: |-
[prometheus_data_collection_settings.cluster]
urls = ["http://myconsulclusterdns.internal/v1/agent/metrics?format=prometheus"]

We would much prefer to use pod annotations as we can use a consistent configuration for each AKS instance, rather than maintaining a different configmap for each one (as the AKS URL for Consul will be different for each instance).
Vishwanath 96 Reputation points Microsoft Employee

2020-08-19T14:52:22.26+00:00

@Daniel Merchant - This is a bug in oss Telegraf that i will track & fix. Thanks for reporting this issue.
Daniel Merchant 6 Reputation points

2020-08-20T09:40:24.797+00:00

@Vishwanath Thank you for getting back to me.
I see the bug you raised https://github.com/influxdata/telegraf/issues/8008
Appreciate your time on this.

Share via

Monitor Consul Cluster With Azure Monitor

Your answer