Issue with label_values Function in AdditionalScrapeConfigs in Managed Grafana

yd7474 60 Reputation points
2023-07-12T11:48:30.29+00:00

I am encountering an issue when trying to use the label_values() function to create a Grafana variable based on the instance label in my managed Grafana, which is connected to a Prometheus instance with additional scrape configs set.

Here is an excerpt of my values.yaml file:

prometheus:
  prometheusSpec:
    additionalScrapeConfigs: 
      - job_name: 'Spring Boot Application input'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_name]
          target_label: instance
          action: replace

As per the above configuration, the __meta_kubernetes_pod_label_name label is replaced with instance as expected. When I run this query:

sum by (url, httpStatusCode, outcome, application, namespace, instance)  (floor(increase(spring_cloud_gateway_verbose_seconds_count{application="$application", instance="test", namespace="my-namespace", url=~"https://$url.*", httpStatusCode=~"2..|3.."}[1m])))

I get a response as expected, such as:

{application="Datos Application", httpStatusCode="200", instance="test", namespace="my-namespace", outcome="successful", url="https://$url.*/assign"} 

However, when I try to create a Grafana variable with all "instances" using label_values(instance), it works perfectly in a k8s self-hosted Grafana instance (of the prometheus-community. But when I try the same in a managed Grafana instance, I get an error:

{status: "error", errorType: "Not Found", error: "Unsupported route. RequestId: 681c3231bdf241529433f2640c8f8a31"}

Can anyone please provide guidance on why this might be happening in the managed Grafana and how I can successfully create this variable? I think it related to the grafana version (prometheus-community vs grafana enterprise).

Thanks in advance for your help.

Azure Managed Grafana
Azure Managed Grafana
An Azure service used to deploy Grafana dashboards for analytics and monitoring solutions.
106 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Brahmnes Fung 1 Reputation point Microsoft Employee
    2023-07-14T00:35:33.98+00:00

    With the Prometheus data source, the Unsupported route error message usually means Grafana made a call to an API to Prometheus and Prometheus doesn't support the API. You can check the data source configuration page to see what Prometheus version. Changing that version to match your Prometheus version, or even something lower, may help Grafana to not use the new API.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.