Skip to content

Switching dashboard does not refresh dashboard variables #858

@pahaeanx

Description

@pahaeanx

I was trying the latest build of trickster and noticed that grafana dashboards do not load correctly.

This is a little hard to describe, but with the current setup I have several downstream victoriametrics nodes that I try to aggregate via alb. See config at the end.

The problem is that when switching between Grafana dashboards it seems that metrics between dashboard loads are weirdly cached.

It's hard to describe but when loading a dashboard initially, say in a new browser session, I get the correct results for the dashboard variables, job, instance, etc.
But when I then browse to another dashboard, which can be completely unrelated I get the values from the old dashboard pre-filled in the new dashboard's variables e.g. when switching from victoria-metrics dashboard to the node-exporter dashboard it would have victoria-metrics pre-filled for the dashboards job selector.

This also seems weirdly sticky, refreshing the browser window (F5) or browsing other dashboards does not help.

The reasons which leads me to believe this is at least partially an issue with the trickster cache is that when I replace trickster with promxy it works immediately. No issues whatsoever.

I've also tried various Grafana settings (cache settings for the datasource, incremental queries, etc) and even Grafana versions -- no change. Similarly, changing the used cache in trickster does help either. I'ved tested redis (valkey) and memory.

Is there a way to completely disable caching in trickster itself so that it only acts as an aggregating proxy? That would probably be the easiest test to nail down if it's am caching issue or not.

Trickster version:
buildTime=2025-06-23T01:38:22+0000 commitID=753d85ef8a73ce54cc15819b81e55076911c2e2b

Sample config:

metrics:
    listen_address: 127.0.0.1
    listen_port: 8481
mgmt:
    listen_address: 127.0.0.1
    listen_port: 8484
frontend:
    listen_address: 127.0.0.1
    listen_port: 8480
    health_handler_path: /-/health

caches:
  default:
    provider: redis
    redis:
        endpoint: 127.0.0.1:6379

backends:
    default-backend:
        provider: alb
        alb:
            mechanism: tsm
            pool:
                - default-backend-NODE01
                - default-backend-NODE02
    all-backends:
        path_routing_disabled: true
        provider: alb
        alb:
            mechanism: tsm
            pool:
                - default-backend
                # Some other backends. Not relevant to the bug

    default-backend-NODE01:
        path_routing_disabled: true
        tls:
            insecure_skip_verify: true
        provider: prometheus
        origin_url: "https://default-backend-NODE01"
        healthcheck:
            timeout: 1000ms
            interval: 2000ms
            failure_threshold: 3
            recovery_threshold: 3
    default-backend-NODE02:
        path_routing_disabled: true
        tls:
            insecure_skip_verify: true
        provider: prometheus
        origin_url: "https://default-backend-NODE02"
        healthcheck:
            timeout: 1000ms
            interval: 2000ms
            failure_threshold: 3
            recovery_threshold: 3
    # ...

I thought It could be related to some caching of VM itself so I tried various tricks to disable caching on a backend level.

    default-backend-NODE02:
        # ...
        paths:
            example1:
                match_type: prefix
                response_headers:
                    Cache-Control: no-cache
                path: /
                # https://github.com/jacksontj/promxy/issues/202
                request_params:
                    +nocache: 1

Please let me know if you need any more information or help nailing this down.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions