-
Notifications
You must be signed in to change notification settings - Fork 183
Description
I was trying the latest build of trickster and noticed that grafana dashboards do not load correctly.
This is a little hard to describe, but with the current setup I have several downstream victoriametrics nodes that I try to aggregate via alb. See config at the end.
The problem is that when switching between Grafana dashboards it seems that metrics between dashboard loads are weirdly cached.
It's hard to describe but when loading a dashboard initially, say in a new browser session, I get the correct results for the dashboard variables, job
, instance
, etc.
But when I then browse to another dashboard, which can be completely unrelated I get the values from the old dashboard pre-filled in the new dashboard's variables e.g. when switching from victoria-metrics dashboard to the node-exporter dashboard it would have victoria-metrics
pre-filled for the dashboards job
selector.
This also seems weirdly sticky, refreshing the browser window (F5) or browsing other dashboards does not help.
The reasons which leads me to believe this is at least partially an issue with the trickster cache is that when I replace trickster with promxy it works immediately. No issues whatsoever.
I've also tried various Grafana settings (cache settings for the datasource, incremental queries, etc) and even Grafana versions -- no change. Similarly, changing the used cache in trickster does help either. I'ved tested redis (valkey) and memory.
Is there a way to completely disable caching in trickster itself so that it only acts as an aggregating proxy? That would probably be the easiest test to nail down if it's am caching issue or not.
Trickster version:
buildTime=2025-06-23T01:38:22+0000 commitID=753d85ef8a73ce54cc15819b81e55076911c2e2b
Sample config:
metrics:
listen_address: 127.0.0.1
listen_port: 8481
mgmt:
listen_address: 127.0.0.1
listen_port: 8484
frontend:
listen_address: 127.0.0.1
listen_port: 8480
health_handler_path: /-/health
caches:
default:
provider: redis
redis:
endpoint: 127.0.0.1:6379
backends:
default-backend:
provider: alb
alb:
mechanism: tsm
pool:
- default-backend-NODE01
- default-backend-NODE02
all-backends:
path_routing_disabled: true
provider: alb
alb:
mechanism: tsm
pool:
- default-backend
# Some other backends. Not relevant to the bug
default-backend-NODE01:
path_routing_disabled: true
tls:
insecure_skip_verify: true
provider: prometheus
origin_url: "https://default-backend-NODE01"
healthcheck:
timeout: 1000ms
interval: 2000ms
failure_threshold: 3
recovery_threshold: 3
default-backend-NODE02:
path_routing_disabled: true
tls:
insecure_skip_verify: true
provider: prometheus
origin_url: "https://default-backend-NODE02"
healthcheck:
timeout: 1000ms
interval: 2000ms
failure_threshold: 3
recovery_threshold: 3
# ...
I thought It could be related to some caching of VM itself so I tried various tricks to disable caching on a backend level.
default-backend-NODE02:
# ...
paths:
example1:
match_type: prefix
response_headers:
Cache-Control: no-cache
path: /
# https://github.com/jacksontj/promxy/issues/202
request_params:
+nocache: 1
Please let me know if you need any more information or help nailing this down.