The Folder That Forbids: Debugging Grafana Anonymous Access with RBAC Scope Analysis

In the middle of an intensive engineering session deploying DeepSeek-V4 on Blackwell GPUs with SGLang's PD-disaggregated architecture, a seemingly simple error message arrived from the user: "Failed to load dashboard — Forbidden in grafana." The assistant had just finished building an extensive monitoring stack — a 29-panel Grafana dashboard with node-health metrics, GPU utilization, KV cache usage, and HiCache hierarchical caching stats — and had successfully uploaded it via the admin API. Yet when the user tried to view it, Grafana returned a 403. The message at index 13102 is the moment of diagnosis and resolution: a crisp, elegant fix born from layered investigation.

The Subject Message

The assistant's message reads:

## Agent Reasoning
The issue is that the anonymous Viewer's `dashboards:read` permission is restricted to a specific folder (`efphzcu8r0agwa`), but my dashboard lives in the General folder, so the anonymous user gets a 403 error. Someone must have previously set up this scoped permission for the anonymous account.

The solution is straightforward: I need to move my dashboard into the "sglang" folder by re-uploading it with the correct `folderUid` parameter set to `efphzcu8r0agwa`. I'll POST to the dashboards API with `overwrite: true` and the new folder UID, which should move the existing dashboard and make it readable by the anonymous user.
**Root cause:** the anonymous Viewer's `dashboards:read` is scoped **only to the `sglang` folder** (`folders:uid:efphzcu8r0agwa`) — but my upload landed the dashboard in **General**, which anon can't read → 403. The original dashboard lived in the `sglang` folder. Fix: move my dashboard into that folder (no config/restart needed).

The message then executes a bash command that SSHes into the remote machine, re-uploads the dashboard JSON with folderUid set to efphzcu8r0agwa, and verifies the result. The API returns success, the page now loads with HTTP 200, and the problem is resolved.

The Reasoning and Motivation

Why was this message written? The assistant had been on a multi-hour journey building and deploying production ML infrastructure. It had just completed a major Grafana dashboard update — version 3 with 29 panels covering node health, GPU metrics, and HiCache — and uploaded it successfully. The user's report of "Forbidden" was the first signal that something was wrong with the deployment.

The assistant's reasoning process reveals a methodical diagnostic arc. Earlier messages ([msg 13098] through [msg 13101]) show the assistant exploring multiple hypotheses:

  1. Anonymous access disabled: Checked grafana.ini → found enabled = true with org_role = Viewer. Hypothesis rejected.
  2. Organization mismatch: Checked org assignment → anonymous user is in Main Org (id=1), same as the dashboard. Hypothesis rejected.
  3. Grafana 13 RBAC issue: Considered that the Viewer role might lack default dashboard read permissions. This was partially correct but not the full story.
  4. Config change needed: Considered bumping anonymous role to Editor or restarting Grafana. This was the fallback plan. The critical insight came in [msg 13101] when the assistant queried the anonymous user's effective permissions via /api/access-control/user/permissions. The response was revelatory: dashboards:read scopes: ['folders:uid:efphzcu8r0agwa']. This single line of data transformed the diagnosis. The anonymous Viewer's read permission was not global — it was scoped to exactly one folder, identified by UID efphzcu8r0agwa. The assistant then cross-referenced this with the earlier search results showing that efphzcu8r0agwa was the UID of the "sglang" folder. The dashboard, uploaded with folderId:0 (the General folder), fell outside this scope.

How Decisions Were Made

The decision process in this message is a masterclass in choosing the minimal intervention. The assistant faced several options:

Assumptions and Their Validity

The assistant operated under several assumptions, most of which were correct:

  1. The dashboard was uploaded to the General folder: This was confirmed in [msg 13100] via the admin API query showing folderTitle: General. Correct.
  2. The anonymous user's permissions were intentionally scoped: The assistant assumed "someone must have previously set up this scoped permission." This is a reasonable inference — Grafana 13's RBAC system doesn't create folder-scoped permissions by default. The fact that the anonymous Viewer had a non-default scope suggests prior configuration, possibly by the user or an earlier automated setup.
  3. Moving the dashboard to the sglang folder would fix the 403: This was the core hypothesis, and it was validated when the page returned 200 after the move. Correct.
  4. The folder UID efphzcu8r0agwa corresponded to the "sglang" folder: This was inferred from the search results in [msg 13098], which showed that UID with title "sglang" and type "dash-folder". Correct. One subtle assumption worth examining is that the API returning 403 after the move was acceptable. The assistant's verification shows anon read sglang-kv now: 403 but anon render page /d/sglang-kv: 200. The assistant didn't flag this discrepancy, likely understanding that the direct API endpoint requires specific authentication headers while the page endpoint serves HTML that the browser can render. In practice, the Grafana frontend makes API calls with the browser's credentials (or lack thereof), and if the anonymous session is properly configured, those calls should succeed. The 403 on the raw API call might be an artifact of curl not sending the right cookies or headers, while the browser-based page load works correctly.

Input Knowledge Required

To fully understand this message, one needs:

  1. Grafana architecture: Understanding that Grafana has organizations, folders, dashboards, and a role-based access control (RBAC) system. Knowledge that anonymous access can be enabled with a specific role and organization.
  2. Grafana API: Familiarity with the /api/dashboards/db endpoint for creating/updating dashboards, the folderUid parameter for placement, and the /api/access-control/user/permissions endpoint for checking effective permissions.
  3. Grafana 13 RBAC specifics: In Grafana 13 (released in 2024), RBAC was overhauled with fine-grained permissions. The dashboards:read action can be scoped to specific folders, which is exactly what the assistant discovered. Understanding this model was crucial to interpreting the ['folders:uid:efphzcu8r0agwa'] scope.
  4. The broader deployment context: The assistant was working on a PD-disaggregated SGLang deployment for DeepSeek-V4 on 8 Blackwell GPUs. The Grafana dashboard was part of a monitoring stack that included Prometheus scraping SGLang metrics and GPU stats. The "sglang" folder in Grafana was presumably created earlier in the session to organize these dashboards.
  5. SSH and remote execution: The fix was applied by SSHing into the remote machine (root@10.1.230.171) and running Python scripts that interact with the local Grafana API.

Output Knowledge Created

This message produced several valuable outputs:

  1. A working Grafana dashboard accessible to anonymous users: The immediate practical outcome. The user could now view the monitoring dashboard without authentication.
  2. A documented diagnostic pattern: The sequence of checks — config file → org assignment → effective permissions → folder placement — forms a reusable template for debugging Grafana access issues. Future engineers encountering similar 403 errors can follow this same chain.
  3. Confirmation of Grafana 13's RBAC behavior: The message empirically demonstrates that anonymous Viewer permissions can be folder-scoped, and that uploading to the General folder bypasses those scopes. This is a subtle but important behavioral detail that might not be obvious from documentation alone.
  4. A minimal, zero-downtime fix pattern: The solution required no service restarts, no config file edits, and no permission model changes. It's a model for how to resolve infrastructure issues with surgical precision.

The Thinking Process

The assistant's reasoning, visible in the "Agent Reasoning" sections across messages [msg 13098] through [msg 13102], follows a clear arc:

Stage 1 — Initial triage ([msg 13098]): The assistant suspects anonymous access configuration. It checks Grafana version (13.0.2), locates the config file, and tests API access. The key finding: anonymous gets 403 on dashboard read, admin gets 200.

Stage 2 — Config investigation ([msg 13099]): The assistant finds the Grafana process and its config at /opt/grafana/conf/custom.ini. It discovers the dashboard is in the General folder. No environment variable overrides are present.

Stage 3 — Anonymous config analysis ([msg 13100]): The assistant reads custom.ini and finds [auth.anonymous] with enabled = true and org_role = Viewer. The defaults.ini shows org_name = Main Org. as inherited. The anonymous user's org query returns org 1 (Main Org.). Everything seems correctly configured, yet the 403 persists. This is the moment of productive confusion — the configuration says anonymous should work, but it doesn't.

Stage 4 — Permission introspection ([msg 13101]): The assistant pivots from config analysis to permission introspection. It lists all dashboards (only one: sglang-kv in General folder), tests anonymous access to it (403), and then — crucially — queries the anonymous user's effective permissions. The result is the smoking gun: dashboards:read scopes: ['folders:uid:efphzcu8r0agwa'].

Stage 5 — Resolution ([msg 13102]): With the root cause identified, the assistant executes the fix. The reasoning is concise and confident — no hedging, no alternative plans. The diagnosis is complete, and the solution is obvious.

Mistakes and Nuances

Were there any mistakes? The most notable near-miss is in [msg 13101], where the assistant considers bumping the anonymous role to Editor and restarting Grafana. This would have worked (Editors have broader read access) but would have been over-engineering — granting unnecessary write permissions and requiring a service restart. The assistant wisely deferred this approach and instead gathered more data by checking effective permissions. The decision to run the permission introspection query before making config changes was the turning point.

Another subtle point: the assistant initially assumed the dashboard was uploaded to the General folder because it used folderId:0 in the upload. But the earlier upload in [msg 13095] didn't specify any folder parameter — it just posted the dashboard JSON. In Grafana's API, omitting folderId or folderUid defaults to the General folder. The assistant's understanding was correct.

The message also reveals an assumption about prior configuration: "Someone must have previously set up this scoped permission for the anonymous account." This is likely true, but the assistant doesn't investigate why the scope exists or whether it was intentional. In a production security audit, one might want to understand whether the folder-scoped anonymous access was a deliberate choice or a misconfiguration. However, for the immediate goal of making the dashboard visible, the fix is appropriate.

Conclusion

The message at index 13102 is a compact demonstration of systematic debugging in action. It transforms a vague error report ("Forbidden") into a precise diagnosis (folder-scoped RBAC permission) and applies a minimal fix (re-upload with correct folder). The assistant's reasoning moves through layers of abstraction — from network-level checks (HTTP status codes) through application configuration (grafana.ini) to the RBAC permission model itself — before converging on the single root cause. The fix requires no service restarts, no config changes, and no security model alterations, only a corrected API call. It's a small moment in a long engineering session, but it encapsulates the essence of good infrastructure work: understand the system deeply enough to find the simplest path to resolution.