The Unit Mismatch That Broke the Offers Panel: Debugging Vast.ai's GPU RAM Filter

Introduction

In the middle of an ambitious infrastructure overhaul — transforming a basic monitoring tool into a comprehensive GPU instance deployment platform — the assistant hit an unexpected wall. The newly built Offers panel, designed to let operators search, filter, and deploy Vast.ai GPU instances directly from a web dashboard, was returning zero results. Not "zero matching results" — literally zero, as if the entire Vast.ai marketplace had evaporated. The assistant had just deployed the updated vast-manager binary to the controller host, verified the service was active, and confidently queried the new /api/offers endpoint only to receive an empty array. What followed was a methodical debugging session that culminated in a single, revelatory message: the discovery that Vast.ai's API uses different units for the same field depending on whether you're reading JSON or writing a filter query.

The subject message, <msg id=1278>, is deceptively short — just two lines of analysis followed by a bash command. But it represents the climax of a debugging arc that spanned over a dozen messages, and it encapsulates a class of bug that plagues every developer who has ever worked with an API that silently converts units between input and output. This article examines that message in depth: the reasoning that led to it, the assumptions that had to be broken, the knowledge it created, and the thinking process that made the discovery possible.

The Context: A New Panel, An Empty Response

To understand why this message was written, one must understand the stakes. The assistant had just completed a major UI overhaul of the vast-manager application — a Go-based management service running on a controller host at 10.1.2.104 that coordinates a fleet of GPU proving workers for the Curio/cuzk Filecoin proving pipeline. The overhaul added an interactive Offers panel with color-coded visual indicators, deploy buttons, host performance badges, and sophisticated filtering. This was not cosmetic: the Offers panel was the gateway to a fully automated deployment workflow where operators could search for cost-effective GPU instances and spin them up with a single click.

After building the updated binary and deploying it via SCP and systemd restart, the assistant ran the first integration test:

curl -s "http://localhost:1235/api/offers"

The response was {"total":0,"filtered":0} — zero offers. This was immediately suspicious because the assistant knew from previous work that the Vast.ai marketplace had plenty of suitable GPUs. The debugging process began in earnest.

The Debugging Trail: From Confusion to Clarity

The assistant's debugging approach was systematic and hypothesis-driven. Each message in the sequence tested a specific theory:

  1. Is the filter too restrictive? (msg 1263) The assistant tried a looser filter with URL-encoded spaces, still getting 0 results.
  2. Is the URL encoding wrong? (msg 1264) The assistant tried URL-encoded operators (%3E%3D for >=), still 0.
  3. Does the vastai CLI work at all? (msg 1265) The assistant ran the CLI directly on the controller host, confirming the same filter returned empty.
  4. Is the filter field name wrong? (msg 1269) The assistant dumped the raw JSON keys from an unfiltered search to verify field names. gpu_ram existed in the JSON output.
  5. What are the actual values? (msg 1270) The assistant printed the first 15 offers with their gpu_ram values. Every single one had gpu_ram well above 12500 — RTX PRO 6000 S cards showed 97887, RTX 5090 showed 32607, B200 showed 183359. The data was there.
  6. Is it a syntax issue with multiple filters? (msg 1271-1272) The assistant tried gpu_ram>=12500 alone, then added cuda_max_good>=13.0, still 0.
  7. Is it a CUDA version field mismatch? (msg 1272-1273) The assistant checked cuda_max_good values and confirmed they were present and correct in the JSON.
  8. Is it a filter syntax problem? (msg 1274-1275) The assistant confirmed num_gpus >= 1 worked (returning 64 offers), but gpu_ram > 12000 returned 0. This was the critical clue: a filter on a numeric field that should match every offer was returning nothing.
  9. Read the help documentation. (msg 1276-1277) The assistant finally checked vastai search offers --help and discovered the documentation: gpu_ram: float per GPU RAM in GB.

The Subject Message: Connecting the Dots

This brings us to <msg id=1278>. The message reads:

gpu_ram is in GB, not MB! The API JSON returns it in MB, but the filter uses GB. So gpu_ram>=12500 means 12500 GB which no GPU has. The correct filter is gpu_ram>12.5 (12.5 GB). Also cpu_ram is in GB in the filter.

>

Let me also check cuda_max_good vs cuda_vers: [bash] ssh 10.1.2.104 'vastai search offers --help 2>&1 | grep -i cuda'

The message is a moment of crystallized understanding. The assistant had been staring at JSON values like gpu_ram=97887 — which is 97,887 MB, or roughly 96 GB — and passing that same number (12500) as a filter, inadvertently requesting machines with 12,500 GB of GPU RAM. No such machine exists. The filter was mathematically correct but semantically wrong.

The assistant also flags a secondary issue: the CUDA version filter field. The JSON response uses cuda_max_good (a float like 13.0), but the filter documentation mentions cuda_vers as the correct field name. This is a second unit/name mismatch lurking in the same system.

The Thinking Process: What Made This Discovery Possible

The discovery in this message was not a lucky guess — it was the product of disciplined debugging. Several cognitive strategies are visible in the preceding messages:

Progressive narrowing. The assistant started with broad hypotheses (is the whole API broken?) and progressively narrowed to specific ones (is the filter syntax different for this field?). Each step ruled out a class of explanations.

Symmetric testing. When gpu_ram>=12500 returned 0, the assistant didn't just try different values — it tested a completely different field (num_gpus >= 1) to confirm the filter mechanism itself worked. This established that the problem was specific to the gpu_ram field, not the filtering infrastructure.

Raw data inspection. Rather than trusting the API's filter logic, the assistant dumped raw JSON from an unfiltered search and inspected actual values. This revealed that the data existed and was well-formed, forcing the conclusion that the filter interpretation was the problem.

Documentation as a last resort. Notably, the assistant did not reach for the documentation immediately. It spent several messages experimenting with the API directly, testing hypotheses through observation. Only after exhausting empirical approaches did it consult --help. This is a pragmatic tradeoff: experimentation is faster when you have a running system, but documentation provides the authoritative answer when empiricism fails.

Assumptions, Mistakes, and Lessons

The primary incorrect assumption was that the API would use consistent units between its JSON output and its filter input. This is a natural assumption — most well-designed APIs maintain unit consistency. Vast.ai's choice to return gpu_ram in megabytes while accepting filter values in gigabytes is a design decision that prioritizes precision in output (megabytes give integer values for typical GPU RAM sizes) while keeping filter input human-readable (gigabytes are more natural for human operators). But the lack of documentation on the output format — or at least a prominent note about the unit mismatch — creates a trap.

A secondary assumption was that cuda_max_good (the JSON field name) would be the filter field name. The help documentation revealed the filter uses cuda_vers. This kind of field-name drift between output schema and query schema is another common API pitfall.

The lesson is broader than this specific API: when building integrations with external systems, always verify that input and output schemas use the same units, field names, and data types. The assistant's mistake was not in writing the wrong filter — it was in assuming consistency without verification.

Knowledge Created

This message created several pieces of actionable knowledge:

  1. Unit conversion rule: Vast.ai's gpu_ram filter uses GB, while the JSON response uses MB. The conversion factor is 1024 (or approximately 1000 for rough estimates).
  2. Correct filter values: For an RTX 5090 with 32 GB RAM, the filter should be gpu_ram>=32 not gpu_ram>=32607. For an RTX PRO 6000 with 96 GB, it should be gpu_ram>=96 not gpu_ram>=97887.
  3. CUDA version filter field: The correct filter field is cuda_vers, not cuda_max_good. The JSON field cuda_max_good appears to be derived from the driver version but is not queryable.
  4. CPU RAM unit: cpu_ram in the filter also uses GB, not MB. This knowledge directly informed the next steps: the assistant would need to update the Offers panel's default filter string in both the Go backend and the JavaScript frontend to use GB-based values, and potentially add a note or automatic conversion in the UI to prevent operators from making the same mistake.

Conclusion

Message <msg id=1278> is a small message with a large impact. In two sentences and a bash command, it resolved a debugging session that had consumed over a dozen messages and threatened to undermine confidence in the entire Offers panel deployment. The discovery that Vast.ai's API uses gigabytes for filter input but megabytes for JSON output is the kind of silent inconsistency that can waste hours of developer time. The assistant's methodical approach — testing hypotheses, inspecting raw data, and finally consulting documentation — turned a frustrating zero-results mystery into a clear, actionable fix. For anyone building tools on top of external APIs, this message serves as a reminder: trust neither the input nor the output until you have verified the units match.