How to Spot a Fake or Degraded LLM API

You are paying for GPT-5 or Claude Opus. Are you actually getting it? When you call a model directly from OpenAI, Anthropic or Google, the answer is almost always yes. When you buy the same model through a cheap reseller, an aggregator, or a “unified API” that undercuts the first party by a wide margin, the answer is sometimes no. This guide explains the ways an endpoint can quietly serve you something other than the model on the label, and gives you concrete tests to catch it.

What “fake” and “degraded” actually mean

There is rarely a literal fake model pretending to be GPT-5 from scratch. The real problem is more subtle, and it comes in four common forms.

• Model substitution. The endpoint accepts a request for an expensive model and answers with a cheaper one. You ask for a frontier model and get a small open model that is far less capable, because the margin is better.
• Silent quantization. The same model name is served at lower numerical precision, for example int4 or fp8 instead of the full-precision weights. It is genuinely the right model, but a compressed version that is cheaper to run and measurably weaker on hard tasks.
• Distilled or fine-tuned stand-ins. A smaller model trained to imitate the target on common prompts. It looks convincing on easy questions and falls apart on the hard ones the imitation never covered.
• Opaque routing. An aggregator routes your call to whichever upstream provider is cheapest at that moment. Some of those upstreams may themselves be quantized or throttled, so quality changes call to call without any warning.

Why it happens

The economics push in one direction. Frontier inference is expensive, and a reseller that promises prices well below the first party has to make the difference somewhere. Honest resellers make it on volume, better infrastructure, or by hosting open models transparently. The dishonest ones make it by shipping you less than you paid for and betting you will not measure it. Most buyers never do, which is exactly why measuring is worth your time.

Symptoms you can notice without any tooling

Before running formal tests, a few patterns are worth paying attention to in normal use.

• Quality collapses on hard prompts. Easy questions look fine, but multi step reasoning, careful instruction following, code, or long context are noticeably worse than you remember from the first party.
• Throughput that does not fit. Suspiciously high tokens per second can mean a smaller or heavily quantized model. Suspiciously slow and unstable latency can mean an overloaded shared box behind a thin resale layer.
• Inconsistency between identical calls. The same prompt at the same settings swings in quality or style much more than temperature alone would explain. That is a classic sign of routing across different backends.

Concrete tests you can run

None of these is a single perfect detector. Together they build a strong picture, especially when you compare the suspect endpoint against a first-party call to the same model as your control.

1. Known-answer probes

Build a small set of prompts with stable, hard-to-fake answers: a tricky reasoning puzzle, a precise coding task with a known correct output, a niche factual question, a format instruction that weaker models routinely break. Run them against the suspect endpoint and against the real first-party model. A cheaper stand-in tends to pass the easy items and fail the ones that separate a frontier model from a small one.

2. Cross-check the tokenizer and token counts

Different model families tokenize text differently. If an endpoint reports usage numbers that do not match the tokenizer the claimed model should use, that is a signal worth chasing. You can sanity-check the expected counts for OpenAI models with the Token Counter and see the exact splits in the Tokenizer Playground, then compare against the usage the API returns.

3. Look for fingerprints and logprobs

First-party OpenAI responses include a system_fingerprint that changes with the backend configuration, and can return logprobs for each token. The token probability distribution of a full-precision model differs from a quantized or substituted one, so logprobs are a real fingerprint. Two things to watch: many resellers strip these fields entirely, and a missing or always-identical fingerprint is itself a yellow flag.

4. Baseline latency and throughput

Measure time to first token and sustained tokens per second under a fixed prompt, several times, and compare against the first party and against what the hardware story implies. A provider claiming a huge frontier model at open-model speeds is claiming something that usually does not hold unless the model has been shrunk.

5. Stress the long context

Substituted and heavily quantized models often keep a large advertised context window but lose the ability to actually use it. Place a specific fact deep inside a long input and ask for it back. Real frontier models retrieve it reliably. Cheaper stand-ins start missing it well before the advertised limit.

6. Check behavior consistency over time

Run the same probe set on a schedule. A trustworthy endpoint stays stable. One that routes opportunistically will drift, and you will see quality move around in a way a fixed model would not.

Red flags before you even test

Some warning signs show up in the marketing and terms, not the outputs.

• Prices far below the first party with no explanation of how.
• No clear statement of whether models are full precision or quantized.
• Stripped response metadata, no fingerprint, no logprobs, no honest usage accounting.
• Vague or missing data-retention and training terms.
• Prepaid credits only, with no refund path if quality is not what was promised.

If you are weighing a specific reseller, the companion guide on vetting cheap LLM API resellers turns these into a full checklist, and the provider comparison shows which gateways are transparent about fees and model hosting.

A simple routine

You do not need a lab. A practical version is: keep a fixed set of ten probes that mix easy and hard, always call a first-party model as your control, run both, and watch known-answer accuracy, long-context retrieval, throughput, and any fingerprint or logprob fields. If the suspect endpoint tracks the control, trust it. If it wins on price but loses on the hard items, you have found the catch. Measuring once at onboarding and then periodically is enough to catch the majority of substitution and silent quantization.

The single most useful habit is simply to keep a control. The moment you can compare a suspect endpoint against a call you know is real, most of these problems become visible.