LLM API comparison template: reproducible results and verifiable costs

Bottom line: To fairly compare two LLM API entrances, conditions other than Base URL and API Key must be fixed, sampling repeated over multiple periods, and compatibility, quality, stability, Token, and billing recorded separately. If you only compare a speed test screenshot or a fixed question, the conclusion cannot be reproduced.

Define the decision problem before comparing

Different businesses require different weights. First write down clearly what this test will decide:

• Common Conversation: Focus on success rate, first byte latency, context, and price.
• Code Agent: Focus on real protocol requirements such as tool invocation, structured parameters, long-term task stability, and Responses API.
• Content production: Focus on constraint compliance, factual errors, long-form consistency, and manual rework time.
• Batch processing: Focus on throughput, 429s, retry costs, billing verifyability, and failure recovery.

Without clear business goals, overall scores can easily mask real blockers.

control variables

In addition to the service address and key, the following conditions must be consistent:

If the endpoints return non-equivalent model IDs or support different protocols, treat them as different products rather than presenting them as a like-for-like model comparison.

How to complete an AIFast HK comparison record

This is a blank record template, not completed speed tests or billing results. Follow the HK quick start to obtain the site's key and model ID, then alternate requests to both endpoints with the same script.

Chat Completions samples from the HK model-check tool do not establish full Responses or Anthropic Messages compatibility. For agents, also follow the Codex checklist or the Claude Code checklist to run native-client tasks. If model mappings or price units cannot be confirmed as equivalent, retain both result sets without claiming the same model is cheaper.

Turn HK check reports into comparable records

A round of HK checks and a single model call are different units of measurement. Use the table below to interpret the report before recording your own requests. Leave unmeasured data blank; do not infer it from the overall score.

HK Docs sends probes from its edge service, not directly from the user's computer. Quick mode usually attempts 3 requests and standard mode 7, including the model directory request; the count is not the number of successful chats. Model generation may incur charges, so use a temporary limited key. If the round deadline is reached, fewer requests may be sent than attempted.

Download the blank comparison CSV. It contains column names only, with no prefilled scores, account access or result uploads. Fill one row per round, recording the mode and token completeness before linking your saved, redacted evidence. For scoring and export limits, see the HK model-check report guide.

To measure single-chat latency, separately record the first valid content event and completion time using the same client. Page load time, HTTP time to first byte and model time to first content token are different metrics. For cost comparisons, reconcile HK usage with matching time zones, groups, full model IDs and request IDs. Keep unmatched charges separate rather than forcing token totals to fit.

Suggested four-level question set

Layer One: Protocol Smoke

Verify /models, minimal chat, error response,model、usage, SSE and tool calls. When the agreement does not meet business requirements, there is no need to continue to expand the sample.

Level 2: Dynamic ability questions

Each round uses a local script to generate random numbers, random field names or random nonces, check calculations, constraint compliance and tool parameter return. Dynamic questions can reduce fixed answer pollution, but cannot prove the identity of the underlying model.

The third level: public tasks

Prepare automatic judgment tasks such as code repair, information extraction, long text constraints, JSON Schema and multi-round dialogues. Questions, expected results, and evaluators must be versioned.

Level 4: Real business samples

Extract and desensitize historical failures. Record task completion rate, manual modification time and business interruption reasons. Real samples are most valuable for decision-making, but they cannot leak user data or internal keys.

Original record template

Keep one row of raw data per request:

run_id,provider,started_at,model_requested,model_returned,http_status,first_byte_ms,total_ms,input_tokens,output_tokens,total_tokens,retry_count,probe_id,passed,error_type,request_id

Create another test configuration file:

suite_version: "2026-07-14.1"
temperature: 0
max_tokens: 512
timeout_seconds: 60
retries: 0
stream: false
sample_windows:
  - low_peak
  - high_peak
  - next_day

Do not record the complete API Key, non-desensitized prompt words, user files, or personal information in the response. Request IDs should also be confirmed not to contain sensitive data before being made public.

Indicators to be calculated

Availability

• Raw success rate: The proportion of first requests that return valid results.
• Final success rate: including the success ratio after limited retries.
• Error distribution: 401, 429, 5xx, timeout, and protocol parsing errors are counted separately.

Don't count success after infinite retries as entry stability. First-time failure rate determines true latency and additional costs.

delay

• Record HTTP time to first byte and model time to first content token separately: receiving HTTP bytes or an SSE heartbeat does not mean the user has received model content.
• Total time taken: How long it takes for the task to be completed completely.
• P50/P95: Observe typical experience and tail slow requests respectively.

Capability quality

• Constraint compliance rate.
• JSON or tool parameter parsing success rate.
• Code test pass rate.
• Real business one-time completion rate.
• Average manual rework minutes.

Token and cost

Summarize input, output, failed requests and retry tokens respectively. Then calculate based on the price snapshot during the test:

Complete task cost = Successful request fee + Failed request fee + Retry fee + Manual rework cost

Prices and model availability change. Record the lookup time and source in each report, and never present a one-time snapshot as permanent pricing.

Sample size and interpretation boundaries

Protocol checks can be run in small numbers, but stability and latency will require at least dozens of requests to see an initial distribution. The smaller the sample, the less likely it is to use conclusions such as "inevitably faster" and "absolutely stable".

Model output is random. Even if temperature=0,Different service implementations, versions and parallelism,strategies may also produce differences. Competency conclusions should be combined with multiple questions, multiple rounds and real business baselines.

The following phenomena can only be marked as "requires verification":

• Return model Inconsistent with the request.
• usage Missing or Token arithmetic exception.
• A certain round of dynamic questions failed.
• Output style or knowledge cutoff readme changes.

Only when multiple pieces of evidence continue to appear and can be reproduced is it appropriate to upgrade the risk level. Black box results cannot replace model manufacturer certification.

Minimal structure when publishing reports

An auditable public report contains at least:

1. Test date, region, network and client version.
2. Model ID, protocol, parameters, and question set version.
3. Sample size, raw data fields, and whether failures are counted.
4. Success rate, P50/P95, capability indicators, Token and cost.
5. Known limitations, exception explanations, and retest plans.
6. A description that the key and business data have been desensitized.

Can be used firstAPI gateway detection toolGenerate protocols and behavioral sampling, and then followTest report interpretation guideView itemized evidence. Continue to use during the selection stage12-item API gateway test list, to avoid a single total score replacing business judgment.