Reddit: Seed 2.0 Lite in the “Rusty Needle in a Haystack” Retrieval Test

One-sentence takeaway

In a retrieval test that evaluates both “finding the right target amid near-miss noise” and “returning NULL when no valid target exists,” Seed 2.0 Lite achieved 66% overall accuracy and outperformed Seed 2.0 Pro in the author's test. This suggests it may be suitable for low-cost resolver/tool-target tasks, but the result should not be generalized to a universal intelligence ranking.

Use cases

• Suitable tasks: Approximate matching of files, labels, tool targets, records, or retrieval snippets, as well as Agent resolvers that need to reject incorrect candidates.

• Unsuitable tasks: Treating the results of a single synthetic retrieval benchmark as the success rate for long-horizon coding, vision, or production finance tasks.

• Applicable model version: The author identifies it as Doubao Seed 2.0 Lite; the specific API provider, model snapshot, and parameters were not disclosed.

• Applicable client, Agent, or API: The test platform was not disclosed; the original post focuses on model comparison, not latency testing for a particular gateway.

• Recommended reasoning tier and parameters: Not disclosed; reproduction must fix the model, temperature, max output, and whether thinking is enabled.

Test environment

• Task structure: Each trial provides one query and a haystack of 1,000 labels, with only one response allowed.

• Sample ratio: Each benchmark run contains 750 positive cases and 250 negative cases.

• Repetition: 100 rounds per model, with the same set of 1,000 cases shuffled each round.

• Target: For positive cases, identify the correct label_id; for negative cases, return NULL among similar but invalid candidates.

• Input size: Each label is approximately 4–35 Gemini tokenizer tokens; all 1,000 labels total approximately 23,000–25,000 tokens.

Input/configuration

The reproducible task rules provided in the original post:

Given a query and a list of 1,000 labels with label_id values.
If the query corresponds to a real target that has been slightly corrupted, abbreviated, misspelled, or degraded, return only the correct label_id.
If the list contains no valid target, return only NULL even if near-matches exist.
Each case may be answered only once; do not add explanations.

Positive example: query 0710B Lewis, with the list containing [label_id=123] 0710B LewisC <random note>, expected 123.

Negative example: query 0720A LewisO, with the list containing only similar but non-matching entries, expected NULL.

Results data

• Seed 2.0 Lite overall accuracy: 66%.

• The author says Lite outperformed Seed 2.0 Pro in this test, but the post does not disclose Pro's specific score.

• Gemini 3 Flash overall accuracy: 72%, and it also outperformed Gemini 3.1 Pro Preview in this test.

• The author notes that Qwen 3.5 Flash's 33% was affected by “returning NULL for most answers,” showing that overall accuracy must be split into positive and negative accuracy. The original post defines Positive accuracy and Negative accuracy, but does not publish a per-model breakdown table.

Conclusion

Lite shows a useful cost/reliability signal for Agent resolver tasks involving “approximate matching + refusal,” and it unexpectedly exceeded Pro. However, the result is affected by the synthetic data distribution, single-turn output, and prompt design; real-world file and tool selection still needs to be retested with real candidate sets.

Limitations

• The test is a synthetic benchmark from a single author; the complete set of 1,000 cases, random seed, model parameters, and provider were not disclosed.

• The statistical method and confidence intervals for the 100 rounds were not disclosed; only overall accuracy and some observations are available.

• The negative-case ratio, tokenizer, and case structure can strongly affect the result, which cannot be directly generalized to arbitrary long contexts or multi-turn Agents.

Reproduction steps

1. Generate a fixed set of 750 positive cases and 250 negative cases, and save the original 1,000 labels, queries, expected answers, and random seed.

2. For each model, fix the prompt, provider, model snapshot, temperature, max output, and single-response constraint.

3. Repeat for 100 rounds, shuffling the cases each round; record overall, positive, and negative accuracy, along with refusal-format errors.

4. Also run the same protocol on real filenames, tool names, and record IDs, then report the difference between synthetic and real data.

Source excerpts or observations (for compliant short quotations only)

• The author defines the correct answer for negative cases as NULL, specifically measuring whether the model avoids forcing the most similar candidate into the answer.

• The author explicitly says the test is for selecting a resolver agent for an agentic orchestrator, not for proving which model is the smartest.