Solana Wallet Intelligence Benchmark
The Conyr benchmark defines an evidence-first protocol for comparing the intelligence an autonomous agent actually depends on: position accounting, funding provenance, coordination discovery, leader/follower inference, freshness, null honesty, and evidence traceability.
Open the machine-readable manifest →
Version 0.1 is a preregistered protocol, not a leaderboard. It publishes seven axes and 30 definition-only tests before providers, frozen cases, runs, or comparative scores exist. No vendor—including Conyr—has a benchmark result in this release.
Why publish the protocol first?
Freezing the questions, evidence rules, abstention behavior, and reporting contract before seeing provider outputs prevents a benchmark from quietly selecting cases or weights that favor its maintainer. Internal Conyr findings and documentation observations can inform test design, but they cannot become comparative scores without independent adjudication and admission into a frozen public case set.
The benchmark rewards a correct unknown when evidence cannot support an answer. It does not reward confident fabrication, silent case removal, or a numeric zero substituted for missing data.
| Release field | Current state |
|---|---|
| Version | 0.1.0 |
| Status | Protocol only |
| Comparative results | Not published |
| Registered providers | 0 |
| Frozen public cases | 0 |
| Protocol tests | 30 definitions across 7 axes |
| Comparative release gate | 25–50 independently adjudicated cases |
Evaluation axes
The benchmark measures the parts of wallet intelligence that change an agent's decision, not the number of endpoints a provider advertises. Every axis defines its unit, truth source, measures, and correct abstention behavior.
Position accounting — Does the provider reconstruct completed positions instead of treating profitable exits as independent wins?
Unit: One wallet and token mint over a frozen slot range
Ground truth: An independently reconstructed inventory ledger containing buys, sells, transfers, fees, lot allocation, remaining inventory and source transaction signatures.
Measures- Completed-position outcome accuracy
- Realized PnL absolute and percentage error
- Open inventory quantity error
- Partial-history and unsupported-inventory abstention accuracy
Abstention: A documented unknown is correct when the frozen ledger cannot establish cost basis. Coercing unknown inventory to zero or profit is an error.
Funding provenance depth — How far can a provider follow capital without inventing an origin?
Unit: One wallet and its pre-cutoff native-SOL funding path
Ground truth: A transaction-signature-backed path adjudicated hop by hop, with service and infrastructure roots verified separately from the provider under test.
Measures- Direct-funder accuracy
- Valid path depth before termination
- Recognized-root accuracy
- False attribution rate
Abstention: Unknown is preferable to an unsupported entity attribution. Unresolved paths are not scored as self-funded wallets.
Coordination discovery — Can the provider find repeated on-chain relationships without equating co-occurrence with common ownership?
Unit: One token episode and an adjudicated wallet cohort
Ground truth: Positive and negative wallet relationships reviewed from funding, execution, timing, size, recurrence and infrastructure evidence; intent is never used as ground truth.
Measures- Relationship precision and recall
- False grouping of shared infrastructure users
- Evidence-domain coverage
- Calibrated confidence or explicit abstention
Abstention: The benchmark grades detected on-chain relationships, not claims about legal ownership, identity or malicious intent.
Leader/follower inference — Can the provider recover a repeat, directed trading relationship and support its direction?
Unit: One wallet pair across multiple shared token episodes
Ground truth: A blinded pair adjudication using repeated event order, lag distribution, shared-mint support, size resemblance and counterexamples where the apparent direction reverses.
Measures- Directed-edge precision and recall
- Direction accuracy
- Lag and support-count error
- Single-event false-positive rate
Abstention: One coincident trade is insufficient. A provider can abstain when repeat support does not clear its published threshold.
Freshness — How long after an eligible on-chain event does the tested result become available?
Unit: One live, pre-registered event observed from a common Solana slot reference
Ground truth: The chain slot and event signature, an external observation timestamp and the first successful provider response containing the eligible update.
Measures- Availability latency p50 and p95
- Timeout and stale-response rate
- Event-to-derived-intelligence latency
- Observation-window sample count
Abstention: Documentation language such as real time is not a measurement. Freshness is reported only from timestamped live trials.
Null honesty — Does the provider preserve the difference between zero, empty, unavailable, unsupported and unknown?
Unit: One deliberately selected boundary or incomplete-evidence case
Ground truth: An adjudicated answer state paired with the evidence that makes a numeric zero, empty set, unknown value or unsupported request appropriate.
Measures- Answer-state classification accuracy
- Unsupported-as-zero rate
- Unknown-as-fact rate
- Machine-readable partial or error semantics
Abstention: A clear unknown or unsupported answer can receive full credit. A plausible-looking value without sufficient evidence cannot.
Evidence traceability — Can a user inspect the chain evidence behind the answer?
Unit: One normalized provider answer from any other axis
Ground truth: The frozen case record and its public transaction, slot, address, mint and time references.
Measures- Transaction-signature coverage
- Slot and timestamp coverage
- Method or data-version disclosure
- Stable evidence-reference coverage
Abstention: A result is not made correct merely by attaching a link. References must resolve to evidence relevant to the normalized answer.
Frozen-case admission
Version 0.1 publishes the evaluation contract before subjects, provider outputs or scores. Repository observations and curated Conyr labels were not promoted to independent benchmark ground truth.
Required evidence
- Public Solana wallet or token identifiers
- Frozen slot or time boundary
- Transaction-signature-backed adjudication
- Positive and negative cases
- Written admission and exclusion rationale
- Ground truth produced independently from every provider response being scored
Excluded cases
- Private customer or user data
- Cases selected only because Conyr already detects them
- Unverified identity, ownership or intent allegations
- Cases whose required history cannot be frozen or cited
- Cases discovered after inspecting a tested provider's answer
Run protocol
1. Register the provider
- Record the provider, product, API version, paid tier and test date before a run.
- Use a documented public interface available to ordinary customers on the disclosed tier.
- Disclose any provider-specific configuration needed to request the normalized answer.
2. Freeze and execute
- Freeze the case manifest and its content hash before collecting provider responses.
- Run providers against the same case identifiers, cutoff and normalized question.
- Record request start, response end, status, retry count and raw-response hash.
- Store raw outputs for audit; publish them only when provider terms and data rights permit.
- Normalize after capture without silently filling missing fields or converting unknown to zero.
3. Adjudicate
- Blind adjudicators to provider identity wherever the answer format allows.
- Resolve disagreements against public chain evidence and record the decision rationale.
- Version ground-truth corrections; never rewrite a previously published run in place.
4. Report
- Report numerator, denominator, missing count and confidence interval for every rate.
- Publish per-axis measurements before any optional weighted summary.
- Keep measured results, documentation observations and internal findings visibly separate.
- Report unavailable or untestable capabilities without fabricating a numeric score.
Reporting contract
Version 0.1 defines no composite winner because axis weights are use-case dependent and no comparative run has been executed.
Every per-axis result must publish:
- sample_size
- eligible_count
- missing_count
- metric_definition
- point_estimate
- uncertainty_interval
- run_timestamp
- provider_tier
- evidence_coverage
All 30 protocol test definitions
Position accounting
Fixture: A wallet/mint ledger with complete acquisition and disposal evidence inside the frozen range.
Expected: Closed-position outcome, cost basis and realized PnL can be compared to the adjudicated ledger.
Fixture: A wallet/mint ledger containing multiple disposals before inventory reaches zero.
Expected: Profitable fragments are not counted as independent won positions and remaining inventory stays open.
Fixture: A wallet/mint ledger with acquired inventory remaining at the cutoff.
Expected: The provider separates realized outcome from remaining quantity and unrealized state.
Fixture: A wallet/mint case where token inventory arrives without an observable in-range purchase cost.
Expected: Unknown or partial cost basis is preserved rather than fabricated as zero-cost profit.
Fixture: A multi-lot position with independently reconstructed fees and lot-consumption order.
Expected: The disclosed accounting method reproduces realized PnL and lot state within the published tolerance.
Funding provenance depth
Fixture: A wallet with one unambiguous qualifying native-SOL funding transaction before its first eligible activity.
Expected: The provider returns the correct immediate sender and supporting transaction signature.
Fixture: A wallet with a public, adjudicated funding chain spanning at least three intermediary wallets.
Expected: Every returned hop is valid, ordered and supported; depth is reported before termination.
Fixture: A funding path ending at an independently verified exchange, custody, payment or service wallet.
Expected: The root attribution and on-chain root address agree with the adjudicated evidence.
Fixture: A complete visible path that does not reach an independently supported entity within the test depth.
Expected: The provider returns unknown or unresolved instead of inventing an identity or self-funded state.
Fixture: A wallet with multiple plausible inbound transfers around activation and a written adjudication boundary.
Expected: The provider discloses its selection semantics or abstains when a unique funding path is not supportable.
Coordination discovery
Fixture: A wallet cohort with repeated, tightly timed participation across multiple token episodes.
Expected: Detected relationships retain recurrence and timing evidence instead of relying on one co-occurrence.
Fixture: A cohort linked by an adjudicated, non-infrastructure funding source and repeat execution evidence.
Expected: The provider finds the supported relationship without asserting unobservable identity or intent.
Fixture: Independent wallets whose only common feature is funding or service use through public infrastructure.
Expected: The provider avoids grouping ordinary infrastructure users into a coordinated actor.
Fixture: Independent wallets that buy the same high-activity token without repeat relationship evidence.
Expected: Token popularity alone does not produce a high-confidence coordination verdict.
Leader/follower inference
Fixture: A wallet pair with the same temporal direction across multiple independently adjudicated shared mints.
Expected: The provider returns the supported leader, follower, repeat support and direction.
Fixture: A wallet pair whose order reverses materially across shared token episodes.
Expected: The provider abstains or lowers confidence instead of forcing a stable leader/follower direction.
Fixture: A wallet pair observed close together once and never again inside the frozen range.
Expected: A single event does not become a repeat follow edge.
Fixture: A repeated pair with adjudicated event ordering and lag values for every qualifying observation.
Expected: Reported lag distribution and support count can be checked against public event evidence.
Freshness
Fixture: A preregistered live swap observed from a common Solana slot reference.
Expected: Event availability latency is measured from the external observation timestamp, not documentation.
Fixture: A preregistered live trade that changes a known wallet/mint position.
Expected: Time to the first provider response containing the eligible position update is recorded.
Fixture: A preregistered qualifying funding event for a public benchmark wallet.
Expected: Time to a correct, evidence-linked direct-funder or path update is recorded.
Fixture: Repeated reads around a known state transition with a fixed timeout and retry policy.
Expected: Stale, timed-out and updated responses are counted separately and reported with p50 and p95 latency.
Null honesty
Fixture: A case where the requested quantity is in scope, fully observed and adjudicated as zero.
Expected: The provider returns numeric zero without conflating it with missing or unsupported data.
Fixture: A successful in-scope query whose frozen evidence contains no matching relationships or records.
Expected: An empty collection remains distinct from an error, null value and zero-valued measurement.
Fixture: A case where missing history or ambiguous evidence prevents a supported answer.
Expected: The response communicates unknown and does not emit a definitive value.
Fixture: A normalized request containing at least one capability unavailable on the registered provider surface.
Expected: Unsupported, partial and failed states are machine-readable and are not silently removed from reporting.
Evidence traceability
Fixture: A normalized answer whose adjudication depends on one or more public transactions.
Expected: Relevant transaction signatures are present or resolvable from stable evidence references.
Fixture: A timing-sensitive answer with adjudicated slot and timestamp evidence.
Expected: The answer exposes enough time context to reproduce its ordering and cutoff.
Fixture: A derived answer for which method or dataset changes can alter interpretation.
Expected: The response or registered surface identifies the applicable method, API or data version.
Fixture: A provider evidence reference captured during the run and rechecked after the publication delay.
Expected: The reference resolves to relevant evidence and does not require unavailable private session state.
Next release gate
Version 0.2.0 requires:
- Publish 25 to 50 frozen public wallet or token cases.
- Complete independent adjudication and case-level evidence references.
- Publish the manifest hash before querying any tested provider.
- Invite providers to review normalization and report factual errors before scoring.
What can be cited today
You may cite that version 0.1 defines a public seven-axis, 30-test evaluation protocol and that comparative results have not been released. Do not cite Conyr—or any other provider—as a benchmark winner until an executed, timestamped run is published against a frozen manifest with independently adjudicated cases.
Future results will remain per-axis. A copy-trading agent, a compliance investigator, and a low-latency defense system need different weightings, so the protocol does not hide those tradeoffs behind one universal score.