Validation and known limitations

This page separates what has been tested from what has not. Nothing here is a formal accuracy figure.

Status in one table

Sentiment

The "~90% reliable" claim

The previous README described the CryptoBERT + VADER ensemble as "~90% reliable". The audit found no labelled dataset, no evaluation script and no stated metric (accuracy, F1, agreement), in-sample or out-of-sample. The production Docker image installs CryptoBERT, but the lean deploy (render.yaml, requirements.txt only) never loads it and runs VADER alone. The claim has been removed and not replaced with another number.

Model domain

CryptoBERT was fine-tuned on crypto social media posts. Most of this product's evidence is news headlines, a different register: formal, compressed, often neutral, with regulatory and legal language. No source-specific evaluation exists. FinBERT (financial news) is scored in shadow for every item so a comparison can be made once a labelled set exists. Switching models without such a comparison would replace one untested choice with another.

Manual review, 7 October 2026

61 live items (57 news, 4 macro) were stratified across 11 narratives, unclassified items and model labels.

Reviewer: the AI coding assistant that carried out this audit, not a human analyst. These labels find failure modes. They are not ground truth and must be re-labelled by a person before any accuracy statement. A human-labelled set of at least 300 items, stratified by source type, is a precondition of the V1 freeze.

Tone failure patterns found:

• Price-analysis headlines read as directional ("…Price Analysis for October 5", "406 Billion SHIB Have Landed: Analyzing the Effect"). Should be neutral.
• Event outcomes missed: "L2 Abstract Is Shutting Down" scored +0.20. Fixed: "shut down" and "shutting down" added to the event cues.
• Exchange outflows read as bearish: "Binance BTC outflows hit highest since 2023" was forced negative by the ETF-outflow cue. Fixed: exchange-outflow phrasing is excluded from the cue.
• Slang and sarcasm: "Bitcoin is headed to $100,000… I'm long as hell" scored -0.67 under VADER ("hell"). Not fixed. A known lexicon limitation.
• Questions and hedged headlines ("Are 16 tokens now commodities?") scored as positive.
• Mixed headlines ("AI boom to crash and hand Bitcoin the bailout") cannot be represented by one score.

Narrative failure patterns found:

• Bare "AI" pulled chip and equity stories (AMD supply chain, S&P earnings) into AI & Crypto. Fixed: bare "AI" removed. AI & Crypto fell from 31 to 7 stories in the same sample, which shows how large the false-positive rate was.
• Company names ("DeFi Development Corp") matched DeFi. Fixed with exclusions.
• Quote and forum pages (Yahoo "… Forum & Discussion") passed relevance. Fixed.
• Price round-ups naming DOGE or SHIB are tagged Memecoins. Open.
• Misses: an ETH treasury company story and a fraud sentencing were unclassified. Open.

Full sample with model outputs: docs/validation/manual_sample_2026-10-07.json.

Full ensemble vs degraded lexicon mode (rc2)

Fixed 400-item sample (scripts/compare_model_modes.py, output in docs/validation/model_mode_comparison.json). This shows how much readings change when the production model falls back. It is not an accuracy test.

The full ensemble reads coverage markedly more positively than VADER alone. A fallback would shift the headline reading by about 0.12 on average. That is why degraded mode is labelled on every page and in history, and why its readings must never be compared with full-model readings. Which of the two is closer to human judgement is unknown until the human sample is labelled.

Largest disagreements: headline questions and explainers ("Why did crypto just drop?") and macro headlines that the lexicon reads as strongly negative and the ensemble near neutral. Also clear negative events ("Wallets drained of $4 million") that the full model reads less negatively (-0.26) than the lexicon (-1.00), because CryptoBERT is positive-leaning on news text.

Question and analysis headlines (rc2)

Regression fixtures in tests/test_preprod_regression.py: "Will Bitcoin fall below $100K?", "Can Ethereum break resistance?", "Why Bitcoin may struggle this week", "Bitcoin price analysis: key levels to watch" and others. For these headlines the event-cue override is now disabled and the score halved, so they are no longer treated like factual events ("Bitcoin falls below $100K"). Explainers of real moves ("Why is crypto down today?") remain reports, as does "How hackers drained $40M". In the 320-item export, 25 headlines are classed as questions and 24 as analysis.

Limitation: this is a pattern rule, not language understanding. Its effect on accuracy will be measured on the human sample, split by headline kind.

Reworded duplicates (rc2 investigation)

Labelled fixture: tests/fixtures/reworded_pairs.py (9 must-merge, 3 optional, 13 must-not-merge pairs).

The investigation found the bigger problem was precision. rc1 merged different events: "Blast is shutting down" with "Abstract is shutting down", Winklevoss with Grayscale Zcash ETF filings, "buys 1,000 BTC" with "buys 2,500 BTC", and October 5 with October 6 price analyses. The adopted veto blocks merges when numbers conflict or, in sentence-case headlines, when each side names something the other does not. A rarity-based rewrite rule added no recall (a big story's distinctive words are not rare in the corpus), so it was not adopted. The recall limitation stays: about half of genuine rewrites are not joined, which understates independent sources for heavily rewritten stories. On the live sample the veto split 585 stories into 588.

Source prior sensitivity

Same 24-hour live window (503 news items, 428 stories):

Priors change the reading by well under one standard error, so their subjectivity has little practical effect. Story weighting matters more. Without it, syndicated stories dominate the reading.

Story clustering

Live sample: 672 news articles resolved to 585 stories from 125 publisher groups. Spot checks confirmed correct groupings (the OKX funding round: 5 outlets, 1 story) and press-release collapse (a Chainwire release on 4 sites counts as 1 source). Known misses: headlines reworded with different vocabulary ("Pudgy Penguins-backed L2 Abstract to shut down after losing tens of millions" vs "Abstract to shut down Ethereum layer 2") stay separate. This overstates story counts and understates independent sources for heavily rewritten events. Kept as an expected failure in the tests.

Velocity and outage recovery

Tested (tests/test_attention_changes.py):

• A 12-hour backlog collected at once does not produce acceleration, because velocity uses publication times.
• Items without real timestamps are excluded.
• On the first live run, every narrative appeared to grow 6 to 15 times against the prior 24 hours. This was an artifact: feeds only return recent items, so the earlier window was under-sampled. Velocity and changes now wait for 24 and 48 hours of continuous collection.
• Change detection is suppressed when news coverage is degraded.

Market reaction

The reaction layer was rebuilt (see Methodology). Under the corrected method there is no usable history yet: every event must be re-observed with a price at the moment of observation. Pre-audit records are excluded for these reasons:

• The baseline price was the price at collection time, applied to the publication time.
• Syndicated copies were separate observations.
• A model trained on those outcomes was fed back into the sentiment index, so later "accuracy" partly measured in-sample fit.
• "Win" counted a ±1% touch within 24 hours. BTC touches ±1% in most 24-hour windows regardless of direction, and that base rate was never shown.

Until the V1 freeze sets FORWARD_RECORD_START, every reaction row is labelled in-sample and descriptive.

Failure-mode tests

Automated (tests/test_public_safety.py):

Observed live during the audit: Reddit's keyless API returned HTTP 403 for every subreddit from the audit machine. The product showed "Social: Unavailable" and a news-only combined reading, as designed. Bitcoin Magazine's feed timed out in one cycle and was listed under failed feeds.

What would make this validated

1. A human-labelled set of at least 300 items, stratified by source type (news, Reddit) and narrative, labelled by two people with agreement reported.
2. Accuracy and macro-F1 for CryptoBERT, VADER, FinBERT and the ensemble on that set, by source type.
3. A choice of model or weights made on a development split and reported on a held-out split.
4. The V1 freeze, then a forward record that is never re-tuned.