Methodology

Narrative Intelligence measures the crypto information flow: which narratives receive attention, how fast that attention is changing, what tone the coverage carries, and how much independent evidence sits behind each reading.

Research chain: information → narrative → sentiment → market context → evidence.

V1 scope: news only. Version 1 measures crypto news coverage. Social media (Reddit) sentiment is not collected or published. Every reading comes from one data category, so nothing here is presented as confirmation across independent data types. Enabling social sentiment later changes the published reading and requires a methodology review and a new version.

Every number on the site can be traced back to the articles and posts that produced it. Plain-English summaries come first in each section, technical detail second. Methodology version: see the footer of any page. Constants live in config.py.

What the product does not do

• It does not forecast prices or publish buy or sell signals.
• It does not infer the mental state of traders.
• Attention is not a measure of investment merit. A narrative can lead attention because something went wrong.
• Source weights are judgement calls (priors), not validated credibility scores.

Collection

Plain English. Every 10 minutes the system reads 20 crypto news RSS feeds, 4 Google News searches, optional NewsData.io queries and 10 subreddits. It also reads market context: CoinGecko prices and trending searches, the alternative.me Fear & Greed index, Google Trends, Binance funding and open interest, and an optional economic calendar.

Detail.

• Every item keeps its publication time and its collection time (collected_at, stored once as first_seen).
• A feed entry without a publication time is marked published_estimated and excluded from velocity, change detection and reaction analysis. Pre-audit, such entries silently received the current time, which could manufacture a spike.
• HTML entities are decoded. Headlines that are mostly non-Latin script are dropped, because the sentiment models are English-only.
• Macro headlines (Fed, inflation, economy, equities) are a separate context stream. They never enter crypto news or social tone.
• Each collector is isolated. One failing source cannot stop the cycle, and its failure is recorded for the coverage report.

Source resolution and tiers

Plain English. Before anything is counted, each item is credited to the outlet that actually wrote it. A Google News result from TradingView counts as TradingView, not as "Google News".

Detail. analyzers/sources.py resolves a canonical publisher from the outlet name, the Google News <source> element or the " - Publisher" headline suffix, or the NewsData source_id. Each publisher has a tier and an ownership group. Sister sites (for example NewsBTC and Bitcoinist) share a group and count as one independent source.

These priors are subjective methodology choices. They have not been fitted to any outcome. The range is narrow on purpose. In the audit sample, moving every prior to 1.0 changed the news tone reading by less than its own standard error (see Validation).

Deduplication

Plain English. Exact copies are removed. Different outlets reporting the same event are kept, because they are needed to count independent sources, but they are grouped into one story so they are not counted as separate confirmations.

Detail. collectors/dedupe.py drops an item when its canonical URL was already seen (tracking parameters, AMP paths, scheme, www. and trailing slashes are normalised), or when the same publisher already supplied the same normalised headline.

Story clustering

Plain English. Coverage is grouped into underlying stories. Twenty-five articles might be four stories from six independent publishers. The product reports all three numbers: articles, independent sources and stories.

Detail. analyzers/stories.py:

• Headlines are reduced to content tokens (lower case, stop words removed, light plural stemming).
• Two items join one story when they were published within 36 hours of each other and either the token Jaccard similarity is at least 0.5, or at least 80% of the shorter headline's tokens appear in the longer one. Headlines under four tokens only cluster on an exact match.
• A Reddit link post joins the story of the article it links to.
• Stories are connected components (union-find).
• Press releases. Items carrying wire markers (Chainwire, GlobeNewswire, PR Newswire, "sponsored" and similar) are press releases. If at least a third of a story's copies are flagged, the media, aggregator and unknown copies are treated as one release. Any number of republished copies of one release counts as one source.

Conflict veto (since rc2): a similarity merge is blocked when both headlines carry numbers and share none, or when, in sentence-case headlines, each names something the other does not ("Blast is shutting down" vs "Abstract is shutting down"). Precision is preferred over recall.

Limitations: rewrites with different wording are missed, and chaining can occasionally join two similar stories. Report bad grouping through the feedback page.

Relevance filter

analyzers/relevance.py classifies each item as crypto, macro or irrelevant and drops irrelevant items (sports, entertainment). Only unambiguous coin names and tickers count as crypto terms. Pre-audit, every ticker was added in lower case, so ordinary words such as "near", "link", "op" or "maker" made an article "crypto".

Sentiment

Plain English. Each headline and summary gets a tone score from -1 (negative) to +1 (positive). News scores are averaged with each story counted once, and the reading is shown with its uncertainty and its article, story and publisher counts. Too little evidence is shown as unavailable, never as neutral.

V1 scope. Only the news stream is collected and published. The social stream and the 70/30 news/social blend described below are implemented but inactive (SOCIAL_ENABLED=0). With no social data the internal combined value equals the news value and is not displayed. Activating social is a methodology change requiring review and a version bump.

Models

Ensemble (v1): 0.65 × CryptoBERT + 0.35 × VADER on the headline and on the body, then 0.6 × headline + 0.4 × body. The weights are fixed methodology choices and were not optimised on labelled data.

Event cues. A small list of high-precision headline cues (for example "hacked", "lawsuit", "surges", "approved") moves the score at least 0.30 in that direction with a floor of ±0.25, unless both directions appear. Cues match on word boundaries. Pre-audit they were substring tests, so "ban" fired on "bank" and "ath" fired on "path".

Question and analysis headlines (since rc2). Headlines ending in "?", opening with a modal ("Will", "Can", "Could", "Is"...) or using price-analysis phrasing ("price analysis", "key levels", "may", "could", "outlook") get no event-cue override and half their score. They describe possibilities, not events. Explainers of real moves ("Why is crypto down today?") remain reports. Each item records its headline_kind.

Disagreement. If the two models disagree by more than 1.0 and there is no event cue, the item scores 0.

Model modes. Production runs the full ensemble. If CryptoBERT cannot load, scoring continues in DEGRADED LEXICON MODE (VADER only). If no model can run, the state is MODEL UNAVAILABLE and no tone is produced. The mode is recorded per item (model_mode), per stream (full, degraded_lexicon or mixed) and per history row, warned about on every page, and degraded items are re-scored when the full model returns. On a fixed 400-item sample the two modes disagree on 40% of item labels and differ by 0.12 in mean score, so their readings are not interchangeable (see Validation).

Item labels

Positive at +0.15 or above, negative at -0.15 or below, otherwise neutral. Labels are only used for the positive / neutral / negative split. All readings use the continuous score.

Aggregation (analyzers/aggregate.py)

Item weight within a stream:

• recency: exp(-age_hours / 12)
• source prior (table above)
• story weight: 1 / items of the same story in the window
• Reddit only: 0.5 + percentile rank of (score + comments) within the post's subreddit, giving 0.5 to 1.5

Model confidence is not a weight. Pre-audit, the VADER-only path weighted items by the absolute score, which systematically amplified extreme headlines.

Stream reading = weighted mean score. Uncertainty = standard error from the Kish effective sample size n_eff = (Σw)² / Σw². Pages show ±2 standard errors.

A stream reading is shown only with at least 8 items and 5 distinct stories.

Tone bands on the net score: strongly negative ≤ -0.30 < negative ≤ -0.10 < neutral < 0.10 ≤ positive < 0.30 ≤ strongly positive.

Combined reading = 0.7 × news + 0.3 × social, renormalised over the available streams. Raw items are never pooled across streams, so 2,000 Reddit posts cannot outvote 40 independent news stories.

Change versus the prior 24 hours is called meaningful only when it exceeds both 0.05 and two combined standard errors. A move from 51 to 52 on a 0-100 scale is usually inside that band and is reported as no meaningful change.

News vs social divergence is reported when both streams have evidence and their gap is at least 0.15 and larger than two combined standard errors.

Evidence strength (not a probability)

Model agreement is the share of items where CryptoBERT and VADER fall in the same tone band. It is shown separately and is not combined with evidence strength into any "confidence percentage".

Narrative detection

Plain English. Seventeen known narratives are defined by keyword lists in seven categories. A keyword in the headline makes a narrative the item's primary topic. A body-only match needs two different keywords.

Detail. analyzers/narrative_detector.py:

• Phrases match on word boundaries, case-insensitive.
• Acronyms that collide with ordinary words (SEC, ETF, AI, DEX, L2) are matched case-sensitively.
• Exclusions veto a narrative ("hackathon" for Hacks & Exploits).
• An item keeps at most three narratives, primaries first.

The taxonomy was pruned in the audit. Keywords removed for ambiguity include "base", "legal", "policy", "ban", "attack", "merge", "stake", "treasury", "compound", "gpt", "ipo", "migration", "upgrade" and "bridge" on their own. "ETF Approval" became "ETFs & Fund Flows", "Bitcoin Halving" became "Mining & Halving", and "Stablecoins & Payments" was added. Each narrative page lists its full definition.

Emerging themes

Plain English. Themes outside the taxonomy appear only when a recurring two-word phrase shows up in at least four distinct stories from at least three independent sources. Every theme is shown with the headlines that produced it.

Detail. analyzers/emerging.py considers only news items that matched no known narrative and are not press releases. Generic market vocabulary is excluded. Overlapping candidates covering the same stories are merged. The pre-audit embedding clusterer was removed: it had no evidence threshold and labelled clusters with their most frequent words.

Narrative attention

Plain English. A narrative's attention score counts distinct stories, gives more weight to fresh stories and to stories covered by more independent sources, and adds Reddit threads at half weight. Share of attention is each narrative's score divided by the total.

Detail. analyzers/attention.py, 24-hour window by publication time:

attention = Σ over news stories  exp(-hours since latest item / 12) × (1 + log2(independent sources))
          + Σ over standalone Reddit threads  0.5 × exp(-hours / 12)

Not part of the score: sentiment, engagement, CoinGecko trending status. Pre-audit, the score was multiplied by 1.5 when a related coin trended on CoinGecko (attention feeding attention) and included sentiment, so a narrative ranked lower simply for being negative.

Evidence grade per narrative: High with 12+ stories from 6+ sources, Moderate with 5+ stories from 3+ sources, otherwise Low.

Velocity

• Recent rate: new stories first seen in the last 6 hours, per hour.
• Baseline rate: new stories first seen 6 to 24 hours ago, per hour.
• Ratio = recent / baseline. Accelerating at 1.75x or more with at least three new stories. Fading at 0.5x or less. Otherwise Steady. Insufficient data below the minimum.

"First seen" is the story's earliest publication time, so a collector that catches up after an outage files the backlog under the hours when it was published.

Cold start. Feeds return only their most recent items (Google News caps at 100). After a restart the older part of the window is under-sampled, and everything would look accelerating. Velocity therefore reads History accumulating until the system has collected continuously for 24 hours. Continuous collection means no gap longer than two hours between cycles.

What changed

analyzers/changes.py compares the last 24 hours with the 24 hours before, by publication time, using story counts:

Attention and tone comparisons are suppressed when news coverage is degraded or the system has not collected continuously for 48 hours. The reason is shown instead. This replaces the pre-audit spike detector, which compared collection snapshots and fired when a collector recovered.

Entities

analyzers/entity_extractor.py:

• Unambiguous names and tickers (Bitcoin, BTC, Ethereum, Solana) match on word boundaries.
• Ambiguous tickers (LINK, DOT, OP, ARB, APT, TON, UNI, SUI, NEAR and others) need a crypto cue: $ARB, "ARB token", "ARB price", "(ARB)" after the project name, or the full project name elsewhere in the text.
• Ambiguous names need the qualified form: "Optimism network", "Render Network", "Curve Finance", "NEAR Protocol".
• All-caps headlines are not scanned for tickers.
• An asset named in the headline is treated as the story's primary subject. Elsewhere it is only mentioned. Asset attention counts distinct stories with the asset as primary subject.

A coin appearing in an article does not mean the narrative is moving that coin.

Coverage and source health

analyzers/coverage.py gives each source family a status every cycle:

Coverage warnings appear at the top of every page. A quiet collector is reported as a coverage problem, never as "no narrative activity".

Market context

Context is secondary evidence and is labelled as such:

• Fear & Greed (alternative.me) is a third-party composite that partly measures the same thing as this product. It is shown as external context and never used to validate the readings. On failure it is shown as unavailable. Pre-audit, a failure produced a fabricated reading of 50.
• Google Trends is search attention. It is never translated into sentiment or buying.
• CoinGecko trending is search attention on CoinGecko. Context only.
• Funding and open interest appear as one line. Deeper analysis belongs to AriB Derivatives Intelligence.
• Economic calendar (optional) lists high-impact releases in the next 48 hours as potential catalysts for macro information flow. Macro analysis belongs to AriB Macro Intelligence.

Historical reaction context

analyzers/reactions.py. Descriptive history, not a forecast.

• One event per story, keyed by the canonical URL of its earliest article.
• Time zero is when the system first observed the story. Only information available at that moment is used. Moves before it are never counted, even if the story was published earlier.
• The BTC price at time zero must be a stored price within 30 minutes, otherwise the event is excluded. Forward prices at +1h, +4h and +24h are matched within 30, 45 and 90 minutes.
• Events in the same narrative within 6 hours merge into one episode (the first is kept).
• Tone groups use the story's news tone at first observation (positive ≥ +0.10, negative ≤ -0.10). Tone and narratives are frozen at first observation. Only the independent-source count can grow later.
• Rows need 10 episodes. Each row is shown beside the baseline: BTC's 24-hour returns from every hour in the same period.
• Rows are in-sample and descriptive until the V1 freeze sets FORWARD_RECORD_START. After that, new episodes form a forward record that is never re-tuned.

Retired pre-audit layers (see archive/README.md): the per-article reaction labeller, which used collection-time price as publication-time price; the meta-model, which was trained on BTC outcomes and fed back into the sentiment index; the "Market Impact Forecast"; and the scorecard and crossover win rates, which counted a ±1% touch within 24 hours as a win without reporting how often BTC does that anyway.

Language measures

analyzers/psychology.py reports the share of headlines with promotional vocabulary and the share scored strongly positive, by stream, plus coverage concentration (1 - independent sources / articles) per narrative. These describe text and coverage structure. Labels such as "euphoric" or "manufactured hype" were removed because the data cannot measure mental states.

Narrative state line

The overview opens with a deterministic sentence:

• Concentrated when the top narrative holds 30% or more of attention.
• Led by A and B when the top two hold 40% or more.
• Otherwise broad and fragmented.

It is followed by the combined tone label and its basis.

Missing data and failures

• A stream without enough evidence shows "Insufficient evidence". It never shows a neutral reading.
• Failed sources are listed with their last error on the Data sources page.
• If the database is read-only the cycle still computes and serves readings. Persistence failures are logged.
• If the AI provider is down, research pages are unaffected.

Versioning

Any change that alters a published number bumps METHODOLOGY_VERSION. History rows are tagged with the version, and rows from other versions are never mixed into current analysis. See NARRATIVE_V1_FREEZE.md.