Dedolytics Research · Technical Whitepaper

Robust, Real-Time Signal Scoring

The statistical methodology behind our proprietary signal-intelligence engine, an in-house SaaS platform that scores noisy, high-volume streaming data as it arrives.

Version 2.0 Dedolytics Research Proprietary methodology

Abstract

We describe the scoring engine at the core of a proprietary Dedolytics SaaS platform. The engine ingests a continuous stream of behavioral and event signals, extracts features, and produces stable, interpretable scores in real time. The contribution is not any single metric but the pipeline that makes scores trustworthy on messy, heavy-tailed, bursty data: a robust normalization stage, a shrinkage-regularized dependency structure, principal-eigenvector signal weighting, and an exponential time-decay model for momentum and forecasting. The same toolkit underpins our client work in demand forecasting, margin analytics, and risk.

1.Overview

Most scoring systems fail in the same place: real-world signal streams are noisy, heavy-tailed, and bursty. A handful of extreme events dominate naive averages, correlations estimated on short windows are unstable, and yesterday's spike is treated the same as a sustained trend. The result is scores that swing wildly and cannot be trusted for a decision.

Our engine is built to be robust to exactly these failure modes. It runs as a proprietary SaaS pipeline: signals stream in, features are extracted per entity, and a multi-stage statistical process turns them into scores that are stable over time and interpretable by a human. Every stage is chosen to resist outliers and small-sample noise rather than assume clean, well-behaved input.

2.Pipeline and feature extraction

The engine consumes a continuous event stream. Each event is parsed into a timestamped, typed record and attributed to one or more entities. From the raw stream we derive per-entity features across several families: volume, timing, network position, and category mix. Feature extraction is incremental, so the state needed to score any entity is maintained in memory and updated as events arrive, which is what makes real-time scoring possible.

The raw features are never scored directly. They pass through the normalization, dependency, weighting, and temporal stages described below.

3.Robust normalization

Signal features are heavy-tailed and span several orders of magnitude, so we normalize in two steps designed to keep extreme values from dominating.

3.1 Log compression

We first compress magnitude with a log transform that is well defined at zero:

x' = ln(1 + x) # log1p, stable at x = 0

3.2 Robust standardization (median / MAD)

Rather than a mean and standard deviation, which outliers distort, we standardize with the median and the median absolute deviation (MAD), scaled so the result is comparable to a standard z-score under normality:

MAD = median( | x'ₕ − x̄' | ) zₕ = ( x'ₕ − x̄' ) / ( 1.4826 × MAD ) # 1.4826 = consistency constant

3.3 Winsorization

Finally we clip the remaining extremes to a robust range so a single anomalous event cannot move a score on its own:

zₕ ← clip( zₕ, P₂, P₉₈ ) # winsorize to the 2nd and 98th percentiles

4.Stable dependency structure

Features are correlated, and those correlations carry signal. But a sample correlation matrix estimated on a short window is noisy and often not well conditioned, which makes any downstream weighting unstable. We regularize it toward the identity with a shrinkage estimator:

K = ( 1 − α ) R + α I, α = 0.1 # R = sample correlation, I = identity

Why shrinkage

Shrinkage trades a little bias for a large reduction in variance. It guarantees a well-conditioned, invertible matrix even on small samples, so the eigen-decomposition in the next stage is stable from one window to the next instead of jumping around with the noise.

5.Signal weighting via the principal eigenvector

Given the regularized matrix, we let the data decide how much each feature should count. The principal eigenvector of K, the direction of greatest shared variance, gives a natural, non-arbitrary set of feature weights:

K v = λ_max v w = v / Σ vᵢ # normalized principal-eigenvector weights # representative solution λ_max ≈ 2.48 v ≈ ( 0.59, 0.54, 0.60 )

Because the weights come from the dominant eigenvector of a shrinkage-stabilized matrix, they are both interpretable and stable: features that genuinely co-move carry more weight, and the weighting does not lurch when the input is noisy.

6.Temporal dynamics and forecasting

A score should reflect a sustained trend, not the last data point. We model each entity's score as a momentum process with exponential time decay, so recent signal is weighted more heavily while older signal fades smoothly:

q( t+1 ) = ρ · q( t ) + ( 1 − ρ ) · s( t ) ρ = exp( −γ · λ_max ) # decay tied to the dominant eigenvalue

The decay rate is tuned to a realistic half-life of roughly 14 days, so a signal's influence halves about every two weeks. Coupling the decay to the principal eigenvalue keeps the temporal behavior consistent with the dependency structure the weights came from. The same recurrence is what the engine projects forward to produce short-horizon forecasts.

7.Real-time architecture

The engine is event-driven end to end. A streaming pipeline ingests events, updates per-entity feature state incrementally, and re-scores on change, so dashboards reflect the current state rather than a nightly batch. State is held in memory and checkpointed, which keeps scoring latency low even at high event volume.

8.Privacy by design

The engine scores signals, not content. It operates on structural and temporal features (who, when, how often, in what pattern) and never needs to store the content of the underlying records. This is an engineering constraint built into the pipeline from the first stage, not a policy applied after the fact.

9.Why it transfers

Nothing above is specific to one domain. Robust normalization, a stabilized dependency structure, eigenvector weighting, and time-decay momentum are the same tools we bring to client work: forecasting demand on messy retail data, separating real margin signal from noise, and tracking risk in real time. The proprietary engine is where we prove the toolkit before it touches client data. The domain changes. The math does not.

This paper documents the methodology behind a proprietary Dedolytics research and engineering platform. Representative constants are shown for illustration. To discuss how the same methods apply to your data, get in touch.