Skip to content

Continuous, Active Federated Learning for Data Streams

Research & Experiment Plan

Working document for project lead, partners, and potential students. This page is the always-read core — the framing every non-trivial task needs. Detail lives in the sibling files linked below; open one only when the task calls for it (see the routing table).

Tags: [STD] standard / re-implement · [EXT] extend an existing method to our setting · [NEW] genuine contribution absent from prior art.

Scope

In scope: learning loop: AL as a label-free streaming filter ↔ trainable SSL backbone, trained using FL over these transient data streams → learning health monitoring / continual learning health metrics → filter correction. For the AL filter, analyze the performance-vs-storage Pareto front.

Out of scope: solving catastrophic forgetting (CF) end-to-end (instead: demonstrate + opportunistically mitigate), or FL efficiency or privacy attacks (rely on privacy-by-design).

Experimental framing: the independent variable (the knob we turn) is the filter selection, together with a choice of open-loop or closed-loop setting (i.e. whether the filter is static, or steerable). The health metrics are the dependent variable / readout (the thermometer), and the finding about the coupled system is the contribution, not any single filter, unless we can show a filter design (e.g. steerable, vs. static) that outperforms others on the health and downstream (e.g. classifier-level) metrics.

In open-loop (Phases 0–2) the base filter is the independent variable: fixed per run, stock knobs, read the thermometer. In closed-loop the independent variable moves up a level to the control policy (open vs. closed, and the controller design), holding the base filter fixed. Closing the loop requires a filter that exposes a steerable operating point — see N-G.

The loop — two feedback edges

The "loop" is two feedback edges, and naming them keeps the experiments clean:

  • Fast (inner): model → filter. The filter scores informativeness against the current backbone, so as the model adapts, what counts as novel/surprising shifts. This edge is intrinsic to any streaming filter and is what creates the coupled dynamics (and the pathologies).

  • Slow (outer): monitor → filter. The health monitor detects which degradation mode is setting in and re-aims selection accordingly. Example: rank dropping (collapse) → push toward coverage; probe-on-past falling (forgetting) → retain/replay under-covered past; overspecialization → down-weight the over-selected mode. This edge is what corrects the pathologies, and it is what makes this a loop rather than a pipeline.

Proposed novelty at a glance

The space is crowded along each axis; the assembly is not. The contribution is the closed loop as a live system, not any single axis. Full statements + the closest prior work each claim stands against are in novelty.md.

  • N-A — The coupling, run as a live loop [NEW]. The two edges operate together, studied live: fast (model→filter)
    • slow (monitor→filter). Prior links selection→collapse offline / does streaming selection with no health loop.
  • N-B — Does the budget flip survive co-adaptation? [NEW] Coverage wins at small budgets, novelty at large — does the flip hold under single-pass causal selection against a model the selection is itself changing?
  • N-C — Self-reinforcing selection (in)stability [NEW]. Selection drives the model, which changes what is selected next — may degenerate or be stabilized; and how FL averaging amplifies or damps it.
  • N-D — Selection × aggregation skew in FL [NEW]. Independent clients over-selecting local tails skew the aggregate; global-aware selection may correct it.
  • N-E — Online health-monitor → control loop [NEW]. A multivariate health signal as a live controller (not a diagnostic) — modulating selection, scheduling replay/rollback.
  • N-F — Federation-level health analytics [NEW]. Flag a degraded client via representation health before its update poisons the global model — the server-side twin of the on-device monitor.
  • N-G — Health-steerable filter (the actuator) [NEW]. Coverage↔novelty mix α + acceptance rate as a live function of the health signal — the actuator that makes the loop closable.

Everything else is [STD] or [EXT]. New work maps to one of N-A…N-G, or it is scope drift.

Plan

Core philosophy: the filters are knobs, not the result; the high level system design is ours. Lead with the cheap existential system-wide experiment, not a filter design. Filter (or other sub-component design) can come later, once the system dynamics are studied (using known subcomponents).

High level methodology

  1. Build the streaming loop + measurement apparatus (effective rank / RankMe, probe-on-past, frozen-backbone baseline B5, etc.) and a positive control per failure mode (see Phase 0). Loop is composed of known methods.

  2. With cheap knobs only (no-filter / random / dedup / loss), on a correlated stream, establish:

  3. (a) model adaptation beats frozen model

  4. (b) streaming induces measurable degradation
  5. (c) the knob measurably moves the health trajectory.

If (a) or (b) fails, reframe immediately. The steerable filter and the closed loop (N-E, N-G) come last, once the pathologies are shown to exist and the monitor is built. We can't close a loop to correct a problem we haven't demonstrated.

Implementation

The building blocks (possible implementations of the individual elements of the figure in the Scope section) to consider are listed in building-blocks.md. Using these, we design a sequential scientific spine that underpins our experiment plan and implementation roadmap.

  • Phase 0 — Instrument calibration [STD]. Each failure mode's instruments (metrics) calibrated — its PC fires, a healthy baseline stays quiet.
  • Phase 1 — Degradation envelope [NEW]. 1a STL-10 pilot (synthetic correlation) → 1b BDD100K (real, make-or-break); sweep pressure incl. initialization Go: a coupling exists. No-Go: reframe to selection-for-efficiency.
  • Phase 2 — Open-loop criterion study [NEW] Centralized; Explore budget flip: at small budgets coverage wins, at large budgets novelty wins. Does the flip survive co-adaptation; does the loop self-reinforce?
  • Phase 3 — Closed loop [NEW] Centralized; build the monitor→filter controller; open vs. closed-loop ablations on health + downstream.
  • Phase 4 — ZOD confirmation [EXT] (pretrained-adapt, realistic scale)
  • Parallel tracks. FL, Monitoring (built once Phase 1 confirms degradation, ready by Phase 3).

Where to read more

Working on… Read
A [NEW] contribution / positioning vs. prior art novelty.md
SSL backbones, filter families (F-a/F-b/F-c), datasets building-blocks.md
Setting up / running a phase — baselines, factor levels, grids experiments.md
Measurements, health monitor, evaluation protocols metrics.md
Citations / lit-review reading-list.md
Phase planning, interpreting a null risks.md