Real-Time ML Serving Without the PhD: Patterns for Feature Stores and Online Inference

Most ML models don't need real-time serving. A batch job that runs overnight and writes predictions to a database covers 80% of use cases: churn scores, recommendation lists, risk ratings, demand forecasts.

For the other 20%, you need predictions at request time. Fraud detection on a payment. Dynamic pricing as a user browses. Content moderation before a post goes live. These use cases can't wait for a batch job.

The problem: most teams overcomplicate real-time serving. They build streaming architectures where a scheduled batch would suffice, or they hand-roll feature pipelines that a feature store would handle. This article covers the patterns that keep real-time ML simple.

Do You Actually Need Real-Time?

Before building real-time infrastructure, answer this honestly:

Pre-computed predictions are dramatically simpler, cheaper, and more reliable. If your use case tolerates predictions that are minutes or hours old, don't build real-time infrastructure. Write predictions to a key-value store and serve lookups.

Feature Store Architecture

Real-time serving requires features at request time. This is where most teams struggle. The model was trained on features computed from a data warehouse. At inference time, those same features need to be available in milliseconds.

A feature store bridges this gap with two storage layers:

Offline store. Historical feature values for training. Backed by a data warehouse or object storage (Parquet files, Delta Lake). Used during model training and batch feature computation.

Online store. Current feature values for real-time inference. Backed by a low-latency key-value store (Redis, DynamoDB, Bigtable). Updated by batch jobs or streaming pipelines.

The critical contract: the feature computation logic is defined once and produces identical results in both stores. Training reads from offline. Inference reads from online. No skew.

Backfill Strategy

When you add a new feature, you need historical values for training. This means backfilling the offline store from raw data. Common approaches:

• SQL-based backfill. Compute features from the warehouse for historical time windows. Simple but slow for large datasets.
• Replay-based backfill. Replay event streams through your feature computation logic. Accurate but requires event replay infrastructure.
• Incremental backfill. Compute features forward from a known-good starting point. Fastest but can't fill gaps before the start date.

Point-in-Time Correctness

The most subtle bug in feature stores: data leakage through incorrect time joins.

When training a model to predict "will this user churn next month?", you need the feature values as they existed at prediction time, not as they exist today. If your training set uses current feature values to predict historical outcomes, your model sees the future during training and performs worse in production.

Feature stores solve this with point-in-time joins: for each training example, retrieve feature values as of the label timestamp. Feast, Tecton, and most mature feature stores handle this natively.

Online Inference Patterns

Pattern 1: Dedicated Model Server

Deploy the model as a standalone service behind an API. The application sends a request with an entity ID. The model server fetches features from the online store, runs inference, and returns predictions.

Best for: High-traffic, latency-sensitive use cases. Fraud detection, real-time pricing.

Stack: TorchServe, Triton Inference Server, KServe, or a custom FastAPI service. Feature fetch from Redis or DynamoDB.

Trade-offs: Additional infrastructure to manage. Network hop between application and model server adds latency.

Pattern 2: Embedded Model (Sidecar)

Deploy the model alongside the application as a sidecar container or embedded library. No network hop for inference. Features are fetched directly by the sidecar.

Best for: Ultra-low-latency requirements (sub-10ms). Edge deployments. Applications where network latency is unacceptable.

Stack: ONNX Runtime embedded in the application, or a sidecar container with a lightweight serving framework.

Trade-offs: Model updates require application redeployment (unless you implement hot-reloading). Harder to scale inference independently of the application.

Pattern 3: Pre-computed with Real-Time Fallback

Pre-compute predictions for the majority of requests via batch. For requests that can't be served from cache (new users, new items, edge cases), fall back to real-time inference.

Best for: Recommendation systems, personalisation, where most entities have stable predictions but some need fresh scores.

Stack: Batch predictions stored in Redis or a database. Real-time inference service for cache misses.

Trade-offs: Cache invalidation complexity. Two code paths to maintain.

End-to-End Latency Profiling

Real-time ML latency is the sum of its parts. Profile each step independently:

Total budget example: A fraud detection service with a 100ms SLA. Feature fetch: 5ms. Inference: 30ms. Pre/post-processing: 3ms. Network: 5ms. Serialisation: 2ms. Total: 45ms. Comfortable margin.

If any single step dominates your budget, that's where to optimise. Don't optimise model inference if feature fetch takes 80% of your latency.

Getting Started

If you're building real-time ML serving for the first time:

1. Start with pre-computed batch predictions. Prove the use case works before adding latency requirements.
2. Add an online feature store (Feast is the simplest starting point). Validate feature consistency between offline and online.
3. Deploy a dedicated model server (FastAPI + ONNX Runtime is a solid minimal stack).
4. Profile end-to-end latency. Optimise the bottleneck.
5. Add monitoring for feature freshness, inference latency, and prediction drift.

Build the simplest system that meets your latency requirement. Add complexity only when measurement proves you need it.