Deploying ML Models
Training a model is only half the battle. Deploying it — making it available to real users in production — is where the real engineering happens. A model sitting in a Jupyter notebook helps nobody.
Deployment means turning your model into a service that applications can call reliably, quickly, and at scale.
Model Formats
Before deploying, you need to save your model in a format suitable for production:
Pickle/joblib: Python's native serialization. Simple but tied to Python. Don't use for untrusted environments — pickle can execute arbitrary code.
ONNX (Open Neural Network Exchange): Framework-agnostic format. Train in PyTorch, deploy in TensorFlow or C++. Great for interoperability.
TorchScript / SavedModel: Framework-specific optimized formats. PyTorch and TensorFlow respectively. Best performance within their ecosystems.
TensorRT: NVIDIA's optimized format for GPU inference. Can give 2-5x speedup over standard models.
Model Export Flow
──────────────────────────────────────────────
│ │
│ Training Framework │
│ ┌──────────────────┐ │
│ │ PyTorch / TF │ │
│ │ Train model │ │
│ └────────┬─────────┘ │
│ │ │
│ ▼ Export │
│ ┌──────────────────┐ │
│ │ .pt / .h5 / .pb │ ← native format │
│ └────────┬─────────┘ │
│ │ │
│ ▼ Convert (optional) │
│ ┌──────────────────┐ │
│ │ .onnx / .trt │ ← optimized format │
│ └──────────────────┘ │
──────────────────────────────────────────────
Flask API
Flask is the simplest way to turn a model into a web API. A few lines of Python and you have a prediction endpoint.
Flask Deployment Architecture
──────────────────────────────────────────────
│ │
│ Client Request │
│ │ │
│ ▼ │
│ ┌──────────┐ ┌──────────────┐ │
│ │ Flask │───▶│ Load Model │ │
│ │ Server │ │ (once at │ │
│ │ :5000 │ │ startup) │ │
│ └──────────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────┐ │
│ │ Process │ │
│ │ Input │ │
│ │ Preprocess│ │
│ │ Predict │ │
│ │ Postprocess│ │
│ └──────────┘ │
│ │ │
│ ▼ │
│ JSON Response │
│ {"prediction": "cat", "confidence": 0.95} │
──────────────────────────────────────────────
Key considerations: load the model once at startup, not per request. Use JSON for input/output. Add input validation. Handle errors gracefully. Add logging for debugging.
Docker Containerization
Docker packages your model and all its dependencies into a container that runs identically everywhere. No more "it works on my machine" problems.
Without Docker: With Docker:
──────────────────────────────────────────────
│ Dev: Python 3.8 │ ┌──────────────┐ │
│ Prod: Python 3.6 │ │ Container │ │
│ Result: BROKEN │ │ ┌──────────┐ │ │
│ │ │ │ Your App │ │ │
│ "Works on my machine!" │ │ │ Python │ │ │
│ │ │ │ Libs │ │ │
│ │ │ └──────────┘ │ │
│ │ └──────────────┘ │
│ │ Runs anywhere │
──────────────────────────────────────────────
A Dockerfile for ML is straightforward: start from a Python base image, install dependencies, copy your model and code, expose the port, and run the Flask server.
Docker Compose helps when you need multiple services — model server, database, cache, monitoring. Define everything in one file and spin up the whole stack with one command.
Model Serving Alternatives
Flask is great for prototyping, but production often needs more:
FastAPI: Like Flask but faster (async support) and automatic API documentation. Built-in data validation with Pydantic. Increasingly popular for ML APIs.
TensorFlow Serving: Purpose-built for TF models. Handles versioning, batching, and GPU optimization out of the box.
TorchServe: PyTorch's equivalent of TF Serving. Model versioning, multi-model serving, and metrics.
KServe: Kubernetes-native model serving. Scales automatically, supports multiple frameworks.
Serving Options Comparison
──────────────────────────────────────────────
│ Option │ Best For │
──────────────────────────────────────────────
│ Flask │ Prototyping, small scale│
│ FastAPI │ Production APIs │
│ TF Serving │ TensorFlow at scale │
│ TorchServe │ PyTorch at scale │
│ Triton │ Multi-framework, GPU │
│ KServe │ Kubernetes deployments │
──────────────────────────────────────────────
Monitoring & Maintenance
Deployment isn't the finish line — it's the starting line. Models degrade over time as real-world data diverges from training data.
Data drift: The distribution of incoming data changes. Your model trained on 2023 data struggles with 2024 data. Monitor feature distributions and trigger retraining when drift exceeds thresholds.
Concept drift: The relationship between features and targets changes. Spam patterns evolve. Fraud techniques adapt. Your model needs to keep up.
Performance monitoring: Track prediction latency, throughput, error rates, and accuracy metrics in real-time. Set alerts for anomalies.
Production ML Pipeline
──────────────────────────────────────────────────
│ │
│ Data ──▶ Feature ──▶ Model ──▶ Serving │
│ Pipeline Store Registry API │
│ │
│ ┌─────────────────────────────────────────┐ │
│ │ Monitoring Layer │ │
│ │ - Data drift detection │ │
│ │ - Model performance tracking │ │
│ │ - Latency & error rate metrics │ │
│ │ - Alerting & retraining triggers │ │
│ └─────────────────────────────────────────┘ │
──────────────────────────────────────────────────
Scaling Considerations
When your model gets popular, you need to handle load:
Horizontal scaling: Run multiple instances behind a load balancer. Simple, effective, works with any model.
Batching: Group multiple predictions into one batch. GPUs love batches — processing 32 images at once is almost as fast as processing one.
Caching: Store predictions for repeated inputs. If the same image comes in twice, return the cached result instantly.
Model optimization: Quantize, prune, or distill your model for faster inference. A 4x speedup from quantization is often worth the tiny accuracy drop.
Deployment Checklist
Before going live, ensure you have:
Input validation (reject bad data gracefully), error handling (don't crash on unexpected inputs), logging (for debugging and auditing), health checks (know if the service is up), versioning (rollback if something breaks), and A/B testing capability (compare model versions).
Start simple with Flask + Docker, then optimize as needed. Over-engineering early is a common and expensive mistake.