Every tool in the modern MLOps stack — from cloud infrastructure and data pipelines through model serving, monitoring, and LLMOps — with complete deployment examples for Django, Flask, and FastAPI.
Cloud providers supply on-demand compute, object storage, managed databases, and ML-specific services. The choice between providers usually comes down to your team's existing ecosystem, the managed ML platform on offer, and GPU availability. All three hyperscalers provide a free tier or trial credits, making experimentation accessible before committing.
SageMaker covers the full ML lifecycle (data labelling, training, hosting). EC2 provides GPU instances (p3, p4, g4dn). Lambda runs serverless inference. S3 stores models and datasets. ECR hosts container images.
Vertex AI is the managed ML platform with AutoML, custom training, and model registry. Unique access to TPUs (Tensor Processing Units) for high-throughput deep learning. BigQuery ML runs SQL-based models on petabytes.
Azure ML is the managed platform. Preferred by enterprises already in the Microsoft ecosystem (Active Directory, Office 365). Exclusive Azure OpenAI Service grants API access to GPT-4, DALL-E, and Whisper with enterprise SLAs.
Infrastructure purpose-built for GPU workloads. Offers H100 and A100 clusters at competitive cost-per-FLOP compared to hyperscalers. Kubernetes-native API, low network latency between GPU nodes.
Straightforward hourly GPU pricing with no complex billing. On-demand A10, A100, H100 instances accessible via SSH or JupyterLab. Popular with researchers who need GPU access without a cloud contract.
A container packages your application, runtime, libraries, and system dependencies into a single portable image. This eliminates environment-mismatch failures between development, testing, and production. Orchestrators manage running containers at scale across clusters of machines, handling restarts, scaling, and network routing automatically.
The universal container standard. A Dockerfile defines the environment reproducibly. Docker Compose manages multi-container applications locally (API + Redis + database). Every ML deployment starts here.
Industry standard for running containers at scale. Handles automatic restart of failed containers, horizontal pod autoscaling, rolling updates with zero downtime, and service discovery. Required knowledge for any production ML platform role.
Certified minimal Kubernetes distribution with a ~70MB binary and 512MB RAM footprint instead of 4GB+. Suitable for edge deployments, on-premise servers, and learning Kubernetes without resource overhead.
Helm Charts are versioned, parameterised templates for Kubernetes resources. Instead of maintaining hundreds of lines of YAML, install a community chart with a single command and override values as needed.
Dockerfile for an ML API
FROM python:3.11-slim
WORKDIR /app
# Copy requirements before source code so Docker caches the pip layer
# and only re-installs when requirements.txt changes
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
# Run as non-root user
RUN adduser --disabled-password --gecos '' appuser
USER appuser
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]docker build -t my-ml-api:v1.0 .
docker run -p 8000:8000 my-ml-api:v1.0
docker run -d --name ml-api -p 8000:8000 my-ml-api:v1.0
docker logs ml-api -f
docker exec -it ml-api /bin/bash
docker-compose up -d
docker-compose downPipeline orchestrators schedule and execute the data ingestion, transformation, and loading steps that feed ML training. Feature stores ensure that the feature transformations applied during training are applied identically at inference time — eliminating training-serving skew, which is one of the most common causes of production performance degradation.
Pipelines are defined as Python DAGs (Directed Acyclic Graphs). Rich web UI for monitoring runs, retries, and backfills. 800+ community operators for connecting to databases, cloud storage, and ML services. Battle-tested at Airbnb, Lyft, and major banks.
Decorate any Python function with @flow and @task to make it an orchestrated workflow. Much less boilerplate than
Airflow. Prefect Cloud provides managed deployment, scheduling, and observability.
Models pipelines as Software-Defined Assets — the datasets, models, and metrics your pipeline produces — rather than tasks. This gives automatic lineage tracking, asset materialisation history, and better testing ergonomics.
Build, test, and run pipelines in a visual notebook interface. Each block (data loader, transformer, exporter) is independently runnable with visible output. Supports streaming, batch, and real-time pipelines.
Defines and serves features consistently between training and serving through an offline store (for batch training) and an online store (for low-latency inference). Prevents feature drift by sharing a single definition.
Production-grade managed feature store with sub-10ms online serving, streaming feature pipelines, and monitoring built in. Used by FinTech and e-commerce companies with strict feature SLAs.
Combines feature store, model registry, and experiment tracking in one deployable platform. Available on any cloud or on-premise. Strong support for feature groups, time-travel queries, and Great Expectations integration.
ML reproducibility requires versioning three distinct artefacts: code (Git), data (DVC or LakeFS), and models (MLflow, DVC, or a model registry). Without all three locked to the same commit, reproducing a specific result — or rolling back a bad model — becomes unreliable.
The universal distributed version control system. Non-negotiable for any ML project. Branch per experiment, tag releases, use pull requests to review changes to training code and model configs.
GitHub has the largest open-source community and GitHub Actions for CI/CD. GitLab offers better built-in CI/CD pipelines and a stronger self-hosted option. Bitbucket integrates with Atlassian (Jira, Confluence) for enterprise teams.
DVC stores a small pointer file in Git while the actual large file (dataset, model weights) lives in S3, GCS, or Azure Blob. Also defines reproducible ML pipelines as stages with tracked inputs and outputs.
Git extension that stores large files outside the repository and replaces them with text pointers. Simpler than DVC, natively supported on GitHub and GitLab. Suitable for models under 2GB without pipeline tracking needs.
Provides branches, commits, and merges on top of an existing S3-compatible data lake without copying data. Create a branch to experiment with a new dataset version and merge it only if the model improves.
DVC commands
git init && dvc init
# Configure remote storage
dvc remote add -d myremote s3://mybucket/dvc-store
# Track a dataset
dvc add data/train.csv
git add data/train.csv.dvc .gitignore
git commit -m "track training dataset"
dvc push
# On another machine
git clone https://github.com/you/project
dvc pullWithout experiment tracking, the results of 50 training runs live in a Jupyter notebook with no record of which hyperparameter combination produced the best validation score. These tools automatically log parameters, metrics, and artefacts per run and provide a searchable history. Model registries add lifecycle management: staging, production, and archived versions with rollback capability.
Log metrics and parameters with two lines of code. Built-in model registry, artefact storage, and a local web UI on port 5000. Can be self-hosted on any cloud. The most widely deployed open-source tracking tool.
Real-time metric plots, system resource monitoring, artefact versioning, hyperparameter sweep orchestration, and dataset tables. The standard at most serious ML research teams. Free tier covers individual use.
Covers experiment tracking, model production monitoring, and dataset management in a single platform. Strong auto-logging for PyTorch, TF, and scikit-learn with minimal code changes.
Schema-less metadata store that logs anything associated with an ML run — metrics, images, numpy arrays, interactive charts, dataframes. Particularly strong for teams doing image or NLP experiments with rich visualisation needs.
Experiment tracking, data management, pipeline orchestration, and model serving in one open-source stack. Self-hostable. Automatic logging requires zero code changes in many frameworks — just add the ClearML agent.
MLflow tracking example
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
mlflow.set_experiment("iris-classification")
with mlflow.start_run():
n_est, depth = 100, 5
mlflow.log_param("n_estimators", n_est)
mlflow.log_param("max_depth", depth)
model = RandomForestClassifier(n_estimators=n_est, max_depth=depth)
model.fit(X_tr, y_tr)
acc = accuracy_score(y_te, model.predict(X_te))
mlflow.log_metric("accuracy", acc)
mlflow.sklearn.log_model(model, "random_forest")
# View runs: mlflow ui → http://localhost:5000ML CI/CD automates the path from a code commit to a deployed model update. A complete pipeline runs unit tests on data processing code, optionally retrains if training data changed, evaluates the new model against the previous production model, and promotes it only if metrics improve. Failed quality gates block the deployment automatically.
Workflows live in .github/workflows/*.yml alongside
your code. Triggers on push, pull request, schedule, or custom events. 2000 free minutes/month
for public repos. The natural choice if you already host on GitHub.
Defined in .gitlab-ci.yml at the repo root. No
external service required. Better suited for self-hosted deployments and teams wanting a single
platform for Git, CI, and container registry.
Each workflow step runs in its own container on Kubernetes. Ideal for ML pipelines where different stages need different environments — data preprocessing in a Python container, training in a CUDA container, evaluation in another.
Kubernetes Custom Resource Definitions for composable Tasks and Pipelines. Lower-level than Argo — you assemble reusable building blocks. Foundation of Red Hat OpenShift Pipelines. Suited for teams wanting maximum control.
By iterative.ai (the DVC team). Posts model metrics, comparison tables, and plots as comments directly on a GitHub or GitLab pull request. Reviewers see the metric delta before approving a merge.
Model serving exposes a trained model's inference function over a network. The requirements span a wide range — a Flask wrapper on a single VM for low-traffic internal tools, to a multi-GPU Triton cluster processing thousands of requests per second. The right choice depends on throughput requirements, hardware, latency budget, and operational complexity tolerance.
Async-native, auto-generates OpenAPI documentation at /docs, and uses Pydantic for request/response validation. The default
choice for a new standalone ML API. Full example in the Deployment Guide section.
Serves PyTorch, TF, ONNX, and TensorRT models from a single server. Dynamic batching groups concurrent requests into GPU-optimal batch sizes automatically, maximising utilisation. REST and gRPC endpoints.
Developed by AWS and Meta for PyTorch models. Packages a model into a .mar archive and serves it with built-in batching, model versioning,
A/B testing, and a management REST API.
Production server for TensorFlow SavedModels. Point it at a versioned directory structure and it automatically discovers and serves new model versions with zero downtime. Runs on CPU or GPU.
Packages any model — sklearn, PyTorch, XGBoost, or custom — into a self-contained Bento with its inference API and dependencies. One command deploys to Docker, Kubernetes, Lambda, or the managed BentoCloud.
Kubernetes operators that serve ML models via a custom resource definition — declare your model in YAML, the operator manages the container and scaling. KServe is the newer iteration and supports canary rollouts natively.
Uses PagedAttention to manage KV-cache GPU memory efficiently. Delivers 10–20× higher throughput than naive HuggingFace generation. Exposes an OpenAI-compatible REST API. The production standard for self-hosted LLMs.
ollama run llama3 downloads and runs an open-source
LLM locally. Serves an OpenAI-compatible REST endpoint. Best for local development, prompt
iteration, and private inference without cloud costs.
Hugging Face's inference server for HF Hub models. Flash Attention 2, continuous batching, quantisation (GPTQ, AWQ, bitsandbytes), and streaming via Server-Sent Events. Deploys any supported model with one Docker command.
Models degrade silently. Data drift (the input distribution shifts), concept drift (the relationship between inputs and outputs changes), and upstream data quality issues all cause prediction quality to erode over time. Monitoring catches these problems before they produce material business impact.
Time-series metrics database that scrapes a /metrics HTTP endpoint on your API at regular intervals. Stores
request latency, error rates, prediction counts, and custom ML metrics. Pairs with Alertmanager
for notification routing.
Connects to Prometheus, InfluxDB, Elasticsearch, and 50+ other sources to render live dashboards. Set threshold alerts to fire on Slack, PagerDuty, or email when a metric crosses a boundary.
Open-source library that generates interactive HTML reports and JSON metrics comparing the current data distribution to a reference baseline. Can run as a standalone monitoring service with scheduled checks and a UI.
Managed observability platform. Log predictions and ground truth labels; Arize tracks performance, data quality, and drift over time with UMAP embedding visualisation for diagnosing NLP and vision model failures.
WhyLogs generates statistical sketches (distributions, null rates, cardinality) of data without storing the raw data itself — making it suitable for high-volume or sensitive data. WhyLabs is the managed monitoring dashboard.
LangChain's observability platform. Traces every step in an LLM chain with prompt content, token usage, latency, and cost. Supports human annotation, automatic evaluation datasets, and prompt version management.
Open-source local observability tool from the Arize team. Runs entirely in your environment — no data leaves your machine. Traces LLM calls, visualises embeddings, and evaluates retrieval quality for RAG pipelines.
LLMOps covers the tooling layer built specifically for large language model applications — chaining LLM calls, retrieving relevant context (RAG), routing queries to tools or agents, and evaluating output quality. These tools emerged from 2022 onward and are maturing rapidly.
Chains, agents, tools, and memory abstractions for building LLM applications. Supports 50+ LLM providers and 100+ vector databases. Most popular starting point for RAG systems and multi-step agent workflows.
Specialises in connecting LLMs to structured and unstructured data sources. Best-in-class RAG: ingest PDFs, databases, APIs into a vector store and query them with natural language. Stronger than LangChain for complex retrieval tasks.
deepset's framework for search, question answering, and RAG systems. More structured API than LangChain — better suited for production NLP products requiring strong document store abstractions and evaluation pipelines.
Drag-and-drop UI for building LangChain flows without writing code. Connect LLM nodes, retrievers, memory, and tool nodes visually. Exports flows to a deployable REST API. Useful for rapid prototyping with non-technical stakeholders.
Traces LLM calls with prompt content, token cost, and latency. Manages prompt versions with a registry, collects user feedback, and runs evaluation datasets. Self-hostable open-source alternative to LangSmith.
Training and inference performance depends heavily on hardware-specific optimisations. Compiling a model for a target runtime — TensorRT for NVIDIA GPUs, ONNX Runtime for CPU, OpenVINO for Intel — can reduce inference latency by 2–5× without changing the model architecture.
CUDA is the parallel computing platform that allows Python code to execute on NVIDIA GPUs. cuDNN provides hand-optimised implementations of convolutions, attention, and activation functions. Both install automatically with PyTorch GPU builds.
Takes a trained model and applies layer fusion, FP16/INT8 quantisation, and kernel auto-tuning to produce an engine optimised for a specific NVIDIA GPU. Typically delivers 3–5× lower latency compared to PyTorch eager mode inference.
Export a model from PyTorch or TensorFlow to ONNX once; run it on any hardware via ONNX Runtime. ORT's CPU execution provider consistently outperforms PyTorch on CPU-only inference tasks and supports dynamic batch sizes.
One-line export of HF Transformers models to ONNX, TensorRT, or Intel OpenVINO. Handles graph optimisation and quantisation. Designed so switching inference backend requires minimal code changes.
Intel's toolkit for deploying deep learning on Intel CPUs, integrated GPUs, VPUs, and FPGAs. Relevant when deploying on Intel-based edge devices or on-premise servers where NVIDIA GPUs are absent. INT8 quantisation gives 2–5× CPU inference speedup.
PyTorch to ONNX Runtime
import torch
import onnxruntime as ort
import numpy as np
model.eval()
dummy = torch.randn(1, 10)
torch.onnx.export(
model, dummy, "model.onnx",
input_names = ["features"],
output_names = ["output"],
dynamic_axes = {"features": {0: "batch_size"}}
)
session = ort.InferenceSession("model.onnx")
x = np.random.randn(5, 10).astype(np.float32)
outputs = session.run(None, {"features": x})
print(outputs[0])Three frameworks for exposing a trained model as a web API. The pattern is identical in all three: load the model once when the process starts, accept serialised input over HTTP, run inference, and return a JSON response. The differences are in how much surrounding infrastructure each framework provides and the async model they use.
/docs Swagger UIFastAPI
"""
Run: uvicorn main:app --host 0.0.0.0 --port 8000 --workers 4
"""
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from contextlib import asynccontextmanager
import joblib, numpy as np, logging
logger = logging.getLogger(__name__)
model = None
@asynccontextmanager
async def lifespan(app: FastAPI):
global model
model = joblib.load("model.pkl") # loaded once at startup, shared across all requests
logger.info("Model loaded")
yield
app = FastAPI(title="ML API", version="1.0.0", lifespan=lifespan)
class PredictInput(BaseModel):
features: list[float] = Field(..., min_length=1)
model_config = {"json_schema_extra": {"example": {"features": [1.2, 0.5, 3.1]}}}
class PredictOutput(BaseModel):
prediction: float
@app.get("/health")
def health():
return {"status": "ok", "model_loaded": model is not None}
@app.post("/predict", response_model=PredictOutput)
def predict(req: PredictInput):
if model is None:
raise HTTPException(503, "Model not available")
try:
x = np.array(req.features).reshape(1, -1)
pred = float(model.predict(x)[0])
return PredictOutput(prediction=pred)
except Exception as e:
logger.error(e)
raise HTTPException(500, "Inference failed")
@app.post("/predict/batch")
def predict_batch(inputs: list[PredictInput]):
x = np.array([r.features for r in inputs])
preds = model.predict(x).tolist()
return {"predictions": preds, "count": len(preds)}Flask
"""
Production: gunicorn -w 4 -b 0.0.0.0:5000 app:app
Dev: python app.py
"""
from flask import Flask, request, jsonify, abort
import joblib, numpy as np, logging
app = Flask(__name__)
logging.basicConfig(level=logging.INFO)
try:
model = joblib.load("model.pkl")
except FileNotFoundError:
model = None
app.logger.error("model.pkl not found")
@app.route("/health")
def health():
return jsonify({"status": "ok", "model_loaded": model is not None})
@app.route("/predict", methods=["POST"])
def predict():
if model is None:
abort(503, "Model not loaded")
data = request.get_json()
if not data or "features" not in data:
abort(400, "'features' key required")
try:
x = np.array(data["features"]).reshape(1, -1)
pred = float(model.predict(x)[0])
prob = model.predict_proba(x)[0].tolist() if hasattr(model, "predict_proba") else None
return jsonify({"prediction": pred, "probability": prob})
except Exception as e:
app.logger.error(e)
abort(500, "Inference failed")
@app.errorhandler(400); @app.errorhandler(500); @app.errorhandler(503)
def handle_error(e):
return jsonify({"error": str(e.description)}), e.code
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5000, debug=False)Django with Django REST Framework
# pip install django djangorestframework joblib
# Add 'rest_framework' to INSTALLED_APPS
# apps/predictor/apps.py
from django.apps import AppConfig
import joblib, logging, os
logger = logging.getLogger(__name__)
class PredictorConfig(AppConfig):
name = "predictor"
model = None
def ready(self):
path = os.path.join(os.path.dirname(__file__), "model.pkl")
try:
PredictorConfig.model = joblib.load(path)
logger.info("Model loaded")
except Exception as e:
logger.error(f"Load failed: {e}")
# apps/predictor/serializers.py
from rest_framework import serializers
class InputSerializer(serializers.Serializer):
features = serializers.ListField(child=serializers.FloatField(), min_length=1)
# apps/predictor/views.py
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status
import numpy as np
from .serializers import InputSerializer
from .apps import PredictorConfig
class PredictView(APIView):
def post(self, request):
s = InputSerializer(data=request.data)
if not s.is_valid():
return Response(s.errors, status=status.HTTP_400_BAD_REQUEST)
m = PredictorConfig.model
if m is None:
return Response({"error": "Model unavailable"}, status=503)
x = np.array(s.validated_data["features"]).reshape(1, -1)
pred = float(m.predict(x)[0])
return Response({"prediction": pred})
# apps/predictor/urls.py
from django.urls import path
from .views import PredictView
urlpatterns = [path("predict/", PredictView.as_view())]Deploying the containerised API
# Render — connect GitHub repo, auto-detects Dockerfile, zero config
# AWS ECR + ECS
aws ecr get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com
docker build -t my-ml-api .
docker tag my-ml-api:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/my-ml-api:latest
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/my-ml-api:latest
# Kubernetes deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ml-api
spec:
replicas: 3
selector:
matchLabels:
app: ml-api
template:
metadata:
labels:
app: ml-api
spec:
containers:
- name: ml-api
image: your-registry/my-ml-api:latest
ports:
- containerPort: 8000
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10/health endpoint.
Log every prediction with a unique ID.
Monitor the first 1000 production predictions before trusting automated alerting thresholds.