Deploying ML Models to Production

Suppose you have just finished a Computer Science project: a fraud-detection model that looks at a UPI transaction and predicts whether it is suspicious, trained and tested inside a Jupyter notebook on your laptop. It scores well on your held-out test data, your teacher is impressed, and the project feels complete. But a real UPI app is not a notebook. It is a live service that people tap "Pay" on hundreds of times a second, expecting an answer in about a second or two, every single day, forever. The moment a model leaves the notebook and starts answering real requests, a whole new set of problems appears that test accuracy never measured. Can it produce an answer fast enough that the payment screen doesn't visibly freeze? What happens when thousands of people are paying at the same moment during a festival sale? What happens six months later, once spending patterns have shifted and the model is quietly making worse decisions than the day it was trained — with nobody watching? None of these are training problems. They are deployment problems, and they are what this chapter is about: how a trained model becomes a running, monitored, safely-updatable part of a real system.

A trained model is just a file

The first idea to get straight is a simple but easy-to-miss one: once training finishes, a model is nothing more than a file on disk — a set of learned numbers saved in a particular format. Training is the expensive, one-time (or occasional) process of searching for those numbers. Deployment is the completely separate job of taking that saved file and making it answer questions for real users, quickly and reliably, again and again.

In Python, saving a trained model usually looks like this:

import joblib
from sklearn.linear_model import LogisticRegression

# TRAINING happens once, offline, on a powerful machine.
# It can take minutes or hours -- nobody is waiting on it live.
model = LogisticRegression()
model.fit(X_train, y_train)

# Save the learned numbers to a file. This file is called
# a "model artifact" -- it is the entire product of training.
joblib.dump(model, "fraud_model_v2.joblib")

That .joblib file contains everything the model learned: the coefficients, the bias term, the structure. It does not contain the training data, and it does not need Python's fit() function ever again. From this point on, "using the model" just means loading that file and calling one fast function on it — which is a completely different job from training it.

From file to service: the inference server

A file sitting on a laptop cannot answer a live UPI request. It needs to live inside a program that is always running, listening for incoming transactions, and replying immediately. This always-on program is called an inference server (inference just means "using a trained model to make a prediction," as opposed to training it). A minimal version looks like this:

# model_server.py -- this program runs continuously
import joblib

model = joblib.load("fraud_model_v2.joblib")

def check_transaction(amount_hundreds, hour):
    """Called every time a new UPI transaction needs checking."""
    features = [[amount_hundreds, hour]]
    fraud_probability = model.predict_proba(features)[0][1]
    if fraud_probability > 0.90:
        return "BLOCK"
    return "ALLOW"

Notice what changed. Training code runs once and is thrown away after it produces the artifact. Serving code runs forever, gets called by strangers thousands of times a day, and a single bug in it can block a real person's real payment. This is why software engineers treat the inference server as production software — with error handling, logging, and testing — even though the "intelligence" inside it is just a handful of numbers from a .fit() call.

Worked example: tracing a prediction by hand

To see exactly what the inference server computes, let's build a tiny fraud model ourselves and trace every number by hand. Our model looks at two features of a transaction: the amount, measured in hundreds of rupees (so 150 means ₹15,000), and the hour of day in 24-hour format (0 = midnight). Here is the training data:

Amount (x Rs.100)   Hour (24-hr)   Fraud?
        2                14          No
      150                 2          Yes
        5                10          No
      200                 3          Yes
        8                18          No
      180                 1          Yes

Three of these transactions are large amounts in the early hours of the morning (₹15,000 at 2 am, ₹20,000 at 3 am, ₹18,000 at 1 am) and are labelled fraudulent. The other three are small amounts during ordinary daytime hours (₹200 at 2 pm, ₹500 at 10 am, ₹800 at 6 pm) and are labelled genuine. A logistic regression model trained on this data would search for two weights and a bias term that separate these six points. Running .fit() on real data produces long decimal coefficients found by an optimization process; to keep the arithmetic traceable by hand, we'll use rounded weights that correctly separate all six points above:

w_amount = 0.1
w_hour   = -0.2
bias     = -9

The model combines the two features into a single number, z, using the same kind of weighted sum you've already seen in linear regression:

z = (w_amount x amount) + (w_hour x hour) + bias

Then z is squeezed into a probability between 0 and 1 using the sigmoid function, P = 1 / (1 + e^(-z)). A very negative z gives a probability near 0; a very positive z gives a probability near 1.

Let's check this model actually agrees with the training labels. Take the second row: amount = 150, hour = 2.

z = (0.1 x 150) + (-0.2 x 2) + (-9)
  = 15 - 0.4 - 9
  = 5.6

A z of 5.6 is strongly positive, so the sigmoid output will be close to 1 — the model says "fraud," matching the label. Now the first row: amount = 2, hour = 14.

z = (0.1 x 2) + (-0.2 x 14) + (-9)
  = 0.2 - 2.8 - 9
  = -11.6

Strongly negative, so the sigmoid output is close to 0 — "not fraud," also matching the label. You can check the remaining four rows the same way; all six come out correctly classified.

Now let's deploy this exact model and send it a transaction it has never seen: amount = 175 (₹17,500), hour = 2 (2 am).

z = (0.1 x 175) + (-0.2 x 2) + (-9)
  = 17.5 - 0.4 - 9
  = 8.1

P(fraud) = 1 / (1 + e^(-8.1))
         ~= 1 / (1 + 0.0003)
         ~= 0.9997   =>   99.97%

Our check_transaction() function compares this to the 0.90 threshold, sees 0.9997 > 0.90, and returns "BLOCK". Now try an ordinary transaction: amount = 6 (₹600), hour = 12 (noon).

z = (0.1 x 6) + (-0.2 x 12) + (-9)
  = 0.6 - 2.4 - 9
  = -10.8

P(fraud) = 1 / (1 + e^(10.8))
         ~= 1 / (1 + 49,021)
         ~= 0.00002   =>   0.002%

Comfortably below the threshold, so check_transaction(6, 12) returns "ALLOW". This entire computation — two multiplications, two additions, and one call to an exponential function — is what actually runs inside the inference server every time a real payment comes through. It takes a modern computer a fraction of a millisecond, which matters a great deal, as you'll see shortly.

The misconception: "if it worked in the notebook, it will work in production"

A very common and very costly assumption is that once a model scores well on a notebook's test set, deploying it is just a matter of copying the file over. In reality, a model's live performance can be badly wrong even when the exact same weights are used, for two distinct reasons — together called training-serving skew.

Cause 1: the features are computed differently online than offline. Our model expects "amount" measured in hundreds of rupees. Suppose the engineer writing the live server forgets this convention and passes the raw rupee amount instead. A completely ordinary ₹600 lunch payment at noon would then be sent to the model as amount = 600 instead of amount = 6:

z = (0.1 x 600) + (-0.2 x 12) + (-9)
  = 60 - 2.4 - 9
  = 48.6   =>   P(fraud) ~= 100%

Nothing is wrong with the trained weights at all — the bug is purely in how a feature was computed at serving time versus training time — yet every single normal transaction now gets blocked as fraud. This is one of the most common real deployment failures, and it is invisible in the notebook because the notebook always computes features the "correct" way.

Cause 2: the world changes after training. Even with correct features, a model trained on last year's spending habits is answering questions about this year's transactions. If UPI transaction limits rise, or if festival-season shopping legitimately produces more large, late-evening payments, the pattern "large amount at an odd hour = fraud" that the model learned may simply stop being reliable. This is called data drift (the incoming data no longer looks like the training data) or, when it specifically affects the fraud/not-fraud relationship, concept drift. No bug exists anywhere in the code — the model has just quietly become outdated.

Both causes point to the same lesson: passing a notebook's test set is necessary but nowhere near sufficient. A production system needs its serving code tested against the exact same feature logic as training, and it needs ongoing monitoring to catch drift after launch — which is covered later in this chapter.

Batch inference versus real-time inference

Not every prediction needs to happen the instant a request arrives. There are two very different serving patterns, and choosing the right one is itself a deployment decision.

Real-time (online) inference answers one request at a time, as it arrives, and the caller is actively waiting for the reply. The UPI fraud check above is real-time: the payment cannot complete until check_transaction() returns an answer, so the model must be fast, and the server must always be running.

Batch inference processes a large pile of inputs all at once, on a schedule, with no one waiting live. For example, IRCTC might run a nightly job that scores every account created that day for suspicious booking patterns, writing the results to a report for a human review team the next morning. Nobody is staring at a loading spinner while this runs, so it can take minutes, and it can even use a bigger, slower, more accurate model than the real-time system could ever afford.

The choice matters because it changes the engineering constraints completely: real-time systems are built around strict speed limits and high availability; batch systems are built around throughput and can trade speed for accuracy.

Latency and throughput: the arithmetic that decides what you can deploy

Two numbers dominate every real-time deployment decision. Latency is how long one prediction takes. Throughput is how many predictions the system can handle per second.

Suppose product design says a UPI payment must confirm within a 2000-millisecond (2-second) budget so it feels instant to the user. The network round-trip, the bank's ledger check, and formatting the response together already use up 1850 ms of that budget, leaving only 150 ms for the fraud model. Our logistic regression above runs in a fraction of a millisecond — no problem. But suppose a different team wanted to deploy a large deep neural network that takes 300 ms per prediction. That single model call alone blows through the entire remaining budget, and the payment would visibly lag. This is a genuine engineering trade-off: either shrink the model (fewer layers, quantized numbers, a smaller architecture) or accept the lag — accuracy gains on a notebook's test set do not matter if the model cannot fit inside the latency budget.

Now consider throughput. Suppose, at peak festival-sale traffic, the fraud-checking service must handle 2,000 transactions every second (a round illustrative number for this example, not a disclosed figure from any real company). If each prediction takes 5 ms on one processor core, one core can complete:

1000 ms / 5 ms per prediction = 200 predictions per second (one core)

To reach 2,000 predictions per second, the service needs at least:

2000 / 200 = 10 cores (or 10 copies of the inference server) running in parallel

This is why deployment engineers care about a model's speed almost as much as its accuracy: a model that is 1% more accurate but five times slower can force a company to buy five times the hardware just to keep up with the same traffic.

Shipping an update without breaking production: versioning, canary releases, and rollback

Suppose your team retrains the fraud model on three more months of data and produces fraud_model_v3.joblib, which scores higher on the offline test set than v2. Should the inference server switch every single live transaction to v3 immediately?

Doing that is risky, because an offline test set can never perfectly represent live traffic — this is the same training-serving gap discussed earlier. If v3 has a subtle bug or was trained on a skewed sample, switching 100% of traffic to it at once means every user is affected before anyone notices a problem. Production systems avoid this with a canary release: route only a small slice of real traffic to the new version first, while most traffic keeps using the version already known to work, and compare their behaviour.

Suppose in one hour the fraud service handles 100,000 transactions. A 5% canary sends 5,000 of them to v3 and keeps the remaining 95,000 on v2. The team already knows v2's baseline false-negative rate (fraud it misses) is about 2 in 5,000, i.e. 0.04%. Over that hour, v3's canary group shows 3 missed frauds out of its 5,000, i.e. 0.06% — close enough to be within normal noise, so the team expands the canary to 50%, then eventually to 100%. But suppose instead v3's canary had shown 40 missed frauds out of 5,000 — a rate of 0.8%, twenty times worse than v2's baseline (0.8% / 0.04% = 20). That is far too large to be noise, and an automated monitoring rule would trigger an immediate rollback: every request is routed back to v2, and v3 is pulled from live traffic for investigation — all without a human needing to intervene at 2 am.

This is only possible because each trained model is saved as a distinct, named artifact (fraud_model_v2.joblib, fraud_model_v3.joblib, and so on) rather than overwriting the same file. Versioning turns "which model is currently deciding whether to block your payment" into a single configuration value that can be changed — and instantly reverted — without retraining anything.

Monitoring and the retraining loop

Canary releases only catch problems in the first hour or so after a deployment. Data drift, discussed earlier, appears weeks or months later, gradually, with no single moment where anything obviously "breaks." Catching it requires ongoing monitoring: logging every live prediction and, once the true outcome is known (a transaction was later confirmed fraudulent or genuine, for instance through a customer complaint or a bank chargeback), comparing the model's prediction against reality. If the live error rate drifts noticeably above what was measured at launch, that is the signal to retrain on fresh data and go through the canary process again with the new version. The diagram below shows the complete loop: training produces a versioned artifact, the artifact is loaded by an inference server that a client app calls in real time, and the server's live predictions are continuously monitored, feeding drift signals back into the next round of training.

Check your understanding

1. A model scores 96% accuracy in a notebook. Explain, using the specific failure mode from this chapter, how it could still make wrong predictions once deployed even though the file has not changed at all.
2. Using the weights w_amount = 0.1, w_hour = -0.2, bias = -9, compute z and state whether the model would output "BLOCK" or "ALLOW" for a transaction of amount = 190 (in hundreds of rupees) at hour = 4. (Threshold: block if P > 0.90.)
3. A company's payment confirmation has a 1500 ms total latency budget. Non-model steps already use 1400 ms. Would a fraud model that takes 80 ms per prediction fit inside the remaining budget? Show the subtraction.
4. A real-time inference server must handle 900 predictions per second, and each prediction takes 3 ms on one core. How many cores are needed at minimum? Show the two divisions.
5. During a canary release, a new model version shows a false-negative rate of 12 in 2,000 transactions, while the old version's baseline is 3 in 2,000. Should this trigger a rollback? Justify your answer with the rates as percentages.
6. Explain, in your own words, why a batch inference job (like a nightly report) can safely use a slower, more accurate model than a real-time system serving live payments.

Summary

Training and deployment are two separate jobs joined by a single artifact: a saved file of learned numbers. Deployment means loading that artifact into an always-running inference server that answers real requests within a strict time budget — and choosing between real-time serving, where a user is actively waiting, and batch serving, where a schedule replaces urgency. A model that performs well offline can still fail live because of training-serving skew: either the serving code computes a feature differently than the training code did, or the real world has drifted away from what the training data represented — both are corrected through careful feature-logic testing and ongoing monitoring rather than through retraining alone. Because no offline test set can fully predict live behaviour, new model versions are rolled out gradually through canary releases, with their live error rates compared numerically against the version already in production, and an automatic rollback ready if the numbers move too far in the wrong direction. None of this is optional polish around "the real work" of building the model — for a system making thousands of real financial decisions a second, it is the work that decides whether the model is trustworthy at all.

Think About It

Think about this: How would you explain deploying ml models to production to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where deploying ml models to production is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting deploying ml models to production to at least 3 other topics you have studied.