MLflow: Tracking Experiments and Managing Models

The Problem: Ananya's Lost Formula

Ananya, a Grade 9 student, keeps a small table in the back of her notebook: how many hours she studied before each of her last four unit tests, and the marks she scored. She wants to build a simple formula that predicts a test score from study hours, of the form predicted mark = slope × hours + intercept. On Monday evening she tries one pair of numbers for slope and intercept. On Tuesday she tries another pair, convinced it is better, but by Wednesday she cannot remember which pair actually gave the smaller error, because she scribbled both attempts on the same page and crossed one out. She has, without knowing it, run into the single most common problem in real machine learning work: the moment you try more than two or three variations of anything, an unaided memory — or a messy notebook — stops working as a record-keeping system. This chapter is about the tool built specifically to solve that problem, MLflow, and along the way you will learn exactly what it means to "track an experiment" and "manage a model," two phrases used constantly in real ML projects but rarely explained from first principles.

A Number to Judge a Guess: Mean Absolute Error

Before comparing two guesses, you need one number that says how good each guess is. Ananya's real data from her last four unit tests looks like this:

Hours studied: 2, 4, 6, 8   |   Marks scored (out of 100): 34, 55, 64, 90

She wants a formula predicted = slope × hours + intercept — the same straight-line equation, y = mx + c, from CBSE Grade 9 mathematics, with slope playing the role of m and intercept playing the role of c. For any chosen slope and intercept, she can compute a predicted mark for each of the four hours values, compare it against the actual mark, and measure how far off she was. Since being 5 marks too high and being 5 marks too low are both equally "wrong" for judging a formula, we drop the sign of each error and look only at its size — its absolute value. Averaging these four absolute errors gives the Mean Absolute Error, or MAE: the average distance, in marks, between what the formula predicted and what actually happened. A smaller MAE means a better formula.

Try Guess 1: slope = 5, intercept = 30.

predicted = 5×2+30, 5×4+30, 5×6+30, 5×8+30 = 40, 50, 60, 70
errors = |34−40|=6, |55−50|=5, |64−60|=4, |90−70|=20
sum of errors = 6+5+4+20 = 35  →  MAE = 35 ÷ 4 = 8.75

Now try Guess 2: slope = 8, intercept = 20.

predicted = 8×2+20, 8×4+20, 8×6+20, 8×8+20 = 36, 52, 68, 84
errors = |34−36|=2, |55−52|=3, |64−68|=4, |90−84|=6
sum of errors = 2+3+4+6 = 15  →  MAE = 15 ÷ 4 = 3.75

Guess 2 is clearly better — its average error is under 4 marks instead of nearly 9. Ananya just did, by hand, exactly what every machine learning workflow does at its core: pick some settings, run the formula, measure the error, and compare. The trouble starts at scale. Real models rarely have just two adjustable settings; they often have five, ten, or dozens of them (learning rate, tree depth, how much past data to include, and so on), and results are often judged by more than one metric at once (error, accuracy, how long training took). Re-copying every combination of settings and every result into a notebook or spreadsheet by hand is slow, and one mistyped number silently corrupts the comparison. Worse, a spreadsheet cannot store the actual trained model file for you to reuse later — only the numbers you remembered to type in. This exact gap is what MLflow closes.

What MLflow Actually Is

MLflow is a free, open-source software library, first released by Databricks in 2018 and now developed as an open-source project used widely across the machine learning industry. It is important to be precise about what MLflow does and does not do: it does not build formulas, choose settings, or learn anything by itself. What it gives you is a small set of function calls that, every time your code runs, automatically write down which settings you used and what result you got — a permanent, searchable lab notebook that a computer keeps for you instead of you keeping it by hand on paper.

MLflow is organized into several components. The two that match this chapter's title exactly are:

• MLflow Tracking — records every run of your code: its settings, its results, and any files it produced.
• MLflow Model Registry — once you have tracked many runs and picked a winner, the Registry is where you formally store, version, and label that winning model so it can be found and reused with confidence later.

(MLflow also includes components called Projects, for packaging code so it reruns identically on another machine, and Model Serving, for exposing a trained model over a web request. Both are beyond what this chapter covers.)

The Vocabulary: Experiment, Run, Parameter, Metric, Artifact

Five terms, precisely defined, since CBSE-style questions often test exact terminology rather than vague description:

• Experiment — a named container for a group of related attempts at the same problem. Ananya's would be named something like exam-score-predictor.
• Run — one single execution of your code with one specific choice of settings. Guess 1 is a run; Guess 2 is a different run. Every call to mlflow.start_run() begins a new run.
• Parameter — an input value chosen before the run started. slope = 5 and intercept = 30 are the parameters of Guess 1.
• Metric — a number measuring how good the result was, computed after the run finishes. mae = 8.75 is the metric of Guess 1. Unlike a parameter, a metric can be logged several times across one run — for a model that improves step by step, you might log accuracy after every step, and MLflow keeps every value so you can later see how it changed over time.
• Artifact — any file a run produced that isn't a single number: a trained model file, a plot, a CSV of predictions. Guess 1 and Guess 2 produce no artifact at all, because the "model" is nothing more than two plain Python numbers living inside the script — there is no separate file to save. That changes later in this chapter, once a real training algorithm builds an actual model object.

Logging Our Two Runs with MLflow

Here is Ananya's exact comparison, rewritten with MLflow doing the record-keeping instead of her notebook:

import mlflow

hours = [2, 4, 6, 8]
actual_scores = [34, 55, 64, 90]

def predict(hours, slope, intercept):
    return [slope * h + intercept for h in hours]

def mean_absolute_error(actual, predicted):
    errors = [abs(a - p) for a, p in zip(actual, predicted)]
    return sum(errors) / len(errors)

# Run 1: first guess
with mlflow.start_run(run_name="guess-1"):
    slope, intercept = 5, 30
    predicted = predict(hours, slope, intercept)
    mae = mean_absolute_error(actual_scores, predicted)
    mlflow.log_param("slope", slope)
    mlflow.log_param("intercept", intercept)
    mlflow.log_metric("mae", mae)
    print(f"guess-1 -> MAE = {mae}")

# Run 2: second guess
with mlflow.start_run(run_name="guess-2"):
    slope, intercept = 8, 20
    predicted = predict(hours, slope, intercept)
    mae = mean_absolute_error(actual_scores, predicted)
    mlflow.log_param("slope", slope)
    mlflow.log_param("intercept", intercept)
    mlflow.log_metric("mae", mae)
    print(f"guess-2 -> MAE = {mae}")

Tracing this line by line: hours and actual_scores hold Ananya's four data points. predict() builds a list of four predicted marks using list comprehension. mean_absolute_error() pairs up actual and predicted marks with zip, takes the absolute difference of each pair, and averages them — precisely the hand calculation done above. Inside the first with mlflow.start_run(run_name="guess-1"): block, a new run named "guess-1" opens; slope, intercept = 5, 30 sets the parameters; predicted becomes [40, 50, 60, 70]; mae evaluates to 8.75, matching the hand-worked value exactly. mlflow.log_param("slope", 5) and mlflow.log_param("intercept", 30) record the two chosen settings against this run; mlflow.log_metric("mae", 8.75) records the result. The printed line reads guess-1 -> MAE = 8.75. The moment the with block ends, MLflow automatically marks the run as finished — that is precisely why start_run is used as a context manager here rather than calling a separate mlflow.end_run(): logging calls made outside an open run would fail, so wrapping them in with guarantees every log_param and log_metric lands inside the correct, still-open run. The second block repeats the same steps for slope, intercept = 8, 20, producing predicted = [36, 52, 68, 84], mae = 3.75, and the printed line guess-2 -> MAE = 3.75 — again matching the hand calculation.

Where MLflow Stores All This: The Tracking Store and the UI

Run this script from an ordinary terminal, and MLflow silently creates a folder named mlruns/ next to it. Inside, every run gets its own subfolder holding small files with its parameters, its metrics, and any artifacts, along with the exact start and end time. This local folder is the default tracking store. In a real organisation, teams usually point MLflow at a shared database or a remote tracking server instead, so every teammate's runs land in one shared place — but the function calls in the code above, log_param and log_metric, stay identical no matter where the data ends up being stored.

To browse those runs without opening raw files by hand, one command is run from the same folder in the terminal:

mlflow ui

This starts a small local web server, normally reachable at http://127.0.0.1:5000, showing a table with one row per run and one column per parameter and metric — exactly the comparison Ananya was attempting by hand in her notebook, except sortable by any column, searchable, and impossible to accidentally lose.

Common Misconception: "MLflow Trains the Model"

A very natural but incorrect assumption is that calling MLflow functions somehow makes the model itself better, faster, or smarter. It does not. Look again at the code above: mean_absolute_error() and the formula inside predict() do all of the actual computation. mlflow.log_param and mlflow.log_metric run strictly after those numbers already exist — they only copy already-computed values into a permanent record. Delete every MLflow line from the script above and Guess 1 still evaluates to MAE 8.75; the prediction and its accuracy are entirely unaffected. MLflow is a record-keeper sitting beside your code, never a participant inside the computation.

A second, subtler point about vocabulary is worth naming explicitly. In Ananya's two runs, slope and intercept were logged with mlflow.log_param because they truly were settings a human typed in by hand — nothing in the code searched for them. That usage matches what real machine learning calls a hyperparameter: a setting chosen by the person running the experiment (learning rate, number of trees, and so on), logged with log_param before training even starts. It is easy to assume every number describing a model always gets logged this same way. It does not. As the next section shows, once an algorithm learns a slope and intercept for itself, those learned numbers are not typed in by a human and are not logged individually with log_param — they live inside the saved model object instead, as an artifact.

From a Formula to a Real Model: scikit-learn and Artifacts

Ananya's slope and intercept were guesses. Real machine learning normally lets an algorithm search for the best-fitting slope and intercept itself, using the same hours-studied data. scikit-learn's LinearRegression, part of the CBSE Informatics Practices and Computer Science Python toolchain, does exactly this: given the data, it mathematically finds the slope and intercept that minimise the total squared error, a method called least-squares fitting. Deriving least-squares is beyond this chapter, but the important shift for our purposes is where the numbers now live: the learned slope and intercept sit inside a Python object, not typed into the script by a human. That object needs to be saved to a file if it is ever going to be reused without retraining from scratch — and that saved file is exactly what MLflow calls an artifact.

from sklearn.linear_model import LinearRegression
import mlflow
import mlflow.sklearn

# X_train, y_train, mae computed earlier in your training script
model = LinearRegression().fit(X_train, y_train)

with mlflow.start_run(run_name="sklearn-linear-regression"):
    mlflow.log_param("model_type", "LinearRegression")
    mlflow.log_metric("mean_absolute_error", mae)
    mlflow.sklearn.log_model(model, artifact_path="model")
    # Newer MLflow releases are moving toward a `name` argument
    # instead of `artifact_path`; both work in current stable MLflow.

LinearRegression().fit(X_train, y_train) runs the least-squares algorithm and returns model, an object holding the learned slope (accessible as model.coef_) and learned intercept (model.intercept_) — computed by scikit-learn, not chosen by hand. mlflow.log_param("model_type", "LinearRegression") records which algorithm was used, a genuine hyperparameter-style choice made by the programmer. mlflow.log_metric records the error, exactly as before. The new line is mlflow.sklearn.log_model(model, artifact_path="model"): this serialises the entire trained object — algorithm type, learned coefficients, and enough metadata to reload it later with mlflow.sklearn.load_model(...) — and saves it as a folder of files under this run's artifacts. This is precisely why the earlier guess-1 and guess-2 runs never called anything like log_model: our plain-Python formula never created a model object, only two numbers plugged directly into an arithmetic expression, so there was never a file to save.

The Model Registry: A Version-Controlled Shelf for Models

After tracking many runs — different algorithms, different slices of data — one clear winner emerges by MAE. The Model Registry is where that winning run's saved model is formally registered under a permanent name, with a version number attached automatically:

mlflow.register_model(
    model_uri="runs:/<run_id>/model",
    name="ExamScorePredictor"
)

Here <run_id> is the unique identifier MLflow assigned automatically when that particular run started, and model_uri="runs:/<run_id>/model" points at the specific artifact folder saved by log_model inside that run — the string "model" matching the artifact_path used earlier. The first time this line runs for a given name, it creates ExamScorePredictor version 1. Register an improved model a week later under the same name, and it becomes version 2 — the registry keeps every version and its full history rather than silently overwriting version 1, unlike saving over a single file called model.pkl on a laptop, where the previous version is simply gone.

Earlier releases of MLflow let a registered version be marked with a global stage label, such as Staging or Production, directly on the model. Current MLflow (2.9 and later) instead recommends attaching short, movable aliases — such as @champion — to whichever version is currently trusted, because a single global stage name caused confusion when the same model needed to be "in production" in more than one place at once. The underlying idea is unchanged either way: the registry keeps a clear, current pointer to the version actually in use, kept separate from the full version history sitting behind it.

Check Your Understanding

• In Ananya's guess-1 run, is intercept a parameter, a metric, or an artifact? Justify the answer in one sentence.
• Using the same four data points (hours 2, 4, 6, 8; marks 34, 55, 64, 90), compute the MAE for a third guess of slope = 10, intercept = 10. Show the predicted marks and each absolute error before averaging.
• Explain, using the vocabulary from this chapter, why guess-1 and guess-2 logged no artifact while the scikit-learn run did.
• A classmate claims, "MLflow makes my model more accurate." Identify exactly what is wrong with that sentence.
• State one concrete difference between what MLflow Tracking stores and what the MLflow Model Registry stores.

Answer key: (1) A parameter — it is a value chosen by Ananya before the run started, not something measured afterwards. (2) predicted = 30, 50, 70, 90; errors = |34−30|=4, |55−50|=5, |64−70|=6, |90−90|=0; sum = 15; MAE = 15 ÷ 4 = 3.75 (tied with guess-2). (3) The plain-Python runs never created a model object — the formula was two numbers used directly in arithmetic — so there was no file for log_model to save; the scikit-learn run produced an actual trained LinearRegression object that had to be serialised to be reusable later. (4) MLflow never touches the computation that produces predictions or errors; it only records values that already exist, after they are computed. It cannot make any model more accurate. (5) Tracking stores the history of every individual run (its parameters, metrics, and artifacts, including failed or discarded attempts); the Registry stores only the specific models a human has deliberately chosen to register, each under a permanent name with its own version number.

Summary

Comparing model attempts by memory or by a hand-kept notebook breaks down almost immediately once more than a couple of settings are involved — Ananya's own two guesses were already hard to keep straight. MLflow Tracking solves this by giving code two simple function calls, mlflow.log_param for a chosen setting and mlflow.log_metric for a measured result, wrapped inside a with mlflow.start_run(): block that groups them into one named run; every run lands automatically in a tracking store (a local mlruns/ folder by default) that the mlflow ui command turns into a sortable comparison table. A run additionally gets an artifact only when it produces an actual file worth saving, most commonly a trained model object logged with a function such as mlflow.sklearn.log_model — which is why the hand-built exam-score formula never had one, while the scikit-learn version did. Once a winning model is identified across many tracked runs, mlflow.register_model promotes it into the Model Registry, where it receives a permanent name and an incrementing version number, with aliases available to mark whichever version is currently trusted. Throughout all of this, MLflow itself never trains, predicts, or learns anything — it only records, organises, and versions the results of computations your own code has already performed.

Think About It

Think about this: How would you explain mlflow: tracking experiments and managing models to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.