MLOps: models that survive production · 4. Reproducible training pipelines

2026-11-14来源 dev.toEnglish

MLOps: models that survive production · Chapter 4 of 12 · AI Engineering · new chapter every Thursday night

By the end of this chapter: Turn a notebook into something that runs unattended.

The problem

Say you are a machine learning engineer who has spent weeks perfecting a model in a Jupyter notebook. You have tuned the hyperparameters, selected the best features, and achieved impressive accuracy on your training data. However, when you try to deploy the model to a production environment, you realize that the model is not reproducible. The accuracy drops significantly, and you are unable to understand why. You have tried to recreate the exact same environment, but it seems like the model is sensitive to some unknown factor. This has cost you several days of debugging, and you are starting to worry that you will miss your deployment deadline.

You are not alone in this struggle. Many machine learning engineers face the same issue when trying to deploy their models to production. The problem lies in the fact that Jupyter notebooks are designed for exploration, not execution. They hide the complexity of state, time, and data distribution, making it difficult to reproduce the exact same conditions in a production environment.

In your case, the symptom is a significant drop in accuracy when deploying the model to production. The metric that has moved is the model's accuracy, which has decreased from 99% in the notebook to 80% in production. This has cost you time, as you have had to spend several days debugging the issue, and it has also cost you sleep, as you are worried about missing your deployment deadline.

Before you start

To follow along with this chapter, you will need to have Python 3.8 or later installed, as well as the pip package manager. You should also have a basic understanding of machine learning concepts and experience with Jupyter notebooks. If you have not already, install the required packages by running the following command:

pip install scikit-learn pandas numpy

Verify that the installation was successful by running:

import sklearn import pandas as pd import numpy as np

If you do not see any errors, you are ready to proceed.

Creating a reproducible training pipeline

To create a reproducible training pipeline, we need to address the issues that make Jupyter notebooks non-reproducible. One of the main issues is the lack of control over the environment. To mitigate this, we can use a tool like conda to create a reproducible environment.

Step 1: Create a conda environment

Create a new conda environment using the following command:

conda create --name myenv python=3.8

This will create a new environment named myenv with Python 3.8 installed.

Step 2: Activate the environment

Activate the environment using the following command:

conda activate myenv

This will activate the myenv environment, and you should see the name of the environment printed on your command line.

Step 3: Install required packages

Install the required packages using the following command:

conda install scikit-learn pandas numpy

This will install the required packages, including scikit-learn, pandas, and numpy.

Step 4: Create a reproducible training script

Create a new Python script called train.py with the following content:

import pandas as pd from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split # Load data df = pd.read_csv('data.csv') # Split data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(df.drop('target', axis=1), df['target'], test_size=0.2, random_state=42) # Train model model = RandomForestClassifier(n_estimators=100, random_state=42) model.fit(X_train, y_train) # Evaluate model accuracy = model.score(X_test, y_test) print(f'Accuracy: {accuracy:.3f}')

This script loads the data, splits it into training and testing sets, trains a random forest classifier, and evaluates its accuracy.

Build it

To build the reproducible training pipeline, follow these steps:

Create a new directory for your project and navigate to it in your terminal.
Create a new file called environment.yml with the following content:

name: myenv dependencies: - python=3.8 - scikit-learn - pandas - numpy

This file defines the conda environment required for your project.

Create a new file called train.py with the content from Step 4 above.
Run the following command to create the conda environment and install the required packages:

conda env create -f environment.yml

Activate the environment using the following command:

conda activate myenv

Run the training script using the following command:

python train.py

This will train the model and print its accuracy to the console.

When this breaks

One common issue that can occur when building a reproducible training pipeline is that the environment is not properly activated. If you see an error message indicating that a package is not installed, it may be because the environment is not activated. To fix this, simply activate the environment using the conda activate command.

Another issue that can occur is that the data is not properly loaded. If you see an error message indicating that the data file is not found, it may be because the file path is incorrect. To fix this, simply update the file path in the train.py script to point to the correct location of the data file.

What it costs

The approach outlined in this chapter has several costs associated with it. One of the main costs is the time required to set up the conda environment and install the required packages. This can take several minutes, depending on the speed of your internet connection and the size of the packages.

Another cost is the disk space required to store the environment and the data. This can be significant, depending on the size of the data and the number of packages installed.

Finally, there is a cost associated with maintaining the environment over time. As new packages are released, you will need to update the environment to ensure that you have the latest versions. This can be time-consuming, especially if you have multiple environments to maintain.

In the interview

In an interview, you may be asked to explain how you would create a reproducible training pipeline. A weak answer might focus on the technical details of the pipeline, without considering the broader context of the project. For example, a candidate might say something like:

"I would use conda to create a reproducible environment, and then install the required packages using pip."

A stronger answer would consider the trade-offs involved in creating a reproducible pipeline, and explain how the approach would be tailored to the specific needs of the project. For example:

"I would use conda to create a reproducible environment, but I would also consider the cost of setting up and maintaining the environment over time. I would weigh the benefits of using a cloud-based environment against the costs of setting up and maintaining a local environment. I would also consider the trade-offs involved in using a pre-built environment versus building a custom environment from scratch."

As a senior engineer, you might be expected to explain how you would scale the approach to a large team, and how you would ensure that the pipeline is maintainable and efficient over time. For example:

"I would use a combination of conda and Docker to create a reproducible environment that can be easily scaled to a large team. I would also use a continuous integration pipeline to automate the testing and deployment of the model, and ensure that the environment is properly maintained and updated over time."

Your tasks

Create a new conda environment and install the required packages for a machine learning project.
Write a Python script that trains a machine learning model and evaluates its accuracy.
Use the script to train a model and evaluate its accuracy, and then modify the script to use a different algorithm and evaluate its accuracy.
Create a new directory for a machine learning project and navigate to it in your terminal. Create a new file called environment.yml that defines the conda environment required for the project, and then create a new file called train.py that trains a machine learning model and evaluates its accuracy.
Use the conda env create command to create the conda environment and install the required packages, and then use the python command to run the training script and evaluate the accuracy of the model.

Your tasks this week

Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.

Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.