"I spent three days manually adjusting my model's learning rate and number of layers, and the result was still far from ideal. If this has happened to you, the solution lies in automatic hyperparameter optimization with Hyperopt." — A common report in machine learning forums, such as Stack Overflow (2025).

In 2026, manual hyperparameter search is one of the biggest bottlenecks in machine learning projects. With increasingly complex models — from transformers to deep neural networks — finding the ideal combination of parameters like learning rate, number of layers, or batch size can consume weeks of work. This is where Hyperopt comes in, an automatic optimization library that uses algorithms like TPE (Tree-structured Parzen Estimator) to intelligently explore the search space.

Unlike traditional methods like Grid Search or Random Search, Hyperopt learns from previous attempts and focuses on the most promising regions. In recent benchmarks, such as the study "Hyperopt: A Python Library for Model Selection and Hyperparameter Optimization" (Bergstra et al., 2013, available at ResearchGate), Hyperopt demonstrated a reduction in tuning time of up to 60% compared to Grid Search, while maintaining the same final quality. Companies like Spotify and Yelp use Hyperopt in production pipelines, reporting significant productivity gains.

In this tutorial, you will build a complete hyperparameter optimization pipeline for an image classification model using PyTorch and Hyperopt. By the end, you will have an optimized model and an interactive dashboard with the search results.

Why Hyperopt is a Relevant Tool in 2026

Hyperparameter optimization has always been a challenge. Methods like Grid Search test all possible combinations, which is unfeasible for large search spaces. Random Search is more efficient but still wastes resources on poor regions. Hyperopt solves this with three main features:

• Intelligent sampling: Uses TPE to model performance distribution and suggest new search points.
• Support for complex spaces: Allows defining search spaces with continuous, discrete, and conditional variables.
• Simple integration: Works with any framework (PyTorch, TensorFlow, scikit-learn) and offers built-in visualizations.

In 2026, Hyperopt is actively maintained by the community, with over 7,000 stars on GitHub and support for distributed tuning. It is the default choice in companies like Spotify, which uses it to optimize music recommendation models, and Yelp, for local search systems.

Step-by-Step: From Setup to Complete Optimization

1. Environment Setup

You will need Python 3.10+ and the libraries below. Install the dependencies:

pip install hyperopt torch torchvision pandas matplotlib

Hyperopt works with pure Python, but for this tutorial, we will use PyTorch for the classification model. Make sure you have a GPU available to accelerate training, although the example works on CPU.

2. Defining the Model and Search Space

We will create a simple convolutional neural network (CNN) model to classify images from the CIFAR-10 dataset. The search space will include hyperparameters such as learning rate, number of convolutional layers, and dropout.

import hyperopt
from hyperopt import hp, fmin, tpe, Trials, STATUS_OK
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

Define the search space

search_space = { "n_layers": hp.choice("n_layers", [2, 3, 4, 5]), "dropout": hp.uniform("dropout", 0.1, 0.5), "learning_rate": hp.loguniform("learning_rate", -11.5, -4.6), # log scale: 1e-5 to 1e-2 "out_channels_0": hp.choice("out_channels_0", [32, 64, 128]), "out_channels_1": hp.choice("out_channels_1", [32, 64, 128]), "out_channels_2": hp.choice("out_channels_2", [32, 64, 128]), "out_channels_3": hp.choice("out_channels_3", [32, 64, 128]), "out_channels_4": hp.choice("out_channels_4", [32, 64, 128]) }

Define the model with variable hyperparameters

def create_model(params): n_layers = params["n_layers"] dropout = params["dropout"] learning_rate = params["learning_rate"]

layers = []
in_channels = 3
for i in range(n_layers):
    out_channels = params[f"out_channels_{i}"]
    layers.append(nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1))
    layers.append(nn.ReLU())
    layers.append(nn.MaxPool2d(2))
    in_channels = out_channels

layers.append(nn.AdaptiveAvgPool2d((1, 1)))
layers.append(nn.Flatten())
layers.append(nn.Linear(in_channels, 10))
layers.append(nn.Dropout(dropout))

model = nn.Sequential(*layers)
return model, learning_rate

The hp module from Hyperopt defines the search space. hp.choice for discrete options, hp.uniform for continuous values, and hp.loguniform for values on a logarithmic scale.

3. Objective Function for Hyperopt

The objective function trains the model with the suggested hyperparameters and returns the accuracy on the validation set. Hyperopt minimizes the returned value, so we use 1 - accuracy as the loss.

def objective(params):
    # Load CIFAR-10 data
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])
    train_dataset = datasets.CIFAR10(root="./data", train=True, download=True, transform=transform)
    val_dataset = datasets.CIFAR10(root="./data", train=False, download=True, transform=transform)
    train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
    val_loader = DataLoader(val_dataset, batch_size=64, shuffle=False)
    
    # Create model and optimizer
    model, learning_rate = create_model(params)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=learning_rate)
    
    # Training
    for epoch in range(10):
        model.train()
        for images, labels in train_loader:
            optimizer.zero_grad()
            output = model(images)
            loss = criterion(output, labels)
            loss.backward()
            optimizer.step()
        
        # Validation
        model.eval()
        correct = 0
        total = 0
        with torch.no_grad():
            for images, labels in val_loader:
                output = model(images)
                _, predicted = torch.max(output, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()
        
        accuracy = correct / total
    
    # Hyperopt minimizes, so we return 1 - accuracy
    loss = 1 - accuracy
    return {"loss": loss, "status": STATUS_OK, "accuracy": accuracy}

Hyperopt does not have native pruning like Optuna, but we can implement simple early stopping logic if accuracy is too low in the first few epochs.

4. Running the Optimization

Now, we create a Trials object to store the results and run the search with the TPE algorithm.

# Create Trials object to store results
trials = Trials()

Run 50 trials with TPE

best = fmin( fn=objective, space=search_space, algo=tpe.suggest, max_evals=50, trials=trials, rstate=np.random.default_rng(42) )

Best hyperparameters (converting indices to actual values)

print("Best hyperparameters (indices):", best)

To convert: hp.choice returns indices, so we need to map them

n_layers = [2, 3, 4, 5][best["n_layers"]] dropout = best["dropout"] learning_rate = np.exp(best["learning_rate"]) # loguniform returns log print(f"n_layers: {n_layers}, dropout: {dropout:.3f}, learning_rate: {learning_rate:.6f}")

Best accuracy

best_trial = trials.best_trial print("Best accuracy:", 1 - best_trial["result"]["loss"])

The fmin function runs the optimization. The TPE (Tree-structured Parzen Estimator) algorithm is the default for Hyperopt, based on the paper "Algorithms for Hyper-Parameter Optimization" (Bergstra et al., 2011, available at NeurIPS Proceedings).

5. Visualizing the Results

Hyperopt does not offer built-in visualizations like Optuna, but we can use pandas and matplotlib to analyze the trials.

import pandas as pd
import matplotlib.pyplot as plt

Convert trials to DataFrame

results = [] for trial in trials.trials: result = trial["result"] params = trial["misc"]["vals"] # Convert indices to values converted_params = {} for key, value in params.items(): if key == "n_layers": converted_params[key] = [2, 3, 4, 5][value[0]] elif key.startswith("out_channels"): converted_params[key] = [32, 64, 128][value[0]] elif key == "learning_rate": converted_params[key] = np.exp(value[0]) else: converted_params[key] = value[0] converted_params["accuracy"] = 1 - result["loss"] results.append(converted_params)

df = pd.DataFrame(results)

Accuracy plot per trial

plt.figure(figsize=(10, 6)) plt.plot(df.index, df["accuracy"], marker="o", linestyle="-", alpha=0.7) plt.xlabel("Trial") plt.ylabel("Accuracy") plt.title("Accuracy Evolution during Hyperopt Optimization") plt.grid(True) plt.show()

This graph shows how accuracy improves over the trials, highlighting the efficiency of TPE in focusing on promising regions.

Advanced Tips for 2026

• Distributed tuning: Use Hyperopt's SparkTrials to distribute the search across a Spark cluster, ideal for large search spaces.
• Conditional spaces: With hp.choice, you can define hyperparameters that are only activated if others are chosen, such as different network architectures.
• Integration with MLflow: Log the trials in MLflow for experiment tracking and comparison between runs.

Conclusion

In this tutorial, you learned how to use Hyperopt to optimize hyperparameters for an image classification model in 2026. With Bayesian search based on TPE, it is possible to reduce tuning time by up to 60% compared to Grid Search, as demonstrated by Bergstra et al. (2013). The tool is mature, well-documented, and widely adopted in the industry.

Now it's your turn: apply Hyperopt to your next machine learning project and see the difference. Share your results in the comments!

Related Articles

• How to Use AI to Create High-Quality Content in 2026
• From Dataset to Ollama: Fine-Tuning LLMs with Unsloth on Your GPU in 2026
• 48% Don't Test, 40% Hallucinate: How to Evaluate LLMs in 2026 — An Analytical Guide