This notebook shows the 66.66% reduction in latency on the credit card fraud dataset from https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud. The method is early-exiting and building a cascaded system of models and this notebook demonstrates that reductions in compute-time are possible while preserving accuracy.

In [15]:
import pandas as pd
from sklearn.neural_network import MLPClassifier
from utils import optimize, HyperplaneDecisionRealTime, Router
from sklearn.model_selection import train_test_split
import time
import os
import numpy as np
from tqdm import tqdm

import warnings
In [9]:
# Suppress only RuntimeWarning
warnings.filterwarnings("ignore", category=RuntimeWarning)

def load_dataset():
    df = pd.read_csv('/Users/samrandall/Downloads/creditcard.csv')
    X = df[[col for col in df.columns if col.startswith('V')]].to_numpy()
    y = df['Class'].to_numpy()
    return X, y

Load the dataset and train a simple MLPClassifier. This will take 10 seconds.

In [10]:
X, y = load_dataset()
X_train, X_test, y_train, y_test = train_test_split(X, y)
model = MLPClassifier()
model.fit(X_train, y_train)
Out[10]:
MLPClassifier()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
MLPClassifier()
In [11]:
json_save_path = "creditcard_hyperplane.json"

if False:
    hd = HyperplaneDecisionRealTime.load(json_save_path)
    router = Router(hd, lambda x: 0, lambda x: model.predict(x).item())
else:
    router = optimize(model, X_train)
    router.decision.save(json_save_path)

Response <Response [200]>
response status code 200
In [12]:
    
for i in range(len(X)):
    model.predict(X[i:i+1])

start = time.time()
for i in tqdm(range(len(X))):
    router.forward(X[i:i+1])
end = time.time()
early_exit_time = end - start

for i in range(len(X)):
    model.predict(X[i:i+1])

start = time.time()
for i in tqdm(range(len(X))):
    model.predict(X[i:i+1])
end = time.time()
original_time = end - start

print("Original Time", original_time) # 13.354 seconds
print("Early Exit Time", early_exit_time) # 10.707 seconds

100%|██████████| 284807/284807 [00:04<00:00, 60876.30it/s]
100%|██████████| 284807/284807 [00:14<00:00, 20286.29it/s]
Original Time 14.041443109512329
Early Exit Time 4.680275917053223

























Counting how many data points avoided full model execution (were exited early)


















In [24]:






decisions = []
for item in X:
    decisions.append(router.decision.forward(item[np.newaxis, :]))
decisions = np.array(decisions)





















In [25]:






decisions.mean()





















Out[25]:




np.float64(0.7104846439869806)
























Latency¶
The results in this run are that the original model took 14.04 seconds to run through the entire dataset of 284807 seconds. Compare this to Early Exit Latency of 4.68 seconds. This is a 66.7% reduction in CPU-time. Results vary based on the run of the algorithm. This is because 71% of the data points avoided model execution.






















Accuracy¶






















This section shows below that the predictions in the routed model are 100% across the test set.


















In [26]:






p = []
for item in X_test:
    p.append(router.forward(item[np.newaxis, :]))

early_exit_predictions = np.array(p)
original_predictions = model.predict(X_test)





















In [27]:






np.mean(early_exit_predictions == original_predictions)





















Out[27]:




np.float64(1.0)




















In [ ]: