Reduce GPU count by 20 - 50%.

How: by reducing the computational footprint of machine learning workloads.

Allocate compute-time to decisions, based on each decision's difficulty. moco determines the difficulty of each decision (where is deeper analysis needed), and compute-time can be allocated based on this measure.

moco outputs a model that is substitutable for the original, enabling instant efficiency gains.

No Retraining. No Accuracy Loss. No New Hardware.

The Economics

Stop Running Full Inference on Easy Predictions

Every prediction a model makes has a cost. The total compute cost to an organization cost scales with the number of decisions that need to be made and the computational difficulty of each decision.

moco analyzes model behavior and automatically routes easy predictions through cheaper decision paths. This increases computational capacity for the difficult cases, resulting in:

  1. Throughput improvement
  2. Latency reduction
  3. Total Cost of Ownership (TCO)-savings (reduces need for servers, and reduces cooling costs)
  4. Power-savings (resulting in longer battery life of devices)

In text classification, moco reduces the cost per inference from $1 / 1M inferences to $0.5 - $0.8 / 1M inferences, saving 20-50% of compute-time related costs.

Built for production-scale inference systems

moco reduces unnecessary inference computation in both real-time and batched machine learning systems operating at production scale.

moco can help businesses that:

  • have massive request volumes
  • have expensive transformer inference bills
  • whose ML decision systems violate service level agreements (SLAs)

Verticals

Content Moderation

Transformer moderation systems processing billions of events per day.

  • Typical models: BERT, RoBERTa, multimodal transformers
  • Typical latency: 5–20ms
  • Typical scale: 100k–750k+ requests/sec
  • Typical annual inference cost: Millions/year
  • 20% lower average FLOPs/inference can save hundreds of thousands to millions per year in GPU infrastructure and compute-time costs.
  • Lower inference load reduces peak GPU utilization, enabling smaller serving fleets and higher throughput on existing hardware.
Search + Tagging

Large-scale indexing and retrieval pipelines operating on billions of assets.

  • Typical models: BERT, CLIP, embedding models
  • Typical latency: 5–50ms
  • Typical scale: Billions of classifications/day
  • Typical annual inference cost: Millions/year
  • 20% lower average FLOPs/inference can save hundreds of thousands annually in indexing and embedding compute costs.
  • Reduced compute per classification enables faster index refreshes, higher indexing throughput, and more models within the same infrastructure budget.
Financial Fraud Detection

Real-Time and Batched Fraud Detection is latency constrained and compute-constrained respectively.

  • Real-Time Fraud Detection Systems: moco unlocks compute budget to apply more analytics to the hard cases, increasing the accuracy of the system, saving false positives and false negatives.
  • Batched Fraud Detection Systems: moco unlocks reduces runs the same inference with less physical hardware, saving money in compute-time and infra.

System Types

Embedded Systems

moco supports embedded inference pipelines where compute and power are constrained.

  • Existing PyTorch models can be transformed into routed hierarchical classifiers.
  • Expensive model execution is conditionally avoided for easy predictions.
  • Reduced average compute-time enables lower-power inference deployments.
  • Compatible with ONNX export and edge deployment workflows.
Battery-Powered Edge Devices / IoT

Edge inference systems benefit from lower power consumption and reduced average computation.

  • Lower average FLOPs/inference extends battery life for always-on systems.
  • Reduced compute demand lowers thermal load on constrained hardware.
  • Smaller inference workloads enable more models to run on-device.
Real-Time Systems

Latency-sensitive systems benefit from avoiding unnecessary expensive computation.

  • Many production systems are bottlenecked by feature loading, graph queries, and deep model execution.
  • moco helps determine when expensive computation can be skipped safely.
  • Reduced infrastructure load enables smaller serving fleets and higher throughput on existing hardware.
Batched Inference Pipelines

Large offline inference jobs directly benefit from lower compute-time per prediction.

  • Reduced average computation increases total system throughput.
  • Lower GPU-hours reduce infrastructure and cloud compute costs.
  • Existing hardware can process larger datasets without scaling infrastructure.

The Science

Coming soon: information on how we think about decisions, classifiers, graphs and data.

Free Compute-Waste Audit

I will analyze historical inference logs, model outputs, or representative datasets to identify unnecessary model computation and estimate potential savings.

The audit evaluates:

  • Potential GPU-hour reduction
  • Inference latency improvements
  • Whether cascaded routing can preserve model accuracy
Book a Free Audit