LLM Cost Optimization: How to Reduce Your AI Spend by 80%

TL;DR: The 80% Savings Playbook

You can cut LLM costs by 80% without sacrificing quality. The trick is matching model capability to task complexity.

Five strategies that actually work:

1. Model right-sizing: Use mini/nano tiers for simple tasks. Savings: 70-95%
2. Prompt caching: Cache static system prompts and few-shot examples. Savings: 90% on cached tokens
3. Batch processing: Use batch APIs for anything that doesn't need real-time responses. Savings: 50% off the top
4. Model routing: Send requests to the right-sized model based on complexity. Savings: 60-80%
5. Fine-tuning at scale: Shorten prompts through task-specific fine-tuning. Savings: 50%+ after payback

These compound. Stack three or four of them and you hit 80% total savings for most production workloads. The calculator at the end shows your specific numbers.

Read time: 12 minutes

Why LLM Costs Get Out of Hand

Every successful AI application follows the same trajectory: costs start small during development, then explode as users show up. A prototype costing $50 per month becomes a $5,000 monthly bill at scale.

Three things drive this:

Output tokens cost way more than input tokens. Most providers charge 2-10x more for generated tokens than for prompt tokens. Open-ended responses drain budgets fast.

Teams stick with flagship models. Starting with GPT-5.2 or Claude Opus during development makes sense. Keeping that default in production means paying premium prices for tasks that smaller models handle just as well.

Repetitive prompts burn money. System prompts, few-shot examples, and context documents get reprocessed on every request. Without caching, you pay full price for the same tokens over and over.

A customer support bot processing 100,000 requests per month could cost $500 or $5,000. The difference comes down to how you build it.

Most teams leave 60-80% of potential savings on the table. The strategies here address each cost driver.

Model Selection: The Biggest Lever

Model selection is the single biggest cost lever you have. Savings over 90% are possible without quality loss for most production workloads.

Think of models in three tiers: flagship models (GPT-5.2, Claude Opus 4.5) offer maximum capability at premium prices; mid-tier models (GPT-5-mini, Claude Sonnet 4.5) handle most complex tasks at a fraction of the cost; lightweight models (GPT-5-nano, Claude Haiku) excel at simple, well-defined tasks for pennies per thousand requests.

The pricing gap is substantial. GPT-5.2 charges $1.75 per million input tokens. GPT-5-nano charges $0.05. That's a 97% reduction. Output tokens show similar gaps.

Here's the thing: over 70% of enterprise use cases perform identically on smaller models. Simple classification, content extraction, sentiment analysis, and structured data tasks rarely benefit from flagship capabilities. You're paying for reasoning power you never use.

Test before assuming you need flagship models. Create an evaluation set of 100-200 representative requests. Run them through different model tiers. Measure quality metrics that matter for your application. The results often surprise teams.

Savings calculated vs GPT-5.2/Claude Opus 4.5 at standard pricing

Your specific results may vary based on task complexity and quality requirements.

LLM API Pricing Comparison (2026)

Rates vary significantly across providers, and discount mechanisms change the economics.

OpenAI Pricing

OpenAI offers the broadest model range, from flagship reasoning models to ultra-low-cost nano tiers. Their pricing rewards caching (90% discount on cached input) and batch processing (50% discount).

Batch API: 50% discount on all models

GPT-5-mini and GPT-5-nano (bold) represent best value for most use cases

Anthropic Claude Pricing

Claude follows a similar tiering strategy with particularly strong prompt caching: 90% discount on cached reads with TTL options up to one hour.

Key Discounts:

• Cached read: 90% discount vs standard input pricing
• Batch API: 50% discount on most models

Prompt caching provides 90% savings on repeated content

Open-Source Providers

Alternative inference providers offer 3-10x cost savings on open-source models with comparable quality. Groq is fast (500-1000 tokens per second through their LPU architecture). Together AI and Fireworks AI offer broader model selection with Llama 4, DeepSeek, and Qwen.

All providers offer Batch API with 50% discount

Groq models (bold) highlighted for exceptional speed - 500-1,000 tokens per second

All major providers offer 50% batch API discounts. Combined with caching, batch-eligible workloads hit 70%+ total savings.

Prompt Caching: 90% Savings on Repeat Content

Prompt caching stores static prompt portions so you don't reprocess identical tokens on every request. When you send the same system prompt with every API call, caching means you only pay full price once.

Three types of content benefit from caching:

System prompts define your AI's behavior, personality, and constraints. These rarely change between requests and should always be cached.

Few-shot examples demonstrate desired output format and quality. If you reuse the same examples across requests, caching eliminates redundant processing costs.

Document context for RAG applications often includes repeated reference material. Structuring prompts to place static context first enables caching even with dynamic queries.

Provider implementations differ. OpenAI and Anthropic automatically detect and cache repeated prompt prefixes after identifying the pattern. Cache activation typically requires 2-3 identical requests. Groq and Fireworks require explicit cache configuration.

Here's the cost impact for a typical support application: 10,000 daily requests with a 2,000-token system prompt. Without caching, those system prompt tokens cost $60 per day at standard rates. With 90% caching discount, the same tokens cost $6 per day. That's $54 in daily savings from a single change.

"""
Prompt Caching Implementation
Demonstrate caching-optimized prompts for cost reduction
"""

# Prompt caching optimization pattern
# Place static content at the beginning for automatic caching

import openai

# Define static system prompt (cached after first request)
# This 2,000 token prompt will be cached after 2-3 identical requests
SYSTEM_PROMPT = """You are a customer support assistant for Acme Corp.

## Guidelines
1. Be helpful and concise
2. Reference our knowledge base when possible
3. Escalate complex issues to human support
4. Always verify customer identity before sharing account details

## Product Catalog
- Widget Pro: $99/month, enterprise features, unlimited users
- Widget Basic: $29/month, essential features, up to 5 users
- Widget API: Usage-based pricing, $0.001 per request

## Common Policies
- Refunds: Full refund within 30 days, prorated after
- Support hours: 24/7 for Pro, business hours for Basic
- Data retention: 90 days after account closure

## Response Format
- Keep responses under 200 words
- Use bullet points for lists
- Include relevant documentation links
"""


def get_response(user_message: str) -> str:
    """
    Static content at beginning enables prompt caching.

    OpenAI automatically caches after ~2 identical requests.
    Anthropic: Use cache_control for explicit caching.

    Args:
        user_message: The customer's question or request

    Returns:
        The assistant's response
    """
    response = openai.chat.completions.create(
        model="gpt-5-mini",
        messages=[
            # Static system prompt - will be cached
            {"role": "system", "content": SYSTEM_PROMPT},
            # Dynamic user message - not cached
            {"role": "user", "content": user_message},
        ],
        max_tokens=500,  # Limit output tokens for cost control
    )
    return response.choices[0].message.content


def get_response_anthropic(user_message: str) -> str:
    """
    Anthropic implementation with explicit cache control.

    Use cache_control blocks for fine-grained caching.
    """
    import anthropic

    client = anthropic.Anthropic()

    response = client.messages.create(
        model="claude-sonnet-4-5-20250514",
        max_tokens=500,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"},  # Explicit caching
            }
        ],
        messages=[{"role": "user", "content": user_message}],
    )
    return response.content[0].text


if __name__ == "__main__":
    # Example usage
    questions = [
        "What's your refund policy?",
        "How much does Widget Pro cost?",
        "Can I upgrade from Basic to Pro?",
    ]

    for question in questions:
        response = get_response(question)
        print(f"Q: {question}")
        print(f"A: {response[:100]}...")
        print()

# Cost Impact Analysis
# --------------------
# System prompt: ~2,000 tokens
# User message: ~200 tokens average
#
# Without caching (per request):
#   Input: 2,200 tokens x $0.00025/1K = $0.00055
#
# With caching (per request, after initial):
#   Cached: 2,000 tokens x $0.000025/1K = $0.00005
#   New: 200 tokens x $0.00025/1K = $0.00005
#   Total: $0.0001
#
# Savings: 82% on input tokens
#
# At 100,000 requests/month:
#   Without caching: $55/month (input only)
#   With caching: $10/month (input only)
#   Monthly savings: $45

Best practices for prompt caching:

• Place all static content at the beginning of your prompt
• Keep system prompts consistent across requests (avoid dynamic timestamps or request IDs in cached sections)
• Monitor cache hit rates through provider dashboards
• Combine caching with batch processing for compounded savings
• Note minimum cache TTL: OpenAI holds cache for 5-10 minutes, Anthropic offers extended TTL up to 1 hour

Batch Processing: 50% Off for Async Workloads

Batch processing submits multiple requests for processing within a 24-hour window at a 50% discount. This is often the fastest path to significant savings with minimal code changes.

The trade-off is simple: you give up real-time responses in exchange for half-price inference. For many workloads, this is an easy call.

Good for batch:

• Document processing pipelines (summarization, extraction, classification)
• Bulk content generation (product descriptions, email templates)
• Data enrichment and labeling
• Model evaluation and testing
• Report generation and analytics

Bad for batch:

• Real-time chat applications
• Time-sensitive notifications
• Interactive user experiences
• Anything requiring sub-minute response times

Implementation requires minimal changes. Most providers offer a batch endpoint that accepts the same request format as their standard API. You upload requests as a JSONL file, submit the batch job, and retrieve results when processing completes.

"""
Batch API Usage Example
Demonstrate batch API for 50% cost savings
"""

# Batch API for 50% cost savings on async workloads
# Trade-off: 24-hour processing window instead of real-time

import openai
import json
import time
from pathlib import Path


def create_batch_job(
    requests: list[dict], output_file: str = "batch_input.jsonl"
) -> str:
    """
    Submit requests for batch processing at 50% discount.

    Args:
        requests: List of request dictionaries with 'messages' key
        output_file: Path for the JSONL file

    Returns:
        Batch job ID for status tracking
    """
    # Format requests for batch API
    batch_requests = []
    for i, req in enumerate(requests):
        batch_requests.append(
            {
                "custom_id": f"request-{i}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {
                    "model": "gpt-5-mini",
                    "messages": req["messages"],
                    "max_tokens": 500,
                },
            }
        )

    # Write to JSONL file (one JSON object per line)
    with open(output_file, "w") as f:
        for req in batch_requests:
            f.write(json.dumps(req) + "\n")

    # Upload file to OpenAI
    with open(output_file, "rb") as f:
        batch_file = openai.files.create(file=f, purpose="batch")

    # Create batch job
    batch_job = openai.batches.create(
        input_file_id=batch_file.id,
        endpoint="/v1/chat/completions",
        completion_window="24h",
    )

    print(f"Batch job created: {batch_job.id}")
    print(f"Status: {batch_job.status}")

    return batch_job.id


def check_batch_status(batch_id: str) -> dict:
    """Check the status of a batch job."""
    batch = openai.batches.retrieve(batch_id)
    return {
        "id": batch.id,
        "status": batch.status,
        "created_at": batch.created_at,
        "completed_at": batch.completed_at,
        "request_counts": batch.request_counts,
        "output_file_id": batch.output_file_id,
    }


def retrieve_batch_results(batch_id: str) -> list[dict]:
    """
    Retrieve results from a completed batch job.

    Args:
        batch_id: The batch job ID

    Returns:
        List of response dictionaries
    """
    batch = openai.batches.retrieve(batch_id)

    if batch.status != "completed":
        raise ValueError(f"Batch not completed. Status: {batch.status}")

    # Download results
    result_file = openai.files.content(batch.output_file_id)

    results = []
    for line in result_file.text.strip().split("\n"):
        results.append(json.loads(line))

    return results


# Example: Batch document summarization
if __name__ == "__main__":
    # Sample documents to process
    documents = [
        "The quarterly report shows a 15% increase in revenue...",
        "Customer feedback indicates high satisfaction with...",
        "Technical analysis reveals potential improvements in...",
        "Market trends suggest growing demand for AI solutions...",
        "Competitive analysis shows our pricing is competitive...",
    ]

    # Create batch requests
    requests = [
        {
            "messages": [
                {"role": "system", "content": "Summarize in 2-3 sentences."},
                {"role": "user", "content": f"Summarize: {doc}"},
            ]
        }
        for doc in documents
    ]

    # Submit batch job
    batch_id = create_batch_job(requests)

    # Poll for completion (in production, use webhooks)
    print("\nWaiting for batch completion...")
    while True:
        status = check_batch_status(batch_id)
        print(f"Status: {status['status']}")

        if status["status"] == "completed":
            break
        elif status["status"] in ["failed", "expired", "cancelled"]:
            print(f"Batch failed: {status}")
            break

        time.sleep(60)  # Check every minute

    # Retrieve results
    if status["status"] == "completed":
        results = retrieve_batch_results(batch_id)
        for result in results:
            print(f"\n{result['custom_id']}:")
            print(result["response"]["body"]["choices"][0]["message"]["content"])


# Cost Comparison
# ---------------
# Processing 10,000 document summaries
# Average: 1,000 input tokens + 200 output tokens per request
#
# Standard API (GPT-5-mini):
#   Input: 10M tokens x $0.25/1M = $2.50
#   Output: 2M tokens x $2.00/1M = $4.00
#   Total: $6.50
#
# Batch API (50% discount):
#   Input: 10M tokens x $0.125/1M = $1.25
#   Output: 2M tokens x $1.00/1M = $2.00
#   Total: $3.25
#
# Savings: $3.25 (50%)
#
# At 100,000 requests/month: Save $32.50/month
# At 1,000,000 requests/month: Save $325/month


# Best Use Cases for Batch API
# ----------------------------
# - Document processing pipelines
# - Bulk content generation
# - Data extraction and classification
# - Model evaluation and testing
# - Report generation
# - Nightly analytics jobs
#
# NOT suitable for:
# - Real-time chat
# - Time-sensitive notifications
# - Interactive user experiences

The 50% discount applies across providers. Combined with prompt caching, batch-eligible workloads can hit 70% or better total cost reduction. If any part of your workload can tolerate 24-hour latency, batch processing should be your first move.

Model Routing: Route by Complexity

Model routing uses a cheap classifier to analyze incoming requests and send them to appropriately-sized models. Simple questions go to nano models. Complex reasoning tasks go to flagship models. Most requests land somewhere in between.

Not every request deserves the same resources. A user asking "What are your business hours?" doesn't need GPT-5.2. A user asking for a detailed technical analysis does.

Here's how it works: Every request first passes through a lightweight classifier (GPT-5-nano or Claude Haiku). This classifier determines task complexity based on query characteristics like length, vocabulary, and intent signals. The request then routes to the appropriate model tier based on the classification result.

A typical distribution: 70% of requests are simple enough for nano models. 25% require mid-tier capabilities. Only 5% genuinely need flagship reasoning power.

The cost impact: If you currently run everything through a flagship model at $15 per million tokens, model routing reduces your effective rate to about $3 per million. The classifier adds negligible cost (under 1% of total spend).

Implementation approaches:

Start with rule-based routing before building ML classifiers. Simple heuristics work well: Short queries under 50 tokens go to nano. Queries containing reasoning keywords ("analyze", "compare", "explain why") route to higher tiers. Structured extraction tasks always use the cheapest capable model.

As you gather data, train a proper classifier on your routing decisions. Monitor quality metrics by tier to tune thresholds. Some teams find 80% of requests can safely use nano models. Others need a more conservative split.

The architecture works across providers. You can route between OpenAI models, across providers, or to self-hosted open-source models. The principle stays the same: pay for capability only when you need it.

Semantic Caching: Cache Similar Requests

Semantic caching extends traditional caching by returning cached responses for semantically similar requests, not just identical ones. If a user asks "What is your return policy?" and another asks "How do I return an item?", semantic caching recognizes these as effectively the same question.

The mechanism uses embeddings to represent queries as vectors. When a new request arrives, generate its embedding and search your vector database for similar cached queries. If similarity exceeds your threshold (typically 0.95 or higher), return the cached response without calling the LLM.

Typical implementations achieve 20-40% cache hit rates on support, FAQ, and documentation workloads. That translates to 15-35% cost reduction on top of other optimizations.

Implementation components:

1. Embedding model: OpenAI's text-embedding-3-small ($0.02/1M tokens) or open-source alternatives like sentence-transformers
2. Vector database: Pinecone (managed), Weaviate, Qdrant, or Chroma (self-hosted options)
3. Cache layer: Redis or your existing cache infrastructure for response storage

Key considerations:

• Threshold tuning matters. Too low (0.90) returns irrelevant cached responses. Too high (0.99) defeats the purpose. Start at 0.95 and adjust based on quality monitoring.
• Cache invalidation matters. Stale responses to policy questions frustrate users. Set TTL based on content change frequency.
• Semantic caching complements prompt caching. They address different redundancy types and can be combined.
• Best suited for high-volume, repetitive query workloads. Unique creative requests see minimal benefit.

Fine-Tuning ROI: When Does It Pay Off?

Fine-tuning is an investment decision, not just a technical one. The upfront cost only makes sense when ongoing savings exceed that investment within a reasonable timeframe.

Fine-tuning pays off under specific conditions: high request volume (over 100,000 requests per month), consistent task type, and significant prompt reduction potential. If your application meets these criteria, fine-tuning can deliver 50% or greater ongoing savings after a brief payback period.

Consider this calculation:

Before fine-tuning: Your application uses 2,000 tokens per request (including system prompt and few-shot examples). At 1 million requests per month with GPT-5-mini pricing, you spend $3,000 monthly.

After fine-tuning: The fine-tuned model requires only 500 tokens per request because it has learned your task without needing examples. Training costs $500 as a one-time expense. Monthly inference drops to $1,500.

Payback: The $500 training investment pays back in less than one month. Every subsequent month saves $1,500.

Break-Even Analysis Example

Scenario: 1M requests/month, 2,000 tokens/request with few-shot examples

Fine-tuning ROI requires >100K requests/month to justify investment

When fine-tuning doesn't make sense:

• Low volume (under 50,000 requests per month): Savings don't justify training investment
• Changing requirements: Retraining for every change eliminates savings
• No prompt reduction opportunity: Fine-tuning for quality alone rarely justifies cost
• Anthropic users: Claude doesn't offer a fine-tuning API

Fine-tuning requires labeled training data and an evaluation pipeline to measure quality. Plan for 2-4 weeks of preparation before the actual training. Monitor fine-tuned model performance over time, as drift can occur.

Self-Hosting: When to Run Your Own Infrastructure

Self-hosting LLMs makes economic sense only under specific conditions. For most teams, API providers remain more cost-effective even at significant scale.

The prerequisites for self-hosting:

1. Volume over 50 million tokens per month with consistent demand. Spiky workloads can't efficiently use dedicated infrastructure.
2. Dedicated ML engineering resources to manage deployment, monitoring, and updates. This isn't set-and-forget.
3. Predictable workloads that justify reserved capacity. APIs handle demand spikes better.
4. Data residency or compliance requirements that preclude third-party API usage.

If you don't meet all four criteria, APIs almost certainly cost less when you account for engineering time.

[Image: Self-Hosting vs API Cost Break-Even Analysis]

The break-even point varies by configuration. Against OpenAI GPT-4o pricing, self-hosted Llama 70B on H100 infrastructure breaks even around 40-50 million tokens per month. Against already-cheap providers like Together AI, break-even pushes past 80-100 million tokens monthly.

Hidden costs to factor:

• Engineering time at fully-loaded cost ($150-250/hour)
• GPU procurement lead times (weeks to months for H100s)
• Monitoring, alerting, and on-call rotation
• Model updates and security patches
• Scaling infrastructure for demand changes

For teams that do cross the break-even threshold, vLLM provides the best starting point. Its PagedAttention implementation delivers 2-4x throughput improvement over baseline serving. Continuous batching and tensor parallelism enable efficient use of expensive GPU resources.

Infrastructure recommendations by model size:

For 7-8B parameter models, a single A100 40GB provides comfortable headroom. For 70B parameter models in FP16, you need at least 2x A100 80GB GPUs with tensor parallelism. Quantization to INT8 allows 70B models to fit on a single A100 80GB with minimal quality impact. For larger models like Llama 405B, plan for 8+ H100 GPUs in a multi-node configuration.

Implementation Priority: Quick Wins to Long-Term

Not all optimizations require equal effort. Prioritize by implementation complexity and time to value.

Quick Wins (Implement This Week)

These require minimal code changes and deliver immediate savings of 30-50%:

• Enable batch API for non-real-time workloads (50% instant savings)
• Enable prompt caching by restructuring prompts with static content first (30-90% on cached tokens)
• Set max_tokens limits to prevent runaway output costs (variable savings)
• Switch to mini/nano models for simple tasks after quick evaluation (70-95% savings)

Medium Effort (Implement This Month)

These require architecture changes and deliver additional 20-40% savings:

• Implement model routing with rule-based classification initially
• Add semantic caching with vector database integration
• Use structured outputs (JSON mode) to reduce parsing costs and token waste

Long-Term Investment (Plan This Quarter)

These require organizational commitment and deliver additional 10-30% savings:

• Fine-tune models for high-volume consistent tasks
• Evaluate self-hosting if volume exceeds 50M tokens monthly
• Negotiate volume discounts directly with providers for enterprise usage

Combined potential savings: 60-85%

Start with Quick Wins. A single afternoon can cut your bill in half

Start with quick wins. A single afternoon implementing batch processing and caching can cut your bill in half. Don't over-engineer with model routing or semantic caching until you've captured the easy savings.

The combined potential of all strategies reaches 60-85% total savings. Most teams achieve 50% reduction within the first week by focusing on quick wins alone. The remaining optimization layers add incremental savings over the following months.

Calculate Your Savings

Theory only goes so far. Use our calculator to see exactly how much you could save based on your actual usage.

Enter your current provider, model, monthly request volume, and average token counts. The calculator shows your current costs and projected savings from each strategy. You can also specify your workload type (real-time vs async) and system prompt reuse percentage to get accurate caching estimates.

[Interactive Cost Calculator - Frontend Component]

The calculator uses current 2026 pricing data and shows compound savings correctly. Combining multiple strategies doesn't simply add percentages. It calculates the true combined effect based on how each strategy applies to your specific workload.

Results include a prioritized list of recommended actions based on your inputs. High-volume async workloads see different recommendations than low-volume real-time applications.

See exactly how much you would save with Inference.net's infrastructure. [Calculate Your Savings]

Frequently Asked Questions

How much can I realistically save on LLM costs?

Savings of 70-95% are achievable by right-sizing model selection to task complexity. Combining multiple strategies (caching, batching, routing) gets you to 80% or more for most production workloads. The specific number depends on your current configuration and workload characteristics.

Is prompt caching automatic?

OpenAI and Anthropic automatically cache prompts after detecting repeated patterns (typically 2-3 identical requests). Groq and Fireworks require explicit cache configuration. Minimum cache TTL ranges from 5 minutes (OpenAI) to 1 hour (Anthropic extended), depending on provider and plan tier.

When does self-hosting make sense?

At volumes exceeding 50 million tokens per month with dedicated ML engineering resources, predictable workloads, and data residency requirements. All four conditions should be met. Below this threshold, API providers remain more cost-effective when you account for engineering time.

Does using smaller models hurt quality?

For over 70% of enterprise use cases, smaller models perform identically on task-specific metrics. Classification, extraction, and structured data tasks rarely benefit from flagship capabilities. Test on a representative evaluation set of 100-200 requests before deciding.

What's the fastest way to cut costs by 50%?

Enable batch API for non-real-time workloads. Instant 50% discount with minimal code changes (typically one endpoint modification). Works with OpenAI, Anthropic, Together AI, and most providers. Combine with caching for 70%+ total reduction.

How do I estimate costs before deploying?

Count tokens in representative sample requests using tiktoken (OpenAI) or the Anthropic tokenizer (Claude). Multiply by expected monthly volume. Add 20% buffer for variance. Factor in the input-to-output ratio since output tokens cost 2-10x more.

Conclusion: Start Saving Today

Your action plan for this week:

1. Day 1: Enable batch API for any async workloads (instant 50% on those requests)
2. Day 2: Audit your prompts and restructure for caching (move static content to the front)
3. Day 3-4: Run an evaluation comparing your current model against mini/nano alternatives
4. Day 5: Implement the changes with the best ROI from your evaluation

Most teams cut their bill in half within the first week.

Measure before and after every change. Track spending by model, endpoint, and use case using LiteLLM, Helicone, or provider dashboards. Data drives continued improvement and prevents regressions.

Build cost monitoring into your operations. Set spending alerts. Review your setup quarterly as pricing changes and new models release.

The 80% savings target isn't aspirational. It's what happens when you match model capability to task complexity, eliminate redundant processing through caching, and use batch APIs for latency-tolerant workloads.

Calculate your exact savings with Inference.net and see what this looks like in practice. [Get Started]