Together AI Unveils Batch Inference API Updates for 2025

Together AI has announced significant updates to their batch inference API, delivering up to 50% cost reduction for high-volume workloads. The updates include new queueing optimizations, improved throughput scheduling, and expanded model support for batch processing.

Key improvements include asynchronous job submission with webhook callbacks, priority queue tiers allowing latency-cost tradeoffs, and automatic batching that groups small requests for efficiency. The API now supports all major open models including Llama 3.1, Mixtral, and Qwen.

The pricing model shifts from simple per-token to throughput-based tiers. Low-priority batch jobs (24-hour completion window) receive 50% discount. Standard priority (4-hour window) receives 25% discount. High priority maintains real-time pricing with guaranteed capacity.

• Up to 50% cost reduction for high-volume batch workloads
• Asynchronous submission with webhook callbacks
• Priority tiers: low (50% off, 24hr), standard (25% off, 4hr), high (real-time)
• Automatic batching groups small requests for efficiency

The batch API updates make previously cost-prohibitive workloads viable. Bulk content generation, large-scale document processing, and mass personalization become economically feasible at 50% cost reduction. This opens new application categories.

The priority tier system enables sophisticated cost-latency optimization. Production systems can route time-sensitive requests to high priority while deferring background processing to low priority. This granularity wasn't previously available.

For existing batch users, migration to the new API is straightforward but not automatic. The updated pricing only applies to new API endpoints. Continuing with legacy endpoints means continuing with legacy pricing.

• 50% savings makes previously cost-prohibitive workloads viable
• Priority tiers enable sophisticated cost-latency optimization
• Migration required to access new pricing—not automatic
• Opens new application categories like mass personalization

Submit batch jobs using the new /v2/batch endpoint. Include priority tier, webhook URL for completion notification, and your request payload. The API returns a job ID for tracking status. Results arrive via webhook or can be polled.

Optimize for batch efficiency by grouping related requests. The automatic batching helps, but intentional grouping improves further. Submit daily batch jobs rather than hourly where latency permits. Larger batches achieve better throughput.

Implement retry handling for the asynchronous flow. Webhook delivery isn't guaranteed. Poll for completion as fallback. Design your system to handle jobs that complete faster or slower than typical—the completion window is maximum, not guaranteed timing.

• Submit via /v2/batch with priority tier and webhook URL
• Group related requests for better efficiency
• Larger batches with longer windows achieve best pricing
• Implement retry handling—webhooks aren't guaranteed delivery

The 50% savings on low-priority tier creates significant optimization opportunity. Analyze your workloads for latency requirements. Many batch processes don't actually need fast completion—they run overnight because that's when they were scheduled, not because they need results by morning.

Consider hybrid approaches. Critical results on high priority, bulk processing on low priority. The priority separation isn't all-or-nothing. A single application can use multiple priority tiers for different request types.

Monitor your actual completion times, not just the windows. Together AI often completes jobs faster than the window guarantee. Understanding actual patterns helps you choose appropriate priority tiers—you may get low-priority pricing with standard-priority-like completion times.

• Analyze latency requirements—many batch jobs don't need fast completion
• Hybrid priority strategies optimize cost for mixed workloads
• Actual completion often faster than window guarantees
• Monitor patterns to optimize priority tier selection

Together AI's batch improvements signal broader market direction. Inference providers are moving beyond simple per-token pricing toward sophisticated pricing that reflects actual resource utilization. Expect similar developments from competitors.

Batch processing enables AI applications that weren't economically viable. As costs drop further, expect explosion of bulk AI processing use cases—content libraries processed at scale, customer communications personalized en masse, document archives analyzed comprehensively.

The async, priority-based model will likely become industry standard. Real-time inference for interactive use; batch inference for background processing. Systems designed for this separation will have natural cost advantages.

• Market moving toward utilization-based rather than simple per-token pricing
• Falling costs enable new bulk AI processing applications
• Async, priority-based model becoming industry standard
• Systems designed for real-time/batch separation gain cost advantages