Skip to content
STIRUP

TECHNOLOGY INTELLIGENCE

Understand what's next.

AssemblyAI Uses AWS Spot Instances for Cost-Effective Speech-to-Text Inference

AssemblyAI leverages Amazon EC2 Spot Instances and On-Demand Instances across multiple NVIDIA GPU families to achieve cost-optimized AI inference at scale for their speech-to-text platform.

Share
AI-generated conceptual illustration accompanying: AssemblyAI Uses AWS Spot Instances for Cost-Effective Speech-to-Text Inference
AI-generated editorial illustration of the article topic; not a photograph, product screenshot or measured result.

Speech-to-text inference and computational demands

Speech-to-text inference is computationally intensive, requiring significant processing power to convert audio signals into text accurately and efficiently. AssemblyAI addresses this challenge by leveraging a mixed fleet of Amazon EC2 Spot Instances and On-Demand Instances across multiple NVIDIA GPU instance families. This approach allows AssemblyAI to run a single inference model per GPU at a scale of thousands of instances, optimizing both performance and cost. By utilizing Spot Instances, which offer unused EC2 capacity at a lower cost, AssemblyAI achieves cost-optimized AI inference at scale for their speech-to-text platform. This strategy enables efficient resource allocation while maintaining the reliability and performance required for real-world applications.

Architecture and infrastructure components

The architecture and infrastructure components of AssemblyAI’s speech-to-text inference solution on AWS are designed to optimize cost and scalability. The system utilizes Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, and Amazon Simple Queue Service (Amazon SQS) to manage workloads efficiently. An Amazon ECS cluster operates within a virtual private cloud (VPC) across multiple Availability Zones, ensuring high availability and fault tolerance. Each task retrieves audio files from Amazon Simple Storage Service (Amazon S3) and writes the completed transcription to a results store, which AssemblyAI configures to deliver the output to the end customer. The use of Amazon EC2 Spot GPU instances helps reduce computational costs, while the Amazon SQS-based retry mechanism ensures reliability in processing audio files.

Cost optimization through Spot Instances and GPU instance families

Cost optimization through Spot Instances and GPU instance families is a key strategy for reducing inference costs when using AssemblyAI speech-to-text on AWS. According to AWS Machine Learning, Spot Instances provide significant savings, though Amazon EC2 may reclaim them with a 2-minute warning when capacity is needed elsewhere. To mitigate this risk, AssemblyAI leverages multiple GPU instance families, as noted in AWS Machine Learning, which includes g4dn instances powered by NVIDIA T4 GPUs for cost-effective baseline performance. Additionally, AWS Machine Learning highlights that Amazon EC2 uses a price-capacity optimized allocation strategy to provision instances from Spot capacity pools with the highest available capacity at the lowest price, reducing the likelihood of interruptions. This approach helps maintain performance while minimizing costs in a dynamic cloud environment.

Resilience and recovery mechanisms

Resilience and recovery mechanisms are critical when using Amazon EC2 Spot instances for speech-to-text inference on AWS. AssemblyAI employs Amazon ECS capacity providers and Spot diversification to optimize costs while maintaining reliability. Each capacity provider is associated with an Amazon EC2 Auto Scaling group, enabling dynamic resource allocation based on workload demands. To reduce recovery time in case of Spot instance interruptions, AssemblyAI creates custom Amazon Machine Images (AMIs) containing model artifacts and container images in approximately 2 minutes, as noted in their configuration strategy. These AMIs streamline the redeployment process, ensuring minimal downtime and faster resumption of inference tasks. By leveraging these mechanisms, AssemblyAI ensures that its speech-to-text inference workloads remain both cost-effective and resilient to the inherent volatility of Spot instances.

Performance and cost savings outcomes

AssemblyAI achieved 70 percent cost savings on inference compute by using Amazon EC2 Spot Instances with GPUs, according to results from their production deployment. This outcome was realized by leveraging multiple GPU instance families, including g4dn, g5, and g6, which are available in European Regions with competitive Spot pricing. To ensure optimal performance and availability, AssemblyAI recommends evaluating Spot placement scores for accelerated compute and following Amazon EC2 Spot Instances best practices. These strategies help organizations balance cost efficiency with the reliability needed for inference workloads.

Sources and reporting notes

Based on published source material. No independent hands-on testing is claimed.

Read more