Skip to content
STIRUP

TECHNOLOGY INTELLIGENCE

Understand what's next.

GS Caltex Integrates SageMaker Canvas with Pipelines for MLOps Efficiency

GS Caltex streamlines model deployment by integrating SageMaker Canvas with SageMaker Pipelines, enabling automated handoff and reuse of validated assets in production workflows.

Share
AI-generated conceptual illustration accompanying: GS Caltex Integrates SageMaker Canvas with Pipelines for MLOps Efficiency
AI-generated editorial illustration of the article topic; not a photograph, product screenshot or measured result.

Integration of SageMaker Canvas and SageMaker Pipelines

GS Caltex integrates Amazon SageMaker Canvas with SageMaker Pipelines to streamline the transition of models developed by process engineers into production workflows. According to Production MLOps at GS Caltex with Amazon SageMaker Canvas, part 1, this integration enables the automation of model handoff while reusing assets created in SageMaker Canvas. The process involves connecting validated preprocessing flows and models directly into SageMaker Pipelines, ensuring consistency and reducing manual intervention. This approach allows GS Caltex to maintain alignment between exploratory data analysis and production-ready workflows. The post highlights how this integration supports scalable and repeatable MLOps practices without requiring significant rework of existing models or pipelines. By leveraging these tools, GS Caltex ensures that models developed in SageMaker Canvas are seamlessly incorporated into production environments, enhancing efficiency and reliability in their operations.

Model and Data Flow Management

In the model and data flow management process at GS Caltex, Amazon SageMaker Canvas plays a key role in preparing and managing assets for production. Data preparation involves building a preprocessing flow with SageMaker Data Wrangler, which is then reused in the production workflow through SageMaker Processing jobs. Model development includes training a model and generating evaluation results, which are registered in the SageMaker Model Registry along with versioning and approval status. For prediction, model outputs are validated, and approved models are used for batch inference every 30 minutes. Operations track the execution history, stored results, and failure handling for each step. The reference architecture illustrates how process engineers transfer SageMaker Canvas assets to ML engineers for deployment and scheduled inference. The production pipeline reuses the preprocessing logic and trained model created in SageMaker Canvas, ensuring consistency and efficiency across workflows.

Implementation and Pipeline Configuration

GS Caltex implemented a production MLOps pipeline using Amazon SageMaker Canvas, beginning with data preparation from Amazon S3. SageMaker Pipelines handles input data retrieval, preprocessing, and batch transformation, storing results back in S3. The pipeline includes several key steps:

  • PrepareInput: Adds a placeholder target column if needed.
  • PreprocessNewData: Applies the SageMaker Canvas preprocessing flow to transform input data.
  • DropTargetColumn: Removes the target column and CSV header to generate feature CSVs.
  • BatchPredict: Executes batch inference using an approved model to produce prediction files.
  • AssembleResults: Combines preprocessed data with predictions to generate the final result CSV.

Each step must be configured individually before connecting them, ensuring a structured and repeatable workflow.

Automation and Scheduling with EventBridge

GS Caltex leverages Amazon EventBridge to automate and schedule inference processes using Amazon SageMaker Canvas. By creating a rule named canvas-process-inference-every-30-minutes with a rate(30 minutes) schedule, EventBridge triggers a SageMaker pipeline at regular intervals. The rule is configured with a target that references the pipeline's Amazon Resource Name (ARN) and an IAM role that trusts the EventBridge service principal and has the sagemaker:StartPipelineExecution permission. If any targets fail to be added, a RuntimeError is raised, ensuring immediate visibility into scheduling issues.

EventBridge integration supports both real-time and batch processing, depending on workload requirements. For example, a 30-minute schedule results in 48 runs per day, which is suitable for processing accumulated data at regular intervals. This approach enables GS Caltex to manage resource lifecycle efficiently, with real-time endpoints remaining available for ongoing requests, while batch jobs provision and release resources as needed.

Outcomes and Operational Efficiency

At GS Caltex, the integration of Amazon SageMaker Canvas into production MLOps workflows has led to measurable improvements in operational efficiency. According to results shared by GS Caltex, the repeated work required to connect SageMaker Canvas models to production has been significantly reduced . Additionally, the company has successfully reused assets that were validated by process engineers within SageMaker Canvas . As illustrated in Figure 1, SageMaker Canvas assets are now connected to a SageMaker Pipelines workflow for scheduled batch inference . This setup allows ML engineers to receive the preprocessing flow and model from SageMaker Canvas, where they can review data formats and prediction results, ensuring alignment with production requirements.

Sources and reporting notes

Based on published source material. No independent hands-on testing is claimed.

Read more