AWS Cross-Account Deployments for Production ML Pipelines

How do you deploy a machine learning training pipeline as a CloudFormation stack from a dev AWS account to a prod AWS account?

Suppose you added feature engineering steps to a component of your machine learning training pipeline (within your development environment).

If you are using CodeCommit, CodePipeline, and CodeBuild for CI/CD, follow these steps to deploy the changes to your production account:

  1. Within your dev account, commit change within SageMaker Studio or Glue Notebooks and push to CodeCommit feature branch
  2. Pull request to dev branch, code review, and merge feature branch into dev branch
  3. Have EventBridge capture the event and trigger CodePipeline, with CodeCommit as the source phase
  4. Proceed to CodeBuild test phase to perform builds, unit testing, integration testing, and other steps (this could happen in the same dev account or a separate staging/test account)
  5. If the test stage succeeds, proceed to CodeBuild deploy phase
  6. Assume prod IAM role within CodeBuild buildspec.yml file using STS CLI (make sure the prod role has the non-prod account as a trusted entity)
  • Execute your required steps for CloudFormation deployments, such as containerizing Lambda code and pushing it to ECR (or zipping the code and copying it to S3), and deploy the CloudFormation template with appropriate parameter overrides (again, using the CLI for the respective services involved). All these commands are executed in the prod account from within the non-prod account inside CodeBuild – how cool is that?!
  • Write production deployment metadata to DynamoDB (whether CodePipeline succeeded or failed – keep track of everything for analytics!)
  • (Bonus) Log into your prod account and verify all changes were deployed successfully
  • If the production deployment succeeded, merge the last pull request into the master branch in CodeCommit as the final step in CodePipeline

This workflow makes cross-account production deployments in AWS seamless, reliable, and consistent. It’s also a great experience to have the entire process unified within one platform without external dependencies (i.e. GitHub or GitLab + Jenkins).

This cross-account deployment process is also cybersecurity compliant because no data is transferred across accounts directly. Everything happens securely within CodeBuild by assuming the production role to deploy code directly into the production account.

What do you think about this deployment process? Comment below! Feel free to suggest improvements, as well. Our solutions are constantly evolving and my team is always looking for improvement opportunities.

If you need help implementing AWS Well-Architected production machine learning solutions, training/inference pipelines, MLOps, or if you would like us to review your solution architecture and provide feedback, contact us or send me a message and we will be happy to help you.

Written by Carlos Lara, Director of Data Science & Machine Learning Engineering

Follow Carlos on LinkedIn: https://www.linkedin.com/in/carloslaraai/

Leave a Reply

%d bloggers like this: