Gradient Group – AWS Machine Learning Blog

How do you deploy Lambda functions as Docker containers through CI/CD? CloudFormation provides us two options for Lambda deployments: Zip the code, copy it to S3, and pass in the S3 path into the CF template Containerize the code, push it to Elastic Container Registry (ECR), and pass in the ECR image URI into the… Read more

We all start our AWS journey in the console. We do everything there. We manually create and configure Lambda functions, Step Functions, IAM roles, S3 buckets, EMR clusters, and any other service we need as we implement a machine learning solution.

We have trained a machine learning model that meets or exceeds performance metrics as a function of business requirements. We have deployed this model to production after converting our Jupyter notebook into a scalable end-to-end training pipeline, including CI/CD and infrastructure-as-code.

Is continuous training (CT) a machine learning operations (MLOps) best practice? It depends on what we mean by CT. Suppose it means continuously invoking training pipelines to ensure models ‘stay fresh’ as new production data lands in the data lake. The training pipeline workflow could be executed automatically on schedule once per month, once per… Read more

Unit testing is a vital element of production software engineering. After all, how do we know for sure that our code always returns the expected result regardless of input? Unit testing is especially important in production machine learning because model training and pre-processing functions do not always throw exceptions when they should. Instead, the errors… Read more

How do we ensure machine learning pipeline components produce the exact result we expect, especially prior to production deployments? We could sanity check by inspecting a few output records by hand, but how do we know for sure that all output records are correct every time? This manual, stage 1 automation “ClickOps” approach is not scalable, consistent,… Read more