In offline experimentations, we are used to testing various machine learning models, training and/or fine-tuning them and then using them for prediction (i.e. inference). Now imagine that we would like to move beyond just offline experimentation and provide our customers access to our amazing models so that they also can use them for prediction. In such cases, we can “deploy” our model to a SageMaker “endpoint”. Then our customers can send their requests to the deployed endpoint and receive real-time predictions. These endpoints provide certain benefits, including:
- Access: An endpoint is just a web address where the model is hosted (or deployed). Therefore, we can use it just like any other web address where we can send the request (i.e. payload) and receive a response (i.e. model prediction).
- Scalable: Once an endpoint is created, Amazon/AWS will take care of dedicating the necessary computational resources to serve our customers. For example, let’s say my laptop is only capable of processing 10 requests per second but I expect to have 10,000 customer requests per second. AWS will scale up the endpoint and provision enough hardware to support all 10,000…