Serverless and MLOps

Serverless is the new 'cool kid' on the cloud-computing block, with rapid adoption across all three major cloud providers. Just take a look at this recent report from Datadog based on data of over 20k customers:

But how 'cool' really is this new kid? Let’s examine the hype and find out why Serverless computing may be a good fit for your ML model deployments.

Speed, Scalability, Efficiency = The Buzz Worthiness

It turns out that Serverless computing is streamlined in ways that yield major payoffs where it most counts. Consider:

Faster Deployments: Serverless eliminates the need for server configuration and management, resulting in reduced time-to-delivery and quicker time-to-value for developers. Less setup and operational friction means more rapid model deployments and iterations.

Scalability: These Serverless systems are built for scale and used by some of the biggest players on “the streets”, like Netflix, Dropbox and Airbnb. This means you can build on infrastructure that has been stress-tested by some of the most demanding applications on the planet.

Cost Efficiency: Pay only for what you use, with no over-provisioning or wasted resources. If there’s no traffic on your API, there will be no cost (unlike with traditional servers or pods on Kubernetes).

Developer-Centric: Because most of the complexity is abstracted away, the dev experience is much more pleasant. This allows developers room to shift their focus to application and model logic, instead of low-level infrastructure maintenance.

Integrated Services: Built-in service integrations enable powerful combinations of compute, database, storage, APIs, workflows (like step-functions), logging, monitoring, and much more.

All in all, pretty amazing stuff!

MLOps & Serverless: A Perfect Pair

Applied to the field of MLOps, Serverless technologies have the potential to streamline and simplify the entire ML lifecycle.

Rapid Prototyping: The reduced overhead translates to more rapid iterations. This enables Data Scientists to get new data and new insights more quickly, allowing models and application logic to evolve more efficiently.

Reliable Model Deployments: By simplifying complexities, reducing fault tolerance, enhancing scalability, and security, model deployments are made significantly more manageable, drastically increasing the likelihood of project success.

Model Maintainability: Observability is built into Serverless systems by default, which enhances the creation of reliable applications from Day One.

Challenges

As cool as all that is, Serverless isn't without its challenges, especially when working with larger models.

Cold Starts: There can occasionally be a slight latency of anywhere from a couple of seconds to a minute or more when a Serverless function is initially invoked, which might be a concern for some real-time ML applications. The length of time involved mainly depends on how long it takes to load the required model artifacts into memory. The good news is that this usually happens in less than 1% of invocations, and there are many ways to dramatically shorten cold-start latency and frequency.

Resource Constraints: Currently, it is challenging if not impossible to deploy large models, especially behemoths like large language models (LLMs), on serverless systems, given the memory, loading- and execution time constraints. However, Serverless platforms are continuously evolving, so be on the watch for novel solutions in this area of applications.

Vendor Lock-In: There are still voices out there who are concerned about a lock-in effect to one specific cloud provider. These concerns usually arise from negative experiences companies had in the past, for example with products provided by IBM or SAP. But with modern cloud providers such as AWS or GCP, we often observe that the advantages of deep integration with a specific vendor’s services surpass any potential future costs of transitioning.

Wrap-Up

Serverless and managed infrastructure is set to change the way we deploy models, making the process faster, easier, and more cost-efficient. Considering all the above and based on my own experience, Serverless is an excellent choice for deploying small- to medium-sized models that use less than 8 GB of memory — a range that encompasses most specialized and self-trained models. The only real downside is that users might still experience cold starts in a small percentage of their requests. Ultimately, whether or not to take up with Serverless is a question of balancing faster deployment times, reduced complexity, lower costs, shorter time-to-market and higher project success rate against possible impacts on model performance—due to smaller model sizes—and the end-user experience. That’s a question only you can answer for your business and use-case.