The On-Prem Promise

Lately, headlines about companies moving away from the cloud and over to on-premises (self-hosted) servers have been hard to miss. Pointing to the lower compute and data transfer costs of these servers, the articles promise drastic savings for those willing to make the switch. But is that really all there is to it? In today's newsletter, I’d like to peel back the layers and offer a high-level, honest comparison of on-prem and cloud solutions in the context of MLOps. Maybe by the end of this article, you’ll agree that there are other important factors that need to be considered before a company decides to ditch the cloud.

1. Hardware Costs and Setup Time

Let’s be real: On-prem hardware takes a long time to set up and demands a sizeable initial (and ongoing) investment. The biggest costs come from operating and maintaining your own server cluster, which requires highly skilled IT professionals (which, unfortunately, seem increasingly hard to find). On top of that, estimating and purchasing the right amount of capacity is hard if not impossible, so resources usually end up underutilized. The cloud, on the other hand, allows a pay-as-you-go approach, so you're only billed for what you actually use. And as for time, you can get started on the cloud and spin up instances within minutes.

2. Control & Customization

With on-prem, you have complete control over your infrastructure and deep customization options. This can be a blessing if your needs are highly specific. The cloud offers some customization too, but only within pre-defined limits. To be honest, though, I’ve rarely seen teams struggle because they were lacking configuration options. On AWS, for example, you could rent plain EC2 server instances and build your setup from scratch if you want, just like you would do locally.

3. Security

On-prem puts the security of your system entirely in your own hands, but that’s not necessarily a good thing: locally managed systems often suffer from huge vulnerabilities due to outdated or unpatched software, inadequate networking setup and access control. Besides that, the servers are located at a physically known location, which means they are susceptible to theft, tampering, or unauthorized physical access. The result is that you have to pay extra for qualified security experts to manage the servers. But not with the cloud: it runs on data centers guarded by multiple layers of defense, and as a client you’re supplied with an entire suite of security features along with user access management, roles, and policies. Just keep in mind that cloud providers usually operate on a shared responsibility model, which means you still have to take some steps to ensure your services are securely configured.

4. Scalability

It’s possible to scale on-prem servers, but it requires constant investment in additional hardware—which is not always particularly convenient (or efficient). The cloud, on the other hand, automatically spins up resources based on demand. This can be a huge advantage in the rapidly changing world of AI and Machine Learning.

5. Reliability & Redundancy

With on-prem, it can take significant out-of-pocket infrastructure investment and manual effort to ensure that the system stays reliably online. Cloud providers, on the other hand, ensure consistent uptime through strict Service Level Agreements (SLAs). Data and workloads are spread across 2-3 fully isolated server centers (called “Availability Zones”) by default, so even if one of them goes down, you’ll never even notice.

6. Flexibility

Upgrading or switching out technologies on-prem can be complicated and time-intensive. The cloud offers the flexibility to experiment with different technologies and configurations without such heavy lifting, sparing you stress and supporting your drive to innovate. It also provides you with a global network of servers across Europe, North America, Asia, Australia, etc. making their solutions almost universally accessible.

7. Managed Services

Cloud suppliers provide a variety of managed and often “serverless” services, where developers don’t have to worry about the underlying infrastructure and can instead focus on writing code and creating business value. AWS, for example, has services such as S3, which provides virtually infinite data storage, and DynamoDB, a high performance, scalable but inexpensive database.

8. Integration

Many cloud services are designed to integrate seamlessly with one another, making it much easier for you to assemble a comprehensive, end-to-end solution without having to jump through integration hoops. A connected ecosystem like this can be much more challenging to build on-prem without dedicated and often expensive software tools.

9. Logging and Monitoring

A cloud service allows you to monitor your application and infrastructure health and respond to system-wide performance changes in real-time. On-prem solutions often must rely on a combination of separate tools to achieve similar observability.

The Bottom Line

It’s true that on-prem typically offers more customization and direct control over the system, with a lower direct cost for running the servers. But as I’ve tried to show above, these advantages are almost certainly more than offset by extensive indirect costs in the form of longer setup times and a long line of greater maintenance challenges. The cloud, on the other hand, has built-in flexibility and scalability, along with a growing suite of managed services that can save you headaches and allow you to focus on writing code and creating business value.

We’re fortunate to live in a time where everyone, even a lone engineer working on a small side-project, has access to the same infrastructure that is running the world’s most successful businesses, including Netflix, Airbnb, Facebook, LinkedIn, Twitch, Spotify and many more. My humble advice: take advantage of it.