> Looking at our VPS provider bills each month makes me cringe.

Hosting costs often go up slowly over the years, and eventually, you have an unsustainable price. Just get quotes from a few other providers and go back to your current host and ask what they can do about the 70% price difference.

> Has anyone done this and what was your experience?

2/3 of the companies I own are on AWS. The other company is on dedicated, colocated hardware. The one on dedicated hardware gets zero benefit from CDN and cloud services as it's just a Django/Mysql monolith where every request and response are going to be different. We moved it off of AWS because there was little benefit, and we would reduce our hosting costs to a few hundred dollars a month for 20x more hardware performance.

> How do you manage the hardware and how much savings in time/$ is there really?

For the two companies on AWS, it saves us three $100k/year salaries per year. So, yes, it's more expensive than colocated hardware, but a lot less expensive than the colocated hardware and the three additional people required to provide support and ensure service levels. For the colocated hardware, we use fabric (an old python ssh automation library) to manage the servers and make heavy use of systemd's logging and systemctl for managing processes and logs. It works well, and there's maybe 1 hr a month of actual admin work, mostly dealing with OS updates and the occasional deployment hiccup.

> less expensive than the colocated hardware and the three additional people required to provide support and ensure service levels

i often hear people comparing cloud vs colo but not including more traditional dedicated hardware rental - seems to me this should be pretty similar, maintenance wise to cloud since H/W is handled by the vendor (just need to monitor hardware faults in addition to OS/app layer & notify DC on issues)

The difference is that in cloud you get live migration (generally) that works. So if whatever machine you are on dies, you don’t have an expensive and disruptive rebuild/restore cycle to deal with.

Since this can take a day+ even once you get new hardware, and architecting most software stacks so you have the redundancy and ability to migrate services around without downtime is hard, and hence expensive if you don’t have the expertise (AND randomly available time), running on dedicated rented hardware really shouldn’t be compared to cloud.

It’s a fundamentally different type of experience for the vast majority of people.

Solutions like VMware vCenter work well for live migrating, cluster balancing, HA priority etc. it’s not exactly a cheap product though.

When did any cloud provider other than Google get live migration?

GCP definitely live migrates and it’s well documented. Azure and OCI do too but details are a bit less clear. AWS does not afaik

You would likely recoup the costs and possibly earn terrific ROI by having 3 additional smart cookies on board full time since it would only take a couple hours per month to manage COLO. Maybe even less. AWS is a beast on its own and unmanaged can lead to a ticking time bomb of a mess.

I'm pretty sure adding $300,000 in salary to manage a $300/mo hosting bill isn't the best idea... but I do agree they'd be able to do more that just wait for something to break. AWS isn't bad if you have some discipline.

> I'm pretty sure adding $300,000 in salary to manage a $300/mo hosting bill isn't the best idea...

$300/month would be one bill from the total cost of operating a service. Just because that bill is relatively low, that does not take away from the value provided by the service.

Let's put it the other way around: would it suddenly become a good idea to hire a bunch of engineers to manage the exact same service if the hosting service decided to bump up it's cost?

Ok, so the AWS bill is roughly $1,500 per company. That includes CDN, DNS, backup/restore, load balancing, serverless functions, database hosting, geographic redundancy, file storage, firewall, log aggregation, a data warehouse shared memory cache, web servers and utility servers. I'd estimate each company could probably host all of it on four or five dedicated servers, and let's say we got the deal of the year on dedicated hosts: $50 per server. So, the hosting bill is $250/month. That leaves $1250 to manage everything and staff for operations on nights and weekends. Since the number of servers is small, there's no geographic redundancy, so we're at higher risk of an outage. There's a huge advantage here, and the truth is that over time, prices have went down, not up for cloud services.

The priceless part is that I'm not ever on call, for most server problems, and my tech team has a much higher quality of life because they are not on call on nights and weekends either.

The flipside is that other company, the one where dedicated servers/colocation made perfect sense. The infrastructure is smaller needing only backup/restore, web server, load balancer, database server. The hosting company can easily handle most of the commodity issues, and our team is only really managing application deployment, so it works.

We have presence both at AWS and in a colo facility.

I wouldn't say it's a slam dunk for either one. The Colo monthly bill is much lower than AWS, but when you add in our initial and yearly hardware expenses they are similar. They run different workloads, not smaller or larger, just different.

I'd generally say that maintenance at the Colo has been low, but we've had some instances where managing our own hardware has delayed rollouts (getting a vendor to replace some bad RAM on a brand new box), and where we put huge amounts of time into investigating hardware problems.

In particular we have a couple of machines that have been in service a year or two, that started experiencing data corruption and performance issues. We have really good hardware support, and they tried, but honestly I think the issue is that the SSDs they were sourcing were weren't great. I've spent probably 100+ hours on that issue alone.

There's also the cost of maintaining your own setup. The number of hours we've poured into Kafka, ElasticSearch, and Kubernetes that probably could have been reduced by just clicking a button at AWS is pretty high.

Also, it's very nice to just have resources. We are spending a lot of money on primary and backup copies of some data at S3, plus egress. It would be cheaper to serve it from the Colo, but then we need ~50TB of storage. Provisioning that amount of storage at the colo wouldn't be cheap. Disks are cheap, storage is not.

> Disks are cheap, storage is not.

I really like how you worded this, because I'm running into the same issue with my current fleet of servers. They all run kubernetes, have 100gb interconnects, are stuffed with NVMe drives and spinning rust (28tb per node, 2x2Tb nvme, and 4x6tb spinning drives) and, I've had the hardest time actually turning all those disks into a reliable, performant and easy to manage storage system.

Have you tried Ceph? Seems purpose built for just that sort of setup (metadata on NVMe, data on spinning). I've been watching it for years, but only recently set up a test, and it worked well even through a power failure at home. My previous company had been using it after I left, and they seemed happy with it, but I haven't asked too much about it. I'd like to start doing some testing with it at work.

I am currently running ceph actually! Out of the box, only having NVME in my storage pool, with 100GB interconnects, I'm getting about 50Mbit/s in Fio write tests.

I'm not a storage expert, or a ceph expert clearly, but so far of the options I've tried (Portworx, Rancher) it's the worst performing, and I don't know why.

Was doing that just recently, dedicated server stuffed with spinning rust.

ZFS to the rescue

> zpool create tank raidz2 sda sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo

> zpool list

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT

tank 127T 1.30M 127T - - 0% 0% 1.00x ONLINE -

> df -h | grep tank tank 102T 256K 102T 1% /tank

PS: not an expert, just my 2 cents. do your homework

Oh yes, I’ma big zfs fan as well. To be clear though I need to be able to survive a node failure in addition to a disk failure

Any specific challenges? I'm running something similar with a little more storage density.

I'd like to chime in from perspective of data center operator.

I don't see technical support being talked about here, but rather lack of it (hardware failures etc). I don't know if it is the norm in industry or not (I'm just a techie), but we do actually have 24/7 support available. With the ability to actually call support over the phone. The 1st line support is easily penetrable - exactly when you ask questions/operations that are not in their competence. Some premium customers even directly calls to upper level support. Kind of not official, but that's what it is when relationships get developer with customers.

So basically, depending on the support level, you can get your hardware troubleshooted/replaced and appropriate actions carried out within OS level (if we administer your OS or you provide access just in time). We actually have value-added support, that means, we do manage infrastructure for customers at certain level: AD, SCCM, MySQL, MS SQL, networking, linux stuff, private cloud, etc. I was actually a techie (until taking another position 2 months ago) for various MS things including SQL Server performance troubleshooting - when some application gets mis-developed and it has impact on performance, it was job for me to identify and/or mitigate the issues at hand.

I don't know if value-added support is standard or not in industry, but we surely try to add to our specialty.

Another point

I colo at a local ISP. I've been with them for about a year and I'm happy. Selection was easy: I wanted a local shop and there's only one in town. I had worked with them before on other projects and figured we would get along well.

I manage the hardware myself, mostly remotely. Occasionally I'll go onsite to upgrade or repair something. I buy used servers and parts from a little over five years ago or so. A lot of folks buy new and replace every five years, so this is a sweet-spot for performance per dollar. Kinda like buying a car just off-lease.

Working a cycle behind has its own benefits. If you pick a very common model you'll get cheap and easy-to-find parts. I was able to get a full set of spares for far less than what a typical hardware maintenance contract would cost per year (and I have them on hand rather than in 2-4 hours). Drivers (especially open source drivers) will be better developed and less buggy and you can probably find someone else's notes about anything that usually goes wrong.

Of course if you need cutting-edge gear, this won't be a good option. But I don't, so I take advantage!

I think whether you'll save money depends a lot on how you do things. There are efficiencies of scale that big providers benefit from, but there are also efficiencies you can leverage if you're small and flexible (like tailoring hardware to your use-cases and not doing/supporting stuff you don't personally use).

I didn't make the move to save money, but to get more power and flexibility. So far, so good!

Good luck! If you decide to dive in, I hope you'll come back and let us know how it goes!

This is a great answer. If any of the details bore you, don't do it ;) Renting from hetzner and other low cost dedicated suppliers will get you almost all the cost savings, and none of the heart stopping horror of having to drive to the DC at 3am and diagnose some knotty RAID failure.

Ah but if you've never done that, you've never lived!

Where do you buy used hardware?

Not OP. Off-lease equipment is readily available online. For one-offs, just use ebay. Beyond that, just reach out to the ebay sellers directly (their username is often the company name).

Do you have any tips on search terms you've had success with? I've seen a number of good listing from defunct startups that fire sale their remaining hardware on Ebay - but always through a direct link shared elsewhere. I don't know where to begin with finding stuff like that myself.

For eBay, I just search "1u server" or a specific make/model ("dell r720") then filter down from there. Once you find a combo that works, then search by that specific model or view the seller's other listings. Many times they'll have pallets full of each type of server and may be willing to offer volume discounts.

The suggestion below to search for "1u server" is probably good to get a feel for what the overall used market is like and what relative availability is. It helps to keep up a little with the product line of whoever moves a lot of equipment in your area (Dell around here) so you'll know what the popular/good models are to watch out for when they hit the used market. It also helps to know what options are available and/or common for the model you're looking at so you know if you're seeing something that's specced up or bare-bones or in between.

Server manufacturers usually have someplace on their websites for hardware or technical guides or spec-sheets. They're often available for 10+ years after they quit selling a model, so you can research a system before you buy a truckload of them.

eBay has a good selection. I prefer to buy from sellers that specialize in server gear -- the descriptions are usually more complete/accurate and they'll (hopefully) know how to pack things so they arrive in good shape.

If you have a good surplus store or recyclers in your town it might be worth checking with them to save shipping (servers are heavy!). If you plan to buy a lot of gear regularly, it may pay you back to do some networking and make friends with folks who buy new and replace after five years -- become their recycler ;)

Don't. It doesn't make sense to try to jump into an area rapidly being commoditized. At most, rent bare metal/dedicated servers rather than VPS.

> Has anyone done this and what was your experience?

It works as expected. Big cost/time upfront to get all the equipment setup, then things mostly just worked. The primary challenge is transferability and standardization.

> How do you select a colo and what do you look for?

Bandwidth availability. Proximity to the team supporting it. Reputation among customers.

> How do you manage the hardware and how much savings in time/$ is there really?

Whatever custom stack is built in the colo must be well documented, and have solid maintenance schedules, and change control procedures. The cost savings exists, but it's never as much as you think.

It's difficult to articulate the specific drawbacks. It's more organizational than operational. Imagine that you have a server dropping at random times. The logs say something ambiguous, and now you have to become full time sysadmin to some elusive problem. No one in your organization will want to deal with these sorts of issues, nor will they want you to burn time on it. There will inevitably be other tasks higher on the list. Operational issues gradually build up like other forms of technical debt. No one receives applause for updating the server BIOS over the weekend. Operational discipline/maintenance inevitably becomes last priority.

What you describe only becomes a bigger problem with staff turnover. It gets to a point where fewer people know the ins and outs of the setup and are prepared to maintain it. You end up burning out a few people, or bringing in people just to deal with low-level operations issues that were created by the move away from the VPS-level of operations.

The question to ask, is it strategic to move to running our own servers? If I had a large, stable, and boring set of apps which consistently didn't take advantage of "cloud features", then it might make sense to consider moving to colocation because the only option to improve profitability may be to reduce cost. That assumes that major changes won't be made to the app that require cloud features. This is rarely the case. The opportunities seem to only appear at huge scale.

> It's difficult to articulate the specific drawbacks. It's more organizational than operational. Imagine that you have a server dropping at random times. The logs say something ambiguous, and now you have to become full time sysadmin to some elusive problem. No one in your organization will want to deal with these sorts of issues, nor will they want you to burn time on it. There will inevitably be other tasks higher on the list. Operational issues gradually build up like other forms of technical debt. No one receives applause for updating the server BIOS over the weekend. Operational discipline/maintenance inevitably becomes last priority.

This sort of thing happens in dedicated hosting too. Unless you want them to just swap a box (which is OK I guess), chances are their support team doesn't know what's going on, and isn't going to spend all day diagnosing your weird issue that only happens sometimes. So someone on your team needs to do it.

I don't have enough experience in cloud hosting to know, but I'd guess it happens there too; although I'd guess if you see weird issues, swapping to a new VM is standard operations there.

The major advantage of owning your own servers is that you, well, own them. Ideally you'd rent space at a carrier hotel like One Wilshire where you have access to numerous providers in the meet me room. That way if whatever it is you're doing suddenly becomes unfashionable, you can't have some activist cloud provider decide to destroy your business on a whim.

Usually it goes the other way around.

Your system critical servers are too costly/too resource intensive and you move them to dedicated.

If you're simply looking to reduce costs, why do you want colo?

Can rent dedicated server from anywhere Hetzner and OVH already mentioned, there's many others.

- Want cheap storage? Backblaze?

- Want cheap VPS? Vultr?

- Cheap storage VPS? Time4VPS or something else?

- Cheap dedicated server? Hetzner/OVH

- Cheap bandwidth? Maybe a CDN provider like bunny.net or a dedicated server provider like https://100gbps.org/ can offload traffic

Plenty of options for what you're looking to optimize for, just need to google

Colo is a whole different game, why go in that direction

Suggestion: share a ballpark of your VPS bill to get better advice. The best guidance will depend on whether your VPS bill is order of $500/mo, $5k/mo, $50k/mo, or higher.

It also might help to share some characteristics of your workload. Is it CPU or disk-intensive? What kind of uptime expectations do you have? How effectively do you expect to be able to predict your requirements in the future?

This is a great suggestion. There are countless times as a consultant where I've discovered overprovisioned capacity, poorly optimized apps, sub-optimal VM types used, etc. Problems from those will still manifest in a colocated environment.

Colo is great.

Sure all the cloud fanbois will tell you that you should only ever use the cloud. But frankly if you have decent tech competence in-house and you do the math honestly, the cloud is far more expensive than colo, particularly for 24x7 loads over a multi-year period.

If you buy quality kit and you put it in quality colo facilities then your hardware management should be minimal.

Your main problem is going to be with your "How do you select a colo and what do you look for?" question.

Whole books could be written on the subject of choosing a colo facility.

If you are serious about colo, you should start by doing a few site tours with different operators. Doing that you can (start to) develop a feel for what differentiates facilities.

There a lots of things you could look at, but the two near the top your list should be:

     - Connectivity: Ideally you want a carrier-neutral facility with a good selection of carriers to choose from
     - Power: Self-explanatory really. How much do I get ? Is it true A&B feed ? Are two good starter questions.

If you look at those two items first, then you can start to draw up a shortlist from that. Once you have the shortlist, you can start nitpicking between the options.

You can choose Equinix for colo. Best in breed provider.

> You can choose Equinix for colo. Best in breed provider.

Bold statement there, you sure you're not an Equinix rep ? ;-)

I can think of a number of geographical markets where Equinix exist but in reality they don't deliver anything the competition doesn't deliver, and in certain geographies the competition is actually better.

> How do you manage the hardware and how much savings in time/$ is there really?

Probably depends upon how many servers you need. We colocate around 2 dozen servers and between things like updates, maintenance, and hardware, we average maybe 1 work day per month managing them.

By far our most common failure is just hard drives. We have a box of spares. Our second most common is motherboard failures - popped capacitors - from servers that are 10+ years old.

Last time anything failed was about 9 months ago. Before then we went a few years without a single hardware failure. But back in the 00's we got a bad batch of power supplies that hand a tendency to catch fire - those were interesting times.

The colo center is just a 5 minute drive from our office. And there's remote hands for simple things.

A company I worked for maintained a very large on prem data center to much success. We maintained some production / user facing infra, but it was mostly for internal stuff.

We had one massive compute cluster running vmware as our personal cloud. Any dev could spin up VMs as they needed. Once this was set up (which was a lot of work), the maintenance cost in $ and time was basically 0. We also had as assortment of baremetal servers used for all sorts of things.

One of the reasons I think it worked so well for us is because IT/Linux/sysadmin skills were very high throughout the company (which I have since learned is rare). Any engineer could manage VM images, recable a server, configure VLANs, etc. If this wasn't the case, we probably would have needed double the IT team, and a lot of the cost savings would disappear.

vSphere has significant licensing and support costs. Last time I checked a minimal, managed vSphere cluster runs like $3600/month. I’m not sure how this can translate to “basically $0”.

If VPS/cloud costs too much for you, try renting dedicated (bare metal) servers. There are lots of options out there, from tiny RPi-type boards all the way to multi-CPU behemoths. You don't need to bear the upfront cost, you're not on the hook for replacing faulty parts down the road, and the total cost of rental over ~5 years isn't massively different from the cost of buying, colocating, and fixing your own hardware.

I know someone who rents a 3990X monster for less than $1K/mo. Renting comparable compute capacity from any well-known VPS/cloud provider would cost at least 10 times as much. I also know someone who rents a cluster of servers with several large disks, pushing hundreds of TB of outbound transfer, again for less than $1K/mo. The bandwidth alone would cost many times as much on AWS, not to mention the storage. Of course you'd be missing the redundancy of AWS, but whether you really need it for your specific use case is your decision to make. Anyway, the point is that most of the savings are realized in the move from VPS/cloud to bare metal, not in the move from rental to colo.

Do you actually need colocation or can you do with a middle-ground of just renting a bare-metal server from Hetzner or OVH?

We specifically colo GPU hardware due to how expensive it is.

For instance this year we spent ~$500k (rough estimate) on purchasing GPU hardware. With cloud that would have costed us ~$80k/mo - so after ~6 months we break even on the hardware and then after that just have to spend the much smaller colo bill. We expect hardware to last us 3+ years so we come out far ahead by buying.

Our work is so specialized anyways that if we went cloud we would still have to have the same dev ops staff anyways so there is not much cost difference on that side of things. We had to invest a little bit more upfront in deployment, but in the end it is peanuts compared to the $80k/mo that we are saving in cloud costs.

The only trouble is when we need to scale beyond our currently owned hardware. In that case, once we are above capacity of servers we own, we make another purchase and in the interim use cloud to bridge the gap.

In the end purchasing hardware ourselves is one of the highest ROI moves we have made and I would highly recommend it to others. Of course if your cloud bill is much lower you might find the cost savings to not be enough to be worth the hassle.

Yep, this is exactly why I'm looking to colo. I have eight or so RTX a5000 gpu's and although colo is expensive, it's an absolute slam dunk in comparison to renting from AWS or even "dedicated" companies that only rent out GPU hardware at slightly better rates. I will say, power and efficiency become a bigger issue when colocating - if our scaling requirements and credit were slightly different we'd likely have preferred a greater up front cost and purchased V100 gpus instead of RTX a5000's.

I'd recommend Colovore for power - they are specialized for this use case and do a great job for us.

We started in the same situation where we had 10x RTX 8000 cards. We've since added tons of A6000 and A100 cards in our cabinet and have had no power issues.

Colo is fraught with peril.

Do your due diligence in vetting any company before committing your gear and uptime to their care.

There’s a whole lot of clowns operating as colo resellers. And by clowns I mean a lot of them range from incompetent to outright scammy.

Yeh, make sure they can actually provide the amount of power and cooling you contract for

Even the honest ones sometimes get it wrong. No fun if your kit is in their facility and they go under. Doesn't happen too often thankfully (and to be fair could happen to a smaller VPS provider too, AWS less likely).

Lots of folks focus on compute and disk cost when they're in the cloud because they're usually the two biggest items on the bill. That's reasonable, but when transitioning a distributed system to a hybrid infrastructure model (eg: Cloud and a Dedicated Colo) it's important to factor in network cost. Cloud providers usually charge for ingress or egress and have marginal cost for inter-DC traffic (eg: availability zone to availability zone on AWS). Distributed systems are chatty by nature, so if requests are constantly leaving your cloud and entering your new DC you're potentially paying twice for the same transaction. This cost adds up fairly quickly. The same thing will occur if you operate an application in two regions on AWS and have a lot of region<>region activity.

Don't manage your own gear. It's a bad idea that almost never helps you, like doing all your car maintenance yourself. Changing your oil yourself? Cheap and easy. Replacing your entire A/C system? Pay someone to do it. Life's too short, and you won't do as good a job as a pro who does it all day.

Things a VPS provider does:

  - Buy hardware (every 3 years)
  - Rack hardware
  - Network the hardware
  - Power the hardware
  - Troubleshoot and replace buggy/shitty/dying hardware
  - Provide an API to:
    - out-of-band power management
    - out-of-band console management
    - remote deployment of VMs
    - out-of-band network maintenance
  - Patch security holes
  - Deal with internet routing issues
  - Move everything to a new colo once the old colo becomes more unstable and expensive

If you want to do all of that, you might as well get someone to pay you to co-lo their stuff too, and then provide an interface for them to use, and then you're a VPS provider.

There is only one good reason to colo your own gear: if you have to. If what you want to do simply isn't possible at a VPS provider, or is too costly, or you wouldn't get the guarantees you need. It's the last resort.

I helped manage dedicated servers in colo for a small company for a few years, and I don't have very good memories about it.

We had a cabinets with inter-connected machines, so we had our own routers, load balancers, two uplinks, etc. It's a lot of stuff to manage for maybe 25 servers (which ran VMs).

Currently I use OVH and Hetzner and find it to be the best of both worlds. Hardware isn't my problem, and the costs are low. We run ZFS and KVM, so moving a VM around is easy.

On another project though, we split $5k of hardware for a server with 20TB of disks, 4U. The server has been running for over 5 years without issues. For a single, special-use server, colo can be nice.

Check the WebHostingTalk forums [0] for deals and promotions, especially from small providers (assuming your business case is compatible with that).

You might also want to start by renting a “dedicated server” instead of colocating your own hardware. The savings will be still be significant (especially with unmetered bandwidth, if you’re comparing to cloud vendor price gouging).

As for personal recommendations, I’ve had good experience with Psychz.net FWIW.

[0] https://www.webhostingtalk.com/

WHT used to be the place for hosting. Psychz.net is old school, but William Lu was always solid back in the day.

(+1 for someone from WickedFire, too)

One trick I've found for colos -> Find a small local ISP or WISP, get to know the owners. Do research on local colos and what they charge for say one rack. Then halve that price, split it amongst 2-3 friends, and make an offer to the ISP. More often than not they'll accept, especially if you can pay 3 months+ in advance.

Find colo with 24/7 access, high up time (good networking, multiple connections to the internet from independent providers and proper setup for working when there's no power) and "remote hands" in case you need to manually reset server.

Rest of the questions you have to answer yourself. Initial server cost will be way higher than VPS plus maintenance and paying for colocation is also not cheap. Servers will have to be upgraded every ~5 years, depending on the scale and you have to buy machine for the worst case scenario, there's no automatic scaling so if you sometimes need 32 cores and 1TB of RAM you have to buy that even if 99% of the time it will sit at idle.

I would rather find cheaper VPSes for non-critical systems or work on optimization of current solution.

I've done this off and on for the better part of two decades. I know a few good colo's across the US if your interested.

Much of it is going to depend on your workloads. If your just running emepheral vm's on something like vmware or another hypervisor, you won't run into much of a problem.

Things start getting a bit more complicated if you are going to be using HA storage/databases. But again, that depends on your workload. And some datacenters will be happy to manage that for you.

There is alot of money that can be saved when your workloads are fairly static though. The key is putting together some hardware automation (jumpstart your hypervisors) and handling the depreciation cycle.

To the people suggesting that renting or installing a few servers in a leased rack space with redundant cooling, power, conditioning and 24x7 security is somehow dangerous or hard, please go home and sleep it off. You are drunk off cloud kool aid.

I’ve done both and am currently doing both. Like other commenters have said it depends heavily on a lot of your specific circumstances.

I’d be happy to give you more advice if you can say more about who’d be managing it, what your current costs are, and roughly what your use case is, etc. For some generic advice though, I’d say renting dedicated servers is typically the way to go unless you have a specific reason you want your own hardware.

In my case, my reason is experimenting with RDMA over Converged Ethernet for machine learning and I couldn’t find machines with GPUs and capable interconnects for rent. If you don’t have specialized requirements though any provider is probably fine

I’ve heard good things about Packet.com. They were acquired and is now part of Equinix Metal. Might be worth having a look at: https://metal.equinix.com/

I have a pretty even split of customers in cloud providers and hosting on rented servers in a data center. Using something like Hivelocity kind of splits the difference--they provide the machine for a low premium and you do all the management. They'll handle all hardware swaps and physical poking/prodding. The price beats the pants off of hosting in AWS/DO, but its pricier than just outright buying the hardware. All things have tradeoffs.

> I was thinking maybe we could offload non-critical systems to our own colocated servers onsite

Definitely have a strategy for cooling the servers in place. If you put two dozen servers of a certain type in a room, how much are they going to warm the room up? How are you going to cool the room off in the summer (and other seasons), and what will the temperature of that room be in the summer on Saturday at 6 PM, and will anybody be around on Saturday at 6 PM, or Sunday at 4 AM if needed? If you have a ventless portable air conditioner in the server room (not that I am recommending it, but I have walked into many on-site server rooms with them), does condensation form in it? If it drops condensation into a bottle, who empties the bottle? What do you do if the condensation bottle fills up at 6 AM on a July Saturday and A/C goes off and then the temperature rises through the July day?

It's good you are thinking about this and planning this, because I have seen this happen in an unplanned manner many times. Two or three "non-critical" (until they crash) systems are put in a room on-site. Then without much planning, another server is added, and then another. Then it starts to get too hot and a ventless portable air conditioner is put in. Then the condensation bottle fills up and you suddenly have a meltdown in the room as one server after another overheats and goes on the fritz. I have seen this happen at small companies, I have seen this happen at Fortune 1000 companies.

So my advice - have heating fully planned out, and be aware that once you set a server room up on-site and its working, other divisions will start wanting to put servers in there, so pre-plan the rooms maximum capacity. I suppose electricity and racking and access and security and such need to be planned out as well. The main problem I have seen is overheating, as people, without planning, keep squeezing just one more server into the room.

> How are you going to cool the room off in the summer

In my (albeit outdated now) experience, cooling in the winter can often be even more of a problem in cold climates, if you have a condenser unit outside in freezing weather.

Lol, definitely still relevant - we just dealt with that. There are specific models with heating blocks for the compressors (one would hope the heat is recycled but I suspect it’s not).

Don't do it, the last thing you want is a raid card malfunction whilst you are on holiday. You'll be talking downtime of days not minutes.

You have to plan for worst case scenarios and how your team mobilise. These things are the "insurance" you are buying when you make the hardware someone else's problem.

> "Don't do it, the last thing you want is a raid card malfunction whilst you are on holiday. You'll be talking downtime of days not minutes."

Erm ... you do realise it is possible to fit more than one server into a colo rack don't you ? ;-)

N.B. Some might also say "who uses RAID cards these days ? ZFS FTW !!"

That’s what remote hands are for. When I worked at a data center we did that for free, even provided parts and billed the customer if they didn’t have any spares.

It can work for some usage patterns, but it’s important to understand the trade offs

Hosted gives up easy hardware replacement, easier redundancy, 24/7 monitoring etc

But a hybrid approach worked well for one place I worked at.

Their work pattern was:

Job scheduled by scheduler and pushed into queue (all AWS systems)

Local cluster of 20 machines pull from that queue, compute the job, and store the result back.

It worked because: 1) if a job failed the scheduler would re-schedule it

2) the local machines were treated as disposable, if one broke it was repaired or thrown away

3) deployment to the local nodes was simply copying a binary and config file

4) the latency didn’t matter as most effort of the jobs was compute

5) the bandwidth didn’t matter because the jobs were small and didn’t require much data

6) the tasks were embarrassingly parallel, so throughput scaled linearly with nodes

Sometimes it can work, but without knowing the specifics of your compute jobs the best I can do are the above generalisations

The experience is pretty much the same except for depending on how secure the facility is.

Look for? Mostly costs.Unless you have some regulation requirements.

Savings is going to be heavily dependent on what you do with the stuff there. If you have legacy stuff or huge full server usage (looking at you accounting systems) a colo can be a lifesaver. If you are 100% wiz bang 2.0 or whatever there is probably little reason to have a colo at all.

Just make sure there is some provision for the fact that its not a one time cost and servers need replaced ever 5 years or so and parts need replaced. You don't want the "computers don't have moving parts why should we replace them" conversation when a critical server is on its last legs at 10years old or more.

Anyone have suggestions for sites in the tri-state area?

The biggest advantage to colo is crystal clear and consistent broadband. Fios fiber is as close as you'll get in an apartment or office, but in most cases "gigabit" service is around 800mbps down and a meager 40-50mbps up :( .

Also, if anyone here is looking to split a rack in NYC or within 1hr of Manhattan I'd be very interested in splitting a full rack of colo. I have some gpu hardware for a side-project I need to get out of my apt and connect to real broadband. My only requirement is 1G/1G broadband and around 1300W of power. (email: hydratedchia at gmail dot com !)

I solve this by building a 3x VPS (one node per continent)/2x home fiber combination (for redundancy). That way you get the best of both worlds:

For real-time stuff you use VPS, for bandwidth/storage you use home since those are cheaper there!

I use all passively cooled, lead-acid backed; Atom (loadbalancer/fast disk) + Raspberry 2/4 (slow disk/workhorse) and my own stack that uses Java.

This week google had a breakdown, so now I'm going to make my backups (AWS in asia and IONOS in US) part of the default DNS.

Only problem I have is IONOS does not easily scale up so I dont know what to do for low latency VPS workhorse in the US!

Why not rent one or more dedicated servers rather than collocation. That way you don’t have to tend to hardware issues while still being cheaper than the cloud.

These days you have to treat colocation as a fungible resource. It's good for batch processing and things like that where you can distribute the work.

How does your billing work? Are you on a plan for that VPS with dedicated, fixed burstable, or 95th percentile burstable? You may be able to find some less cringe worthy bills if you change how you are billed. Although that might mean you have to change providers, too.

I won't give you any advice on the feasibility and price, but if you want cheap Colo, HE offers a full cabinet in California for 400/mo. That's a gr deal. In Chicago just a rack without the internet is around 1200

Did you consider moving this non-critical load to a cheaper VPS provider? You are unlikely to be able to beat them on price when taking into account hardware and engineering time.

colo rather than stuffing it in a cabinet please. a decent colo provider should be able to handle the rest of the stuff for ya after you send them the hardware.

> Has anyone done this and what was your experience?

I have 2x 2u in two different datacentres in the UK. Connected together via IPSEC using pfSense.

4u is common, and a rack would be nice.

Your costs will come based on racking units, bandwidth and transit/connection speeds. 1Gbit is going to be more than 100mbit but you can normally negoiate this.

Hardware costs are up-front but when the server is humming, they last for a good period of time.

You don't normally gain DDoS protection so if you require it, factor that in.

> How do you select a colo and what do you look for?

Choose whether you would like:

Commercial (Cogent, Peer1, Rackspace) who own actual buildings and run as an datacentre as a datacentre. Try to provide the greatest, hand you an account manager and the ability to ticket the DCop monkeys.

Or, independent who own industial units called datacentres. Who tend to over decent internet feeds and hold more of a independent feel. However they may lack the 24/7 support you may need. Or enterprise features that you may find within commercial.

In terms of selection I recommend taking a look on WebHostingTalk.com under their colocation section. Google yeilds good results. Find a provider, get in contact and take a tour of the DC and see how it feels.

> How do you manage the hardware and how much savings in time/$ is there really?

My servers are all second hand eBay purchases and have had a lot of TLC to get up to scrap. Once tamed, they purr. The amount of work on them is near to none. Create a new VM, Update OS; general SysAdmin stuff.

I would recommend that if your looking for sanity: buy new. eBay servers are hit/miss and you never know what condition they will arrive in.

iLO/LOM is configured so I always have remote management unless the DC goes dark. Servers are resillent too, I have one server which has two ram sticks failling, but still operating with no issues.

Network issues can arise and you have to ensure it's not you causing the mishap before you can get actual help. You most likely won't get DDoS protection on independent-owned but is a growing trend so you may get lucky.

I moved from VPS to colocation and refuse to use the cloud. The cloud has its purposes but for me, I'd rather bare metal and hypervisors. Paying another company to host my information where I may not even have full control. Just doesn't settle with me if I am to provide client services. Plus, I can actually own the internet space these servers sit upon. No restrictions with what services I desire.

My OS is FreeBSD hosting virtual machines with bHyve within a jail. I will always advocate for colocation.

   FreeBSD 11
   5:09PM  up 791 days, 18:17, 8 users, load averages: 0.66, 0.82, 0.88

   FreeBSD 12
   5:11PM  up 174 days, 21:19, 2 users, load averages: 0.59, 0.68, 0.70
   5:12PM  up 174 days, 20:03, 1 user, load averages: 0.12, 0.11, 0.09

I like this setup. While I don't run much FreeBSD as of late, it certainly helped keep the pagers quiet over the years.

Are you exclusively hosting other FreeBSD vhosts under bhyve?

I have three different jails configured for the three different OS groups. Windows, Linux and BSD.

And when client want their own set of resources I just create a jail, give it an IP and they can do what they want.

I transitioned from VPS to rented dedicated servers years ago which was significantly more cost effective.

I recommend if you do this to try to keep your stack as portable as possible, it was relatively easy for me since I was already using Docker and started testing Rancher/K8s on the dedicated servers. This was years ago and I'm fully committed to K8s at this point.

This year I actually took it a step further and ended up just building a little 4U server that I colocated at a smaller data center that I was already renting dedicated servers from. I needed this for high data volume and latency needed to be as minimal as possible (CPU and storage together) while keeping recurring costs minimal.

For your questions:

> Has anyone done this and what was your experience? Relatively straight forward, a lot of up-front cost but has been overall about the same/breaking even with higher performance / results. I went with one that allowed me to rent IPv4 addresses without doing any peering or extra work, essentially just supply the computer, set it up, and let it go.

> How do you select a colo and what do you look for? For me, cost and latency. I've been looking into colocating another server in Asia but haven't had a lot of luck picking a specific data center yet.

> How do you manage the hardware and how much savings in time/$ is there really? Honestly, management has been pretty minimal. My server was entirely new so nothing has broken, I just keep tabs on it every couple weeks and make sure my ZFS storage hasn't turned unhealthy.

For some absolute numbers for you, my server specs and cost: 4U Colocated 4x 8TB HGST HDD ( Setup with RAID10 essentially, so 16TB usable space ) 2x 2TB NVMe SSD ( One actually isn't used currently, but is in the slot and available ) AMD Ryzen 9 ( 32 threads / 16 cores ) 4x 32gb G.Skill ram ( 128gb )

I also have a spare 256GB Samsung 2.5in SSD on standby (literally in the case, just unplugged) in-case something happens to the NVMe drives.

All-in, up-front was around $4k USD, monthly is $95 USD (all costs included), and I really only need to check on it every now and then and let Rancher/K8s take it from there. Previous costs were around $200-300/mo for a few different dedicated servers and S3 storage.

There have been incidents at the data center I went with which is definitely something you'd need to plan for, the one I went with seems to average 1 incident every 1-2 years. There was an incident a couple months ago at the data center (power outage), something happened with my server which actually required re-formatting the NVMe drives and re-setting up everything over the data center's supplied portable IPMI-ish interface, which required them to schedule a time to hook it up and then use it. Not every data center will have this or be as cooperative about it.

---

I'd definitely caution jumping over to colocation, start with renting dedicated servers at the very least.

This entirely depends. I know several businesses that save about 10X over cloud providers by using their own equipment. You have to do your own analysis.

Dedicated over cloud? Yeah, I am personally saving 40x on bandwidth alone.

Colo over renting? Savings seemed pretty small, and hassle was large...

Compared to cloud yes, but dedicated servers are cheaper than running colo, it takes a couple hardware failures to completely screw your budget up, in addition to the maintenance.

Unless you have specific impossible to rent hardware needs, it's never worth it.