The Worst Server Setup Mistake You Can Make

There’s one server setup mistake you can make that will screw you up for years to come. It’ll haunt your business, your developers and your server admins, and be hugely expensive to fix. It’s surprisingly easy to make. The mistake?

Setting the timezone to anything other than UTC

It seems like it’s going to be innocent. What’s the problem with setting my server’s timezone to EST (or your timezone of choice) you ask?

If you use a timezone with Daylight Savings (which is most of them), once a year (upon DST change) many scheduled jobs will just “go missing”. Of course, nobody ever runs nightly cron jobs for backups, or sending emails or anything else important, EVER THIS IS A WELL KNOWN FACT

DST is different in different timezones, and at some points the overlap will cause a bug when interacting with other systems (I’ve heard of this cause billing delays when one system was billing another with different days and billing was getting rejected because the dates were different).

I have (in the past), seen servers where the system timezone is set to PST, but the database is set to UTC, causing more crazy bugs, because for 8 hours a day, calling now() in the database has a different date from the application that talks to it.

There are thousands of bugs out there that are caused by timezones. UTC assuming code interacting with non UTC assuming code can, and has caused silent data corruption.

Migrating later than the start of the infrastructure can be really painful, because of your app getting coupled to the timezone it’s in.

At some point, you’ll want to run servers in different continents, or hire employees across timezones, both of which will be made more painful if your system doesn’t use UTC throughout.

There are millions of stories like this in the wild - debugging timezone related bugs is incredibly painful, migrating off of one of the other timezones to UTC is painful as hell as well.

How? Well UTC doesn’t have daylight savings. It’s universal, well agreed upon (though sadly it still has leap seconds, which can get kinda weird). Nearly every mature engineering organization runs all their servers on UTC, and you should too.

Use UTC as the only timezone for your servers.

It’s IMPORTANT that you do this at the start of your server setup. Moving the system timezone is not to be taken lightly. Thankfully, doing it at the start is trivially easy, and saves you from hundreds of problems down the road.

Often servers you rent from hosting providers aren’t set to UTC, so make sure you change this as soon as you set up the server.