For the longest time, I chose not to set up any metrics for my projects, including my personal site (the one you're currently viewing), since I didn't expect any remarkable metrics to come out of this process whatsoever. Now, that I'd like to see how my posts are performing, I've ended up with next to no fine-grained metrics except the number of requests going through Cloudflare, which is nice but clearly not enough. That's why I spent the last days setting up a very cost-effective and hopefully long-lasting infrastructure for persisting the metrics I want to track.
🌠Going back to self-hosted services
The sheer convenience we get from managed services is no doubt a big factor in choosing where and how to host the software we build. Not having to worry about deployment strategies, data persistence, security and other areas of the application lifecycle is really pleasant. Focusing on and building the actual product instead of having to maintain and spend resources on the infrastructure behind your business is far more effective than building everything from scratch. As long as we're not constrained by external factors, we'd most likely go with an off-the-shelf solution for our problem, and that's the smarter path down the road. I developed this mindset out of personal experience having gone the other way, and I would take an existing product over building it myself every time. That is if it wasn't for the money.
I spent a fair bit of time comparing existing analytics and metrics services and was surprised to find out that while having a managed service to store my metrics in would have been amazing and a no-brainer to pick, the price-to-performance ratio or "bang for the buck" on most products I checked out was mostly out of the range I wanted to pay for a personal endeavor. This is not meant to be criticism against any product, SaaS businesses depend on that recurring revenue after all, but it looked like I had to try something different.
For the reasons I described above, self-hosting software on "bare-metal" infrastructure is mostly discouraged by now. It's been a long time since I last set up a machine to be used for a real production environment if I can call personal use that. A primary incentive for going with real platforms like AWS, Google Cloud or whatever your personal favorite is, will be the endless integration possibilities across the group of services: Deploying a feature can be a matter of pushing your code and watching an automated system building and deploying it into a testing environment before pushing it out to the public, all in a matter of seconds. And even if setting this up means fighting with some YAML files, those workflows are much more pleasant than SSH'ing into your box, pulling the latest code changes and rebuilding your services, watching the downtime obliterate any other traffic you'd like to receive.
But it doesn't have to be like this, right? As developers, we can build tooling around this misery, and we did that for years up until the managed era came to be. So if it's just that, let's build some assistance for our problems!
🔠Building our way to greatness
The notable absence of CI/CD pipelines was one of the biggest pain points of going back to self-hosted, but thankfully it's not that hard to check for remote pushes and run a few commands when necessary, so that's pretty much how and why helferlein got to be. And as simple as it was to build, the immediate benefits it added empowered me to stop worrying about the simple things again and allowed me to build services to ingest and process incoming metrics, similar to the experience I'd have in other environments.
I didn't hesitate to go with InfluxDB, being the straightforward time series solution I've used for projects before, to store all important data in, deployed a Chronograf instance to build fancy dashboards around it and hooked up the ingestion service to InfluxDB. Most of the time spent was reading up on writing compose files, figuring out how to use multiple container networks and how to print colored text in bash. I'm still confident, that the latter effort was worth it!
The biggest learning point I got from this journey is that going back to bare-metal is one of the best options to build a cost-effective architecture and even if I wouldn't use it for real products, it's a highly-rewarding opportunity to get to tinker around with all components again. The best part of it all: I'll pay around $3 for a complete metrics setup that will scale, and in the unlikely case that I need to scale even further, it's not that much more expensive.
I hope you enjoyed catching up on my experiences setting up my own metrics. If you've got any questions, suggestions or any feedback in general, don't hesitate to slide into my DMs or send a mail 🚀