Update: Tailing the End of 2025

Hello again,

It has been quite some time now since I last have written here. We are in July; when I originally drafted this, it was June. We are already partially into Q3 now, whaaaaat?!! 😓

Something you might not know about me is that about 2 years or so ago I did decide that I would double down on a specific dream I have always had. I had a great time working for an MSP in the past and honestly felt like it was something I could setup and maintain and actually improve and provide value to the industry ultimately. Well the thing is it's a splendid little pipe dream, but it does take quite a bit to learn the ways and all that is necessary to run a MSP as a solo op, even when you have connections to world experts. I decided to set a date for Q1 2026 with the goal of being marketing phase ready by then.

Sometime before November as before mentioned in an earlier blog post I had learned the ways of ceph in my home production environment without proper backups. 🤦‍♀️ My initial configuration was working splendidly but the innovative mindset led to an attempt to refactor the networking configuration, a would be valid configuration was not possible to lack of implementation in the current state of ceph. The result of this was that I lost most if not all of my application data and configurations I had built up and migrated across many systems for around 10 years. There was much stuff I was able to salvage, all was not lost. But this is not what we should be focused on. This mass extinction event was part of a bigger lesson that I should have learned a long time ago and that I hope my reader will have learned without THE HARD WAY™️. This is why we spent a big deal of time earlier this year and recently learning the ways of configuring reliable backups!! We would like to "guarantee"🥴.. With a proper 3,2,1 strategy. Or DamnNearIt™. *knock on wood* This won't happen again! This is my first time in 7-10 years or so that I lost data. The last time I'd imagine was what did send me on a journey that led me to start maintaining a old full tower as a NAS storage box utilizing Ubuntu and ZFS sometime back in 2016-2017.

Well since this mass extinction event, we did recover much of what perished in the fires. We decided not to give up or let it be a discouragement. I pledged myself to raise out a refreshed and revitalized environment from this rubble akin to a phoenix. I embraced this event honestly. It was actually a conscious decision when I wiped my ceph disks and started over from scratch. In the back of my head I knew likely my data was okay if I was patient to recover it. I even hacked together an automated daemon that can manage the rebuild process of the Ceph monitor store for Proxmox.

At that given time, I did not know that Ceph does not have ipv4/6 dual stack support. I couldn't make the storage accessible, I was stubborn to have dual stack support, I did not revert the settings; I could not afford a test environment. But at the end of the day my final thought was that I did not have clients and my #1 focus should be learning and preparing to be a service to this industry, the scar of some limited data loss will be a good reminder for me going forward that proper backups are MANDATORY if I am to enter the professional space and things must be done by the book. That situation would be more critical in that space but in this context a bit of minor punishment is a tool! I will never forget.

I got distracted with rehashing my bad.

Since then we have refreshed things in the lab. We have configured backups for all hosts. Everything can be recovered. Wait! Actually I don't have android backups. Maybe that should be a project in the near future? But yes! We overhauled backups.

Now we have a whole monitoring host on an isolated VLAN keeping watch on all the logs and metrics utilizing the typical grafana, prometheus, loki trifecta. We also tossed out beszel and left it there while we learned those mentioned before.

We have crowdsec running on the firewalls as a bouncer and on the monitoring server as a LAPI server.

We have automations based off a codebase stored in our private Gitlab instance utilizing a gitlab runner.

We have our environment configuration in version control so all deployments, etc; are well documented and we have a history of changes and retain their last status in case of disaster.

We have SSO (Using Zitadel) for all of our services with a mix of native support and use of oauth2proxy.

We have reviewed a lot more software in the recent months for features useful to our community.

Thank you for being here. If you are reading this, you must be a certain type of individual. You must have a great deal of patience. I hope that we can learn and grow together in the future. I appreciate your support and I encourage you if you found anything interesting and want to make suggestions for something we should explore totally let me know. I am very impacted by my interactions within the communities I have become involved with over the years. I am very thankful for the help and community we have as tech hobbyists even if there can be drama, can't deny we are blessed. None of this stuff is any kind of niche thing like it seemed when I first was tinkering as an adolescent. There are 0 to hero courses on so many advanced topics at our fingertips and we have booming communities where other nerds gather and kick back for morale. We can accomplish so much together and are stronger in our communities, even if they can be distracting sometimes.

I want to mention a few of the projects that I have worked on in the past. Virtual Display Driver Wizard is overdue for an update. It has been since December that the developers of the driver itself released a Public Production/Stable Release. We would really like to see the configuration tool updated so our users can enjoy that experience if still desired or if any feature is not evident on the improved versions. We are behind on reviewing the current status of the project and using a legacy version at this time.

I want to remind you. I have been away for some time now from certain objectives that I still have. I had to divert my attention from things pending the November incident 😅. Like I said it has been a huge learning experience. I have been developing a lot as a technologist. We have intent to update past projects, to refresh old documentation, create new projects and provide crash courses for things we can share with each other. I have been putting this effort into myself first, I do plan to re-engage with those prior goals however and do feel much regret to the immediate impact we have all seen to these many other projects of mine that have suffered in these many months.

I have a few more goals for my knowledge in certain areas to reach a necessary level to accomplish larger more distant goals, but making considerable progress. Starting to learn the DevOps/GitOps mindset is a milestone for me and I feel a necessity for me to be able work more efficiently in realizing ideas I might have going forward. From November 2024 to now I have been very busy and put a lot of hours in.

I can't name any specific dates, currently I am actually on the verge of burn out from grinding for this progress. But we have reached a lot of milestones for our goals in the mission of building out this fresh base environment. I am probably not going to be so OCD that I must fully translate everything to IAC before I focus on other projects. But I do have to figure out what my plan is going forward financially as I am currently unemployed and Precision Plan IT needs more labor and love to be market ready. I am still at least hopeful we can start to look forward to seeing some external facing progress again in the near future. You haven't heard the last of me!!!!!!!!!!!!!!!!!!!

Love this community, I really want to contribute!

Thank you again for being here and reading this Q3 '25 So F Me Right update! Together we are a part of the greater future, regardless if we see it or not. I have bumped into so many exceedingly talented developers, and knowledgeable people who have shaped my journey. Maybe you feel like you are a "no one", reading this. I know I sure as hell do while I am writing it. But the truth is the connected nature of the internet and hobbyists, the small players have a chance to make a ripple. And with stuff like vibe coding and just the ridiculous level of availability of information, everyone can get involved how ever they see fit and its just good times. Not everyone has to be an engineer or a pioneer. A lot of us are just having fun.

Have a rockin awesome summer!

~ So F Me Right