After decommissioning the vmst.io WriteFreely instance last week, I moved my blog back to Ghost. But then this weekend I remembered what an absolute shit experience writing or editing on a mobile device is with Ghost. A few people mentioned that they just don’t blog from mobile, but I like to be able to do all my work from as many devices as possible (I push vmst.io updates from my phone just because I can.)

There are still no native Ghost apps, first or third party, and the web interface is terrible on a small screen. Autocomplete doesn’t work, which means capitalization, punctuation and spelling is especially awful even for me.

I have been listening to the Core Intuition podcast a lot recently, and decided to give Micro.blog and MarsEdit (on Mac) a try. Combined with the recently refreshed Micro.blog iOS app, I think it’s what I need. But what really sold me was the nearly flawless migration from Ghost using the json export file. After finding a nice theme, I flipped the DNS CNAME over.

I’m still playing with the automatic posting to my Mastodon account, as well as how the micro.blog site will play with ActivityPub. This post will help me see that better 😏

I went to an ENT a few weeks ago because I have been getting tonsil stones for a few years now, especially bad in the winter months. He said “yep your tonsils need to come out, this is going to be terrible, plan to be out for at least a week” 😯

That adventure is now scheduled for the week of June 26.

While I’m there I also mentioned that I snore really badly at night. It’s bothered my wife for years and more recently my children have joined in on making it clear they hear it all night too. He ordered an at home sleep study which consisted on wearing a magic Bluetooth ring on my finger for two nights. 😴

I was skeptical that they’d get good results but the doctors office called with the results today, and apparently I stop breathing while sleeping up to 30 times an hour. 😧

So I’m getting a CPAP. 😤

Now at the old age of 39, I have two chronic health conditions, with anxiety/depression being the other.

On Friday, the vmst.io team made the decision today to sunset our WriteFreely offering for vmst.io members. Since launching in January we've only had 23 users create blogs, of which only 12 users ever generated any content, many of which were a single post. In total we had about two users aside from myself that used it with any frequency.

My usage was mostly to justify it's continued existance.

Because WriteFreely requires a MySQL database backend (all our other services use Postgres) the costs of keeping the service operational outweighed the benefits to our members.

I will be moving some of the content that I generated there, here.

Elan Hasson asked in a Matrix chat group of Mastodon admins how he should go back and watch classic Star Trek. It got me thinking about a list of episodes to help jumpstart your knowledge without having to watch every Original Series episode (of which some haven't aged super well, or are confusing to new viewers in the context of modern Trek.)

Here's the list that I came up with:

• The Naked Time: Season 1, Episode 4
• Mudd's Women: Season 1, Episode 6
• The Corbomite Maneuver: Season 1, Episode 10
• The Menagerie: Season 1, Episode 11 (Part 1) and Episode 12 (Part 2)
• Balance of Terror: Season 1, Episode 14
• Arena: Season 1, Episode 18
• Tomorrow is Yesterday: Season 1, Episode 19
• Space Seed: Season 1, Episode 22
• Errand of Mercy: Season 1, Episode 26
• City on the Edge of Forever: Season 1, Episode 28
• Amok Time: Season 2, Episode 1- Mirror, Mirror: Season 2, Episode 4
• The Doomsday Machine: Season 2, Episode 6
• The Trouble with Tribbles: Season 2, Episode 15
• The Ultimate Computer: Season 2, Episode 24
• Assignment: Earth: Season 2, Episode 26
• The Enterprise Incident: Season 3, Episode 2
• The Tholian Web: Season 3, Episode 9
• Let That Be Your Last Battlefield: Season 3, Episode 15
• Turnabout Intruder: Season 3, Episode 24

I feel like that would be the core of some really good episodes, episodes that help round out some of what you see happen later in The Next Generation, Strange New Worlds, Discovery, and even Picard. As well as some just fantastic science-fiction stories, like City of the Edge of Forever.

We’ve been using Netlify for the docs.vmst.io site for a while now, but it wasn’t until recently I realized it could be used for other things that we host.

I’ve now successfully moved:

• element.vmst.io, our Element Web portal for accessing Matrix.
• elk.vmst.io, our Elk alternative web client for Mastodon.
• semaphore.vmst.io, our Semaphore alternative web client for Mastodon.

This means fewer Docker containers and custom build operations are needed to maintain these deployments. Any new releases of these platforms can be deployed automatically in a few minutes and new deployments can be tested easier.

This morning I removed just over 3000 dead instance connections from vmst.io.

I know Mastodonizens don’t follow their follower accounts as closely as some folks on other networks, but I wanted to let you know you this operation might have caused some go missing in this purge.

These are instances that have quit responding for at least seven days, and in some cases they haven’t been around since November. A couple weeks ago I saw an uptick in the amount of SSL errors in Sidekiq communicating with various remote instances and noticed a trend of instances that were stood up around 90 days ago, with Let’s Encrypt certificates, that have since been abandoned.

It's likely that folks were trying to self host and then realized what a pain in the ass it is 😉 As @supernovae@universeodon.com just put it to me in the project Discord, "lots of failed dreams in there."

I’d say that most of the domains listed didn’t have anyone following or being followed by vmst.io users, but some did. These accounts would not have been in communication with us for some time, as Mastodon stops actively trying to communicate with systems after they have failed for seven days.

While some of these were removed manually if they were larger instances, this process was largely accomplished by doing a dump of the unavailable_domains table in the Mastodon Postgres database, and exporting the list to a giant string of domains.

RAILS_ENV=production ./bin/tootctl domains purge away.we.go not.here.anymore bye.bye.birdie

In my spot checking of these instances, most of them had no direct follow/follower relationship with our users, but some of them did. Going forward this process will be run more frequently.

I've doing some blue/green (maybe it's just A/B) testing on vmst.io this week with server updates and Mastodon dependency testing. Specifically focused on some upgraded versions of Nginx and Puma (which are now fully live for everyone) and some backend libraries for the streaming API to things like Redis and Postgres.

I've been rolling out more updates this way, since things are pretty happy and functional with multiple systems running each component. I haven't decided what the right period of time to let things bake-in is, but I suppose that depends on what is changed.

The streaming updates are only on half the things right now. So if something either works better or fails spectacularly 50% of the time, that’s why.

I hate running old code. I've been running Debian 11 stable for a long time, but some of the packages that I end up using are past where they are on other distributions, so I recently flipped to the testing branch. So far I've not run into any issues other than needing to redownload/update some of the Ruby Gems used by Mastodon.

I'm also currently testing HAProxy as a replacement for Stunnel in our Mastodon Redis configuration. So far so good. I've noticed some activity in the Mastodon GitHub focused on a replacement for the current Redis libraries that don't work with TLS connections. Being able to completely remove Stunnel/HAProxy from the setup would be a fantastic simplification for me.

I've been doing things with computers, since last October.

In addition to launching a Mastodon instance at vmst.io I've since launched an integrated WriteFreely platform at write.vmst.io. As a result I will be blogging there regularly from now on.

The vmstan.com site will remain here for the time being, until I can maybe migrate the content into WriteFreely.

• https://write.vmst.io/vmstan
• https://vmst.io/@vmstan
• https://whois.vmstan.com

One of my core beliefs about Mastodon and the Fediverse is that there needs to be transparency in the operations of the various projects and instances.

This includes transparency in both the financial and technical operations of the instance. For the financial piece, we have a single source of truth on our Funding page that has total income from memberships as well as monthly/yearly expectations of expenses. We'll be providing a more granular update here soon that shows exactly where we are in terms of free cash flow. (Preview: we're solid for a bit based just on membership rates + one time donors to date.)

Another Server Post?

I don't intend to post about our infrastructure changes on a monthly basis, it just seems like I've done nothing by rearrange things on an almost daily basis, so I feel it important to discuss them. Every time I've posted something like this, I've had some kind of feedback from other #mastoadmins or even members with suggestions of ways to do this better. I'm not saying that things are settled, but I feel like we're in a good place right now.

Last time I provided an update was December 15, 2022. Just before Elon Musk did something else really stupid, I don't even remember what it was at this point. In the course of a few hours the headcount of the instance doubled.

Sometimes the only ways to test a system are by putting it under stress. Since that last update it became obvious that the way I had Sidekiq laid out wasn't optimal. Last month I had the front end nodes (Kirk and Spock) doing both Web services, and Sidekiq ingress queues. (Ingress is responsible for handling the influx of data from other instances, so anytime someone else posts an update and they let us know, we have to go fetch that data and any attachments.) One day I woke up and the site was down because someone somewhere posted a video and the front end nodes were being crushed by ffmpeg processes.

Not ideal.

Changes Since December

Scotty is now responsible for handling the bulk of Sidekiq activity, including the Ingress queues. The front end nodes now only handle the primary Mastodon Web functions.

Uhura is now responsible for handling the Mastodon Streaming API, as well as acting as a HTTPS proxy for other backend services like our internal metrics and monitoring. One deviation from Kirk and Spock that handle web traffic, is that Uhura is running Caddy as a web proxy instead of nginx. This is somewhat experimental but has not presented any known issues. Uhura now also hosts our status.vmst.io system powered by a self-hosted instance of Uptime Kuma. This replaces the services of Uptime Robot.

Exec no longer hosts Elastic Search, which has been moved to the free tier of AWS and replaced with Open Search. Exec runs smaller versions of Scotty's Sidekiq queues plus the scheduler queue, but is primarily responsible for the management, backups and automation of the entire Mastodon system.

Kirk and Spock are now running fully updated versions of nginx, with restrictions on TLS versions (1.2 or higher now required) and ciphers. As a result support for older browsers (IE8) or systems (Java 7) have been dropped. This is unlikely to actually effect anyone negatively.

I have also moved this site back to the Digital Ocean static site generator. I was running there previously, then moved to Netlify as a test. While there was nothing wrong with Netlify it just didn't make sense having things in another control panel with no added benefit.

The backup processes have been streamlined:

• I have dropped the trial of SimpleBackups that I was running as it was too expensive for what it provided.
• For the backup of Postgres we use pg_dump and redis-cli for Redis.
• The native b2-cli utility is then used to make a copy to a Backblaze B2 bucket.
• The CDN/media data is sync'd directly to Backblaze B2 via rclone.
• This is done using some custom script that process each task and then fire off notifications to our backend Slack.
• Backups run every 6 hours. Database backups are retained for 14 days, currently. This may be adjusted for size/cost considerations down the road.
• All backups are encrypted both in transit and at rest.

Previously a lot of the automation functions for things like firing off notifications to the Mod Squad in Slack about reported posts, new users, or various server activities were done using Zapier.

Recently I started working with n8n as a self-hosted alternative and have moved almost all of our processes there and expanded out to many more. Zapier is still used to notify and record donations via Patreon and Ko-fi because they have some native integrations that have proven troublesome to recreate in n8n, but the intent is to move away as soon as possible.

Mastodon Software Stack

One thing that I wanted to touch on before I close this out, is the idea of Mastodon forks (such as Glitch or Treehouse) or doing other local modifications to the Mastodon stack.

Our goal is to run the latest stable version of the Mastodon experience as soon as it's available, with a goal of being live here within 72 hours of being published. If there are security related updates we intend to take those on even quicker to protect our infrastructure, users and your data. In order to do this we run unmodified versions of the Mastodon code found on GitHub, specifically consumed via Docker using the images provided by Mastodon to Docker Hub.

In order to mitigate an issue with the Streaming API and backend PostgresSQL database, we've had to do one modification to the streaming/index.js that disables an self-signed SSL check by editing two lines in the code. Because of this, that component is not running via Docker but instead running as a native Linux systemd service. Once this workaround is not necessary in the upstream Mastodon code, we will revert to the mainline Docker image.

Other than those changes necessary for the functionality of our system, we do not intend to modify or customize Mastodon code in any other way that changes the user experience.