What I’ve been doing for the last couple of months

Summary

The short answer is moving my web presence from a Dreamhost shared hosting account to a linode and a free wordpress.com account. This doesn’t sound exciting, but things do seem to be working very much better.

Why the move?

When I moved to dreamhost, I thought the service and support were very good. There were a couple of warning signs — I put up a badly designed mysql view and they moved it on me. But at that time, they emailed you when they did something like that. The email did suggest that if you really needed to do things like that you should be on a higher-priced account, but I managed to fix it without too many problems for anyone.

In any case, I got 2 websites and email and mailman mailing lists and guest accounts for a couple of friends for about $10/month. And I was able to mostly set this up from the dashboard, without doing a lot of apache and postfix configuration.

Problem

In the summer of 2014, the SerpentPublications.org wordpress site was hacked. Initially it was just hijacked to a site supporting some view of politics in the Middle East, but even when I thought I had put my own site back, something was wrong.

The first symptom I noticed was that whenever I’d done something to the site (like change the sidebar), I was unable to log in and both the serpentpublications site and the laymusic site would be unresponsive for half an hour or so.

Dreamhost support was initially unhelpful, but eventually someone told me that this was what happened when your site was consuming more resources than they wanted it to; it would then be throttled for a period. They made some suggestions for what to do about this (including of course to upgrade to a higher-priced plan). Eventually I figured out that if I used the CloudFlare account associated with the DreamHost account and told it that SerpentPublications.org was “under attack”, the throttling wouldn’t happen.

I also realized that I was getting a lot of page accesses of the non-existent page /wordpress/?wp-subscription-manager from a Polish site. When the throttling happened, these would go from a few dozen an hour to hundreds a minute. Blocking the IP addresses used for this didn’t seem to help.

There were major disadvantages to the CloudFlare “under attack” mode. I couldn’t use wget to download from the site; there was a several second delay on the first access of the site; and I got emails from disgruntled users whose browsers or ISPs weren’t letting them access the site correctly.

Solution

The Serpent Publications site is in fact not a very good fit for wordpress, so I looked around to see what else I could use. A widely recommended candidate was Drupal.

I will make another post about rewriting the site in Drupal, but the short version is that for how beautifully designed it is for what I want to do, it’s really badly maintained, and it took me several months to figure out workarounds for a number of widely reported and longstanding bugs.

I had decided that the DreamHost “support” wasn’t what I wanted to go on using, so I looked around for another host. I had an abortive adventure with Arvixe, which stayed up just long enough to let me get most things moved there, and then went down for about two weeks. I had been thinking this would just replace what I had from Dreamhost, but they have mail, database, and files on the same machine, and make no effort to move anything anywhere else if that machine goes down, nor did they seem to be making any effort to put up a new machine while they fixed the old one. So I stayed on DreamHost for a while.

Eventually I heard from enough people that they liked Linode virtual machines and that they were really reliable. I went to check my email one night and discovered that Serpent Publications was down, and I investigated and it turned out that they had decided that it was using too many resources of some kind, so they moved the whole directory to serpentpublications.org.moved or some such. Without telling me anything about what had happened or what they had done. So I set up a Linode that night and started putting the new Drupal site there.

Of course, this kind of solution means you have to do your own configuration of apache and mailman and postfix, so that’s why it’s been months not days. But the Drupal site is working quite well, and I think I’m consistently getting two or three times the traffic I was on the old site. And I have some things automated better than I did on the old site, and I have shell access to things like the mailman administrative commands that I didn’t on DreamHost.

Other pieces of the solution

WordPress.com

The LayMusic.org site is in fact what WordPress was designed for, and it was working quite well, but I decided maintaining WordPress wasn’t what I wanted to do, so I moved it to WordPress.com, which will give you a fairly big site for free with only a few ads. I was worried about my python program that lets me write posts in html in emacs, but it works fine. My old posts are a bit odd-looking because the hard newlines get interpreted as html breaks even when they’re in the context of an html paragraph, but I’ve figured out how to deal with that for new posts.

Mail

This has been pretty time-consuming. I had to configure postfix on the linode, because Mailman needed it, but it was enough of a pain to keep that working (I had already been muttering hard about why Hillary Clinton would run a mail server on purpose) that I didn’t really want to have to set up anything further for getting my own mail.

I first tried just having postfix forward all the laymusic.org mail to a gmail address, but that didn’t really give me what I wanted. For instance, their spam filtering is really good, but not impeccable. At first, I was checking my gmail spam filter every few days, but it’s a terrible interface even by webmail standards, so I figured out how to download the spam folder via imap so I could do it in gnus. This is easier, but it means that any training you do on the spam filter isn’t training the google filter.

But looking at the mail logs on the linode, I kept seeing messages from gmail like:

Our system has detected that 550-5.7.1 this message is likely unsolicited mail. To reduce the amount of spam 550-5.7.1 sent to Gmail, this message has been blocked.

and

Our system has detected an 421-4.7.0 unusual rate of unsolicited mail originating from your IP address. To 421-4.7.0 protect our users from spam, mail sent from your IP address has been 421-4.7.0 temporarily rate limited.

And I was missing some mail. I looked in the linode forums, and someone there recommended zoho.com as a free place to get pop or imap mail, so I’m trying that.

I just implemented this part of the solution yesterday, so I can’t tell how good a solution it is yet, but so far so good. I had some trouble figuring out their interface, but the support was pretty responsive.

Summary

Five stars for linode. There has been no down time since I set up the account two and a half months ago. My one interaction with their support was fairly trivial (I actually figured out the solution myself shortly after opening the ticket), but timely and pleasant. There was one episode of CPU usage going to 99% (it’s normally 1-4%) and I got an automated email warnign me about it, but being completely polite and non-threatening.

Four stars for WordPress.com. The transfer went extremely smoothly. My one interaction with support was quite successful. (Something had stopped working; they had changed something without updating the documentation.) The lost star is for the html rendering being odd. I should discuss this with their support.

Three stars for Drupal. I’ll discuss the problems in more detail in another post, but the bottom line is that although it’s a wonderful program in a lot of ways, you couldn’t really recommend it to anyone who didn’t have a lot of experience dealing with buggy software and documentation written by developers.

Three stars for Gmail. It mostly works, and their spam checker is better out-of-the-box than anything else I’ve ever seen, but if you can’t rely on their spam checking, you should use something else.

Two stars for DreamHost. I miss them running the mail server for me, and it’s likely that really doing shared hosting does cost more than $10/month. But I find their solution to that problem irritating. I don’t see why I should pay more than $10/month for a few hundred visitors a day, and they should figure out how to sell people what they can provide, and not pretend to be doing something for $10/month and then not do it.

One star for Arvixe. Apparently they used to be good, and then they got bought out by someone. Avoid. If you really need a host for testing and reliability isn’t an issue, they are pretty cheap, but don’t let them anywhere near your email.

This has really been a lot more work than I was expecting it to be; I hope this summary helps someone else.