March 13

Impending Server Changes

I’m going to be making some changes to the server.  I recently had my server “hacked”, which was basically just some skiddie finding a way to post one of those pharmaceutical spam ads on a couple of my WordPress sites.

Hacked - I do not think it means what you think it means.
Hacked – I do not think it means what you think it means.

I’m taking this time to figure out a couple of things.  The plan is:

  • figure out how to use Let’s Encrypt for HTTPS (for TTORP, the Story Teller Forum, and my Fitness Forum).
  • find a way to deploy + update WP sites via git (including initial setup)
  • work on a better CMS for Crazed(Sanity) sites
  • update my “deploy” system to work with GitHub and generic git (not just BitBucket.org)

That’s actually quite a bit of stuff.  It’s going to take a while to get this all setup.  It’s equally possible that I’ll post about impending downtime as I am to simply just do it: pretty much all my sites are (extremely) low traffic.  So, there, I said it.

UPDATE, April 13, 2017

I’ve started migrating some of the sites.  I’ve also started to get some SSL certificates, but ran into an issue with a “rate limit” on LetsEncrypt.org… which apparently means I have to wait for a week before getting more.  (This really means nothing right now, since none of my sites previously used SSL.)

Unfortunately, my main website (crazedsanity.com) is old and not really built to be moved around.  I’ve apparently relied on large amounts of server magic to make it work in the past, along with some not-so-easy-to-debug errors (like a script reporting that it doesn’t exist when accessed directly, because… reasons).

I’m hoping to have all the issues ironed out soon and finish the migration.  Then I can finally get on to playing with new tools and coding languages for reals.

Category: Code, PHP, Rant, Software Development, System Administration | Comments Off on Impending Server Changes
February 19

Beware the five o’clock stupids

My brain gets a little “squishy” after working on something for awhile.  Pounding away at the same thing gets a little monotonous.  Over.

And over.

And over.

And over.

And over.

And oevr.

And orev.

And voer.

Adn over.

And oevr.

And OVER.

AND OEVR.

AND OEVER.

ADN VEORE.

AND OEFER.

(see what I did there?)

Wait… what do you mean you can’t find /lib/std++.so?  WTF IS THAT?  Oh… shit…

At 5:00, relative to your timezone.  Stupid strikes.  BEWARE THE FIVE O’CLOCK STUPIDS.

February 17

SSH and Multiple Keys

I do a lot of stuff with SSH.  And I’m a fan of using separate keys for most things: the generic public key

~/.ssh/id_dsa.pub

for my specific workstation; and then a special, non-workstation-specific key for other stuff.

So I’ve got a key for things like:

  • Digital Ocean
  • SourceForge.net
  • BitBucket
  • GitHub

Frustratingly enough, once one has enough identities, one runs into a problem: Too many authentication failures.  Every time I see it, I have to go figure out why, and a workaround.  And that’s a couple of precious minutes that I’ll never get back.

Well, I’ve finally found a solution…  I actually already had it, I just forgot. My salvation lies in this file:

~/.ssh/config

The key is actually the first part, “Host *.hostname.com”, which is like a per-domain catch-all:

GSSAPIAuthentication no
Host *.hostname.com
    PubkeyAuthentication no
Host test
    Hostname www.test.foo
    User danf
    IdentityFile /home/danf/.ssh/digitalocean_id_rsa.pub
Host *.server04.com
    IdentityFile /home/danf/.ssh/source.server04.com_dsa.pub
Host cs
    Hostname indigo.crazedsanity.com
    User danf
    IdentityFile /home/danf/.ssh/digitalocean_id_rsa.pub
    IdentitiesOnly yes
    ServerAliveInterval 20

Oh, and this configuration allows me to use shortcuts that don’t have to resolve to real hosts and don’t require entries in my /etc/hosts file. So I can literally type ssh cs and get to where I want.

EDIT: apparently, there’s not a catch-all.  At least not a global one… but you can set one for an entire domain (like “*.hostname.com”) and that works magically.  So… close enough.  At least for government work.

Category: System Administration | Comments Off on SSH and Multiple Keys
January 30

The Death of Buzzkill

Did you try to go to BuzzKill.org, and now find yourself wondering how you got here?  Probably not, unless you’re a mindless bot.  But if you’re reading this (and are not said bot), you deserve an explanation.

BuzzKill.org isn’t dead.  It’s still around, but it redirects to here.  I got tired of maintaining it–or, more accurately, fixing it when I realized it was broken–so I just redirected it to this website.  If you want to know who I am, go here.

It really wasn’t doing anything anyway.  There was a gallery, which showed some crappy old images that me and a buddy created a couple decades ago, and showed blog posts from CrazedSanity.com.  So for now and the indeterminate future, you’ve landed yourself at it’s new home.

Oh, if you’ve got yourself a spiffy email @buzzkill.org, don’t worry.  That’ll work just like before. 🙂

Category: System Administration | Comments Off on The Death of Buzzkill
March 21

No Space Left On Device

I got this message while testing something on my web server. I was quite baffled to find out that there was, in fact, quite a lot of space remaining.

What Is Tail?

The “tail” program is a command line utility for Linux based machines.  To read the last few lines of a text file (such as a log), the command is “tail /path/to/file.log”.

However, the true power of the tail program comes when adding the “-f” argument.  Now, it will show the last few lines of the file, along with anything appended to the file.  Running “tail -f /var/log/nginx/error.log” shows me, in real time, the errors that are occurring on my server.

Searching For A Solution…

I read a few articles on the web about this problem, after I tried freeing up some space.

Ultimately, the problem was due to CrashPlan’s greediness (and a bug in the “tail” program).

How To Fix It… For Reals

Instead of configuring the system to have a higher value for “fs.inotify.max_user_watches”, just restart the crashplan service.  Or Dropbox, or whatever it is.  Just restart a service, then try running “tail -f” again (on virtually any file): if the error went away, then the culprit was that last service you restarted.

Category: System Administration | Comments Off on No Space Left On Device
February 24

Nginx and PHP as Different Users (Pooling)

So, after having installed a few WordPress sites on my server (namely this one), I ran into some permissions errors.  I couldn’t get plugins nor any updates to install.

The problem first appeared simply as a prompt to enter FTP credentials.  That was bizarre, so I hunted to find why that was.  I realized it was a permissions issue by reading this page (among others).

I scoured the Internet for answers (in other words, I tried a lot of different search terms in Google) for a way to make Nginx, the webserver software I used, to run as the proper user.  I’d setup different users for different websites, so just changing the webserver’s default user/group wasn’t the answer.

My first solution, albeit an ugly one, was to give everybody read+write+execute permissions on my WP folders.  That was an ugly kludge, but it worked.  And so it sat for some time.

Then I finally found how.  Through some bit of serendipity, I found an article on Apache and suExec.  I changed the term to Nginx with suExec, and found the answer… sort of.

Nginx, PHP-FPM, and Pooling

So the key was the “pooling” part of PHP-FPM that I’d basically ignored.  I had read the configuration file, but didn’t really understand it.

But after reading this article about pooling with Nginx and PHP-FPM, I found the answer.

So PHP-FPM can be configured to run different pools.  Basically, that means that there are multiple main processes for PHP, and they can run as different users.

Easy.  Added a new pool, changed it’s name and the user, and the new process (well, processes) appeared, with the correct user.  But how could I attach that to my website, so it ran as the correct user (instead of www-data)?

It’s All In The Socket

The bit of magic that makes Nginx hand off the PHP work to the correct pool is the socket.  The new pool needed to have a unique socket, then the affected websites needed to be reconfigured to use the socket corresponding to the appropriate pool.

I went back and changed my new pool to have a unique socket name, then restarted the php5-fpm process.  I then went and changed my website’s configuration file to use the corresponding socket.

Before restarting Nginx, I changed the permissions on my website’s folder to no longer be world readable/writable.  Then I attempted to delete an old plugin: as expected, I got a permissions error.  Restarted Nginx, then tried again, and it worked.   Woot!

Category: Nginx, PHP | Comments Off on Nginx and PHP as Different Users (Pooling)
February 17

Nerd Rage on MySQL and Postgres

MySQL vexes me SOOOOO MUCH.  Why not just use PostgreSQL?

I know, I know, you’re thinking, “I have no idea what you’re talking about.  I don’t even know how to pronounce those two things.”

So, MySQL can just be pronounced “my squeal,” and PostgreSQL can simply be pronounced “post gres.”  There, one part down.

Choosing a database is a developer thing, I guess.  But… WHY?  Here comes the classic car analogy.

On the left, we’ve got the MySQL coupe.

It’s kinda plain looking.  It has a history of doing unexpected things, like not stopping when you press the brakes, and continuing to accelerate when you release the gas.  It looks like any normal car, but there are some rather devious things under the hood, and is definitely not “standards compliant.”

On the right, we’ve got the PostgreSQL coupe.

It looks sleeker, more like a sports car than a run-of-the-mill coupe.  Even though it costs the same as the MySQL coupe.  It does everything you expect it to.  Everything about it is standards-compliant.

So, why choose MySQL at all?  Doing so has zero benefit.  Choosing PostgreSQL means it’s actually pretty easy to convert to MySQL later (right… “hey, I’m trading in my Lamborghini Diablo for a No-Name Turdmobile).

If you’re a developer, and you’re working with a database, just use PostgreSQL.  It will save you time in the end.  SERIOUSLY.

Category: Rant, Software Development, System Administration | Comments Off on Nerd Rage on MySQL and Postgres