I’ll keep this short. I recently installed Ansible 2.0 to manage the Turtl servers. However, once I ran some of the roles I used in the old version, my handlers were not running.

For instance:

# roles/rethinkdb/tasks/main.yml

- name: copy rethinkdb monitrc file
  template: src=monit.j2 dest=/etc/monit.d/rethinkdb
  notify: restart monit

# roles/rethinkdb/handlers/main.yml

- include roles/monit/handlers/main.yml

# roles/monit/handlers/main.yml

- name: restart monit
  command: /etc/rc.d/rc.monit restart

Note that in Ansible <= 1.8, when the monitrc file gets copied over, it would run the restart monit handler. In 2.0, no such luck.

The fix

I found this github discussion which led to this google groups post which says to put this in ansible.cfg:

[defaults]
...
task_includes_static = yes
handler_includes_static = yes

This makes includes pre-process instead of being loaded dynamically. I don’t really know what that means but I do know it fixed the issue. It breaks looping, but I don’t even use any loops in ansible tasks, so

This will be short but sweet. When deploying some new servers today, I ran into a problem where no matter what, sudo bitched about syntax errors in my sudoers file. I tried a bunch of different options/whitespace tweaks/etc and nothing worked.

deploy ALL= NOPASSWD: monit restart my-app

Looks fine right? Nope.

Use absolute paths

This fixed it:

deploy ALL= NOPASSWD: /usr/bin/monit restart my-app

Everyone in the world's advice is to "just use visudo" but I couldn't find any info on what was actually causing the syntax error. Hopefully this helps a few lost souls.

So I got to thinking. There are some good caching reverse proxies out there, maybe it's time to check one out for beeets. Not that we get a ton of traffic or we really need one, but hey what if we get digged or something? Anyway, the setup now is not really what I call simple. HAproxy sits in front of NginX, which serves static content and sends PHP requests back to PHP-FPM. That's three steps to load a fucking page. Most sites use apache + mod_php (one step)! But I like to tinker, and I like to see requests/second double when I'm running ab on beeets.

So, I'd like to try something like Varnish (sorry, Squid) but that's adding one more step in between my requests and my content. Sure it would add a great speed boost, but it's another layer of complexity. Plus it's a whole nother service to ramp up on, which is fun but these days my time is limited. I did some research and found what I was looking for.

NginX has made me cream my pants every time I log onto the server since the day I installed it. It's fast, stable, fast, and amazing. Wow, I love it. Now I read that NginX can cache FastCGI requests based on response caching headers. So I set it up, modified the beeets api to send back some Cache-Control junk, and voilà...a %2800 speed boost on some of the more complicated functions in the API.

Here's the config I used:

# in http {}
fastcgi_cache_path /srv/tmp/cache/fastcgi_cache levels=1:2
                           keys_zone=php:16m
                           inactive=5m max_size=500m;

# after our normal fastcgi_* stuff in server {}
fastcgi_cache php;
fastcgi_cache_key $request_uri$request_body;
fastcgi_cache_valid any 1s;
fastcgi_pass_header Set-Cookie;
fastcgi_buffers 64 4k;

So we're giving it a 500mb cache. It says that any valid cache is saved for 1 second, but this gets overriden with the Cache-Control headers sent by PHP. I'm using $request_body in the cache key because in our API, the actual request is sent through like:

GET /events/tags/1 HTTP/1.1
Host: ...
{"page":1,"per_page":10}

The params are sent through the HTTP body even in a GET. Why? I spent a good amount of time trying to get the API to accept the params through the query string, but decided that adding $request_body to one line in an NginX config was easier that re-working the structure of the API. So far so good.

That's FastCGI acting as a reverse proxy cache. Ideally in our setup, HAproxy would be replaced by a reverse proxy cache like Varnish, and NginX would just stupidly forward requests to PHP like it was earlier today...but I like HAproxy. Having a health-checking load-balancer on every web server affords some interesting failover opportunities.

Anyway, hope this helps someone. NginX can be a caching reverse proxy. Maybe not the best, but sometimes, just sometimes,  simple > faster.

After being a customer for the Rackspace Cloud (formerly Mosso) for quite some time, I'm happy to say that my business and anyone who listens to our advice will never be using this hosting service, ever again.

Rackspace is an amazing company. They are know for having great servers, great support, great everything. You can't beat them. Mosso was a side project that was swallowed up by them which aims to run websites in a real, actual cloud. This is a valiant cause. To be able to upload a site to one server and have it scale infinitely over however many servers their datacenter has without ever having to touch it...that's a miracle. It's a great idea, that unfortunately just doesn't work.

Mosso has repeatedly let us down, again and again. Their service is always going down. It's hard to find a month where one of our sites hosted on the "cloud" hasn't seen at least an hour of down time. I'd expect this from a shoddy "HOST 100 SITES FOR $2.99/mo!!" host, but not from someone charging a base rate of $100/mo. Here's what it boils down to: you're paying Mosso a lot of money for the privilege of beta testing their cloud architecture. Great business model.

And while Rackspace is known for fanatical support, the Rackspace Cloud is known by us for support that is fanatical about ignoring or avoiding the issues plaguing them on a week-to-week basis. Questions go unanswered, support requests ignored, etc etc.

So all in all, it's been a terrible experience. And yes, we have been using them for more than a month...a little over a year now. Yes, we stuck it out and payed outlandish hosting rates for horrible service. Why? Because I really do wish it worked. I wish I could put a site on it and have it be up 100% of the time. That's the point of a cloud, no? To have >= 99.999% uptime? I really wish I could put a site on there and let it scale with demand as it grew without ever having to touch it - and I can do this - but the price is my site goes down for long periods of time at short intervals (oh, plus the $100/mo). We tried to give them the benefit of the doubt, and tried to believe them every time they told us that this was the last downtime they'd be having (yes, we heard it a lot). I just can't lie to myself any more though. Mosso sucks.

So please save yourself some time and realize that it's too good to be true. The Rackspace Cloud is the most real and cool cloud hosting you'll ever see, but as far as I'm concerned they are still alpha-testing it, and your site WILL go down. Want hosting that scales automatically, is zero customer maintenance, always up, and has amazing support? You won't find it anywhere.

Mosso comes close, but they just can't get it right. Save your money and learn how to scale on a good VPS provider.

Wow. You'd think it would be easy. In fact, it should have been. Compile a module, load it from apache. Recompile PHP with --enable-fastcgi...oh wait, I already had it in there (always thinking ahead!!). Change some apache settings.

Right? Yeah, right. It took two days. I can't even really remember why. The biggest problem was that running make && make install in the mod_fastcgi source was NOT yielding a 'mod_fastcgi.so' as the documentation PROMISED! In fact, it installed mod_fastcgi.la instead, a highly useless file.

So how did the master get out of this bind? Beats me, try asking him. As for me, I had to run 'ld -Bshareable *.o -o mod_fastcgi.so' which is mentioned in some document from a long time ago in a galaxy far, far away.

Let me interject and say that the information on the FastCGI website is "not very well documented."

Day 2. I figured, what's the point of FastCGI if it's not set up to connect to a remote App server? Maybe I don't HAVE an external server set up, but we can pretend. Well that's another nightmare. There's a good external FastCGI guide written about it, and guess what it worked. Not really a nightmare at all, come to think of it. Quite pleasant.

All in all, shouldn't have taken 2 days =P (I'm a tinkerer)...but fuck it, I have FastCGI now, ready to connect to all those App servers I have churning away in the background (one day).

In all the excitement, I also compiled and installed the apache worker-MPM. A few tests with ab didn't really show any noticeable difference. But threads are cool, right?

Next up: figure out how to configure Apache to pass all requests ending in .php (whether the file exists on the web server or not) to our "app" server. Is this possible?

So tonight I helped a client set up PHP5 on IIS 7 using MSSQL 2005. These things always work great in theory but judging my my use of the word "theory" in italics, you can probably guess that things weren't so smooth in practice.

The client was smart enough to get FastCGI working through IIS...something I would have probably rolled over on. From then on, it was an upward battle getting a simple PHP prototype project going.

In the later versions of PHP 5, it would seem that all mssql_* functions have...been... removed? There is an ntwdblib.dll that needs to be replaced to play nicely with mssql 2005...but it doesn't exist in the latest releases. How strange. I ended up reverting to 5.2.5, making me a not-so-bleeding-edge pushover :'(. It's cool though.

Then MSSQL doesn't accept normal logins, only windows ones, and it's bloomin' impossible finding out how to change that.

One thing Microsoft seems to have actually done right is release a rewrite module (much like mod_rewrite) that you don't have to frickin' pay for, which is nice. On a side note, I really hated Windows Server 2008. It's like Vista in every way, except that the design is slightly different, somewhat. Sorry, MS, but get your shit together plz, kkthxbai.

Anyway, we got everything going. What a pain in the ass though!

If you're wondering, I'm more of a Unix guy ;). And yes, I have used a computer before.