New Zend Framework book

The other day I received a book — Zend Framework 1.8 – Web Application Development — for review. I’ll read through it as soon as I can and write a review that I will publish on my blog.

1:1 NAT with a Linksys WRT54GL router (with Tomato firmware)

For a long time I have used a Debian based machine called megatron as a gateway at home. Megatron had two NIC’s where one was connected to an SDSL modem, and the other was connected to a Linksys WRT54GL router (which is running the Tomato firmware). These two switched places a while back so that the router is connected to the modem, and megatron is behind the router. There are a couple of services running on megatron that needs to be accessible from the internets, so I had to do some iptables magic on the router to be able to do this. This post is more of a reminder to myself of how to do this, but there might be someone else out there who wants to do the exact same thing.

Earlier megatron had two official ip addresses (I have 5 from my ISP) on the NIC connected to the modem. One of them is used for SSL traffic to megatron and the other is used for everything else. The setup now is that megatron only has one NIC with two internal addresses: 192.168.1.10 and 192.168.1.11. My router has three addresses. Lets say these are: 193.n.n.122, 193.n.n.123 and 193.n.n.124. The first one is the one I will let the router have and the other two I will forward to megatron.

First I had to add two addresses to the router since it only had one. To do this I logged in the router using ssh and ran the following commands:

# Add ip addresses
ifconfig vlan1:1 193.n.n.123 netmask 255.255.255.248 broadcast 193.n.n.127
ifconfig vlan1:2 193.n.n.124 netmask 255.255.255.248 broadcast 193.n.n.127

To test if these two worked I simply pinged the new ip addresses.

Now I needed to tell the router to forward traffic on these two addresses to the ip’s specified on megatron. iptables to the rescue:

# To megatron
iptables -t nat -I PREROUTING -p all -d 193.n.n.123 -j DNAT --to-destination 192.168.1.10
iptables -t nat -I PREROUTING -p all -d 193.n.n.124 -j DNAT --to-destination 192.168.1.11

# From megatron
iptables -t nat -I POSTROUTING -p all -s 192.168.1.10 -j SNAT --to-source 193.n.n.123
iptables -t nat -I POSTROUTING -p all -s 192.168.1.11 -j SNAT --to-source 193.n.n.124

# Accept all ports
iptables -I FORWARD -p tcp -d 192.168.1.10 -j ACCEPT
iptables -I FORWARD -p tcp -d 192.168.1.11 -j ACCEPT

And that’s that really. One last thing I had to do was to make these changes permanent. This can be done by putting the ifconfig and iptables commands in this post in the Administration->Scripts part of the Tomato web-gui. Click on Administration and then Scripts in the gui and enter the commands in the firewall tab:

Remember to click the save button on the bottom of the page after these changes.

Firefox + Spotify in Wine

I run Ubuntu on all my workstations. I also run Spotify in Wine and use Firefox 3.0 as browser of choice. Until now I haven’t been able to click on the spotify:* links in Firefox and have it open them in Spotify. The following small changes made this possible:

Start Firefix and enter about:config in the address bar. Click past the warning that comes up, right click somewhere in the list of settings and click on new -> boolean. Create the following setting:

network.protocol-handler.expose.spotify

and set it to false.

Right click again to add another setting:

network.protocol-handler.external.spotify

Set this to true.

Now, lets create a small shell script that Firefox can use to start Spotify. I chose to place it in the Spotify installation folder:

vim ~/.wine/drive_c/Program\ Files/Spotify/spotify.sh

Enter the following:

#!/bin/bash
wine "$HOME/.wine/drive_c/Program Files/Spotify/spotify.exe" /uri "$1"

Then make the file executable with the following command:

chmod +x ~/.wine/drive_c/Program\ Files/Spotify/spotify.sh

Now, click on a spotify link (like this one for instance) and select the newly created shell script to see the magic happen!

Force Zend_Http_Client_Adapter_Test to fail

At work we have a component in our internal framework for fetching content from remote sources. The component utilizes something called availability caching which means that it will cache the remote content and use it when the remote server is down. We use Zend_Http_Client internally and to fully test our component we needed the remote server (which is not really a remote server at all in our test suite) to fail on demand.

In our tests we simply use the Zend_Http_Client_Adapter_Test adapter with the Zend_Http_Client instance instead of the default adapter. Using this we can tell the client what to return so we can make sure our own component behaves like it should. I implemented a method in the test adapter called setNextRequestWillFail() which takes a boolean flag as a parameter. The result when calling this method with boolean true is that the adapter will throw a Zend_Http_Client_Adapter_Exception exception on the next request. This allowed us to force the “remote server” to fail so we could test our availability cache.

The fix has been commited to Zend Framework trunk and will hopefully be added soon.

Small contribution to phploc

From http://github.com/sebastianbergmann/phploc/tree/master:

phploc is a tool for quickly measuring the size of a PHP project.

I just made a small contribution to it by adding counters for constants and class constants. My account over at github is located at http://github.com/christeredvartsen. Sebastian pushed it to the master repository about an hour ago.

I have also patched our phpUnderControl installation at work to output this information and contacted Manuel Pichler about it. More about that later!

RE: Solving problems

This morning I learned some more about the issues described in my previous post: Solving problems.

The BIG-IP load balancer we are using has a feature called OneConnect. This feature can increase network throughput by managing connections between the BIG-IP and the backend servers more efficiently.

After browsing the support pages over at f5.com I found a document that had an overview of OneConnect. The document states that the FastHTTP profile (a profile in the BIG-IP) uses an implementation of OneConnect that transforms Connection: close headers to Xonnection: Close. fritz (the Varnish that BIG-IP was using) was using the FastHTTP profile, and that is why it returned these Xonnection: close headers. I thought this was a bug, but it is actually a feature (it’s not a bug, it’s a feature).

I guess the cURL library used from PHP in the Dr. Front application on appfront does not understand what this header is and keeps a connection open to the BIG-IP. The connection between the BIG-IP and wildbill (the backend server that Dr. Front did some HTTP HEAD requests to) is probably closed so Dr. Front never receives the data it should. I thought cURL had some sort of timeout pr. default but it doesn’t seem like this is the case.

I took a quick peek at the part of the code in Dr. Front that does these HEAD requests:

// ...
$ch = curl_init();
 curl_setopt($ch, CURLOPT_URL, $url);
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
 curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
 curl_setopt($ch, CURLOPT_NOBODY, true);
 $data = curl_exec($ch);
// ...

There is nothing in that code that has anything to do with timeouts. cURL has an option called CURLOPT_TIMEOUT that can be used for this. One line of code should be added before the call to curl_exec:

curl_setopt($ch, CURLOPT_TIMEOUT, 10);

This line would then tell curl_exec that it had at least 10 seconds to do it’s magic. I have not tested this yet but once the developer of Dr. Front is back from his vacation I’ll get him to try this in combination with the Varnish that returns the Xonnection: close header.

Resources:

• SOL7208: Overview of the OneConnect profile (requires login)

Solving problems

Frontpage of vg.no

Edit: I have posted a follow-up to this post: RE: Solving problems

At work we have an application called Dr. Front to manage the front pages of different sections on our site. If you look at our main front page you can clearly see that it is not automated. This page is generated in Dr. Front and then moved to our content servers.

The machine that this application runs on has been acting very strange the last couple of weeks. Suddenly it’s using up all available memory and the machine starts to swap, and then it dies. Since I love solving problems I thought I could have a look at this to try and fix it. Keep in mind that I’m a developer and no sysadmin. My approach to solving this problem is probably not the best one, but at least I learned something along the way. To fully understand this post you should have some basic Linux knowledge. Hopefully you’ll be able to pick up a few tricks along the way.

I’ll throw in a small drawing of this part of our setup (don’t you dare to laugh of my dia skills!)

Network overview

When a user makes a request to for instance www.vg.no the request hits one of the two load balancers, then one of the Varnish machines (which contain rules that describes which request goes to which backend server) and then the backend server for the content you want. When one of the Apache servers request content from www.vg.no they also have to go through Varnish since www.vg.no/foo might hit a different backend server than www.vg.no/bar. So, any request to www.vg.no (or any of our other hosts for that matter) made from an Apache goes all the way out to one of the load balancers, and then back inside the network. When I started looking at this problem I was not aware that our setup was like this. The BIG-IP load balancer has just been been installed and the two Varnish machines has been split so each load balancer only talks to one Varnish. I thought we only had one load balancer that talked to both Varnishes. Let’s move on…

appfront is the machine that runs Dr. Front (written in PHP by Aptoma) and wildbill is the machine that contains the main front (among other things). When a front editor saves a front it gets moved over to wildbill from appfront. When you guys look at some page on http://www.vg.no/, wildbill is most likely the backend server you get content from. Now that you know some more about our setup, lets get on with hunting down the problem.

The very first thing I did was to log into appfront and look at top. I saw a bunch of httpd processes there so I decided to take a look at /server-status (Apache Module mod_status). This module gives you detailed information on the requests that is currently active. Keep in mind that appfront has ExtendedStatus On in its Apache configuration. Some information that I use in this post might not be available from /server-status if ExtendedStatus is Off.

Some of the processes that /server-status showed me had been running for hours. To check /server-status I simply ran the following command on appfront:

links -dump http://localhost/server-status|less

I used a one-liner that Audun at work gave me (and modified by yours truly to be even more magic) to sort the processes that was waiting:

links -dump-width 200 -dump http://localhost/server-status|grep -P '^([ ]+)([\d]+)-[\d]'|awk 'BEGIN {print "Seconds, PID, Method, Url\n--"}  $4 ~ /W/ {print $6, $2, $13, $14|"sort -n"}

This command dumps the content from /server-status using links (with a custom width to not make it wrap longer lines). The content is fed into grep that finds lines starting with some spaces, a number followed by a hyphen and then another number. These are the lines in the output that holds info on the processes. The filtered output gets fed to Awk which changes the output a bit so it’s more easy to read. The $4 ~ /W/ part of the Awk command makes sure that the 4th column in the input to Awk is equal to W. Read more about Apache’s /server-status to learn more about the output. The content is then sorted on the number of seconds each process has been running for before it’s displayed. Here is an example of the output:

Seconds, PID, Method, Url
--
0 27485 GET /server-status
668 27480 GET /drfront-1.3/
1784 925 GET /drfront-1.3/drfront/?do=login
2167 29437 POST /drfront-1.3/drfront/?do=keep-alive&fronttype_id=25
2197 29434 POST /drfront-1.3/drfront/?do=keep-alive&fronttype_id=25
2238 926 POST /drfront-1.3/drfront/?do=saveAndBurnFront&fronttype_id=25

Now I had the PID’s of a lot of processes. These PID’s can be used with strace to see what they are up to. strace can be used to trace system calls and signals. As root I did:

strace -p PID

where PID is the ID of the process that is hanging. First I took a peak at the one who had been hanging for the longest amount of time (PID 926). The output I got from strace was:

poll([{fd=25, events=POLLIN}], 1, 1000) = 0
poll([{fd=25, events=POLLIN}], 1, 0)    = 0
poll([{fd=25, events=POLLIN}], 1, 1000) = 0
poll([{fd=25, events=POLLIN}], 1, 0)    = 0

It just kept on going like that forever. poll is a C function that waits for an event on a file descriptor. This means that the process is waiting for an event to happen on some file. In the output from strace I saw {fd=25, events=POLLIN}. If we take a peek in the man page for poll we can see that the first argument is a struct and fd is the file descriptor and events is which events to wait for. The event in this case is POLLIN which simply means “There is data to read”. The process is simply waiting to read data from the file, which clearly never happens.

Now that I knew the id of the descriptor I could look in /proc on the file system to see which file it was. Before I did that I looked at the rest of the processes that was hanging. Next up was the process with PID 29434. The output this time was:

flock(18, LOCK_EX

The rest of the troubled processes gave this output. It seemed that the process hanging for the longest amount of time had an exclusive lock on a file, and since the process never died it never removed the lock. All other processes that also wanted this exclusive lock would wait for it forever. If we look at the man page for flock we can see that the first argument is a file descriptor. Since I knew the PID and the id of the file descriptor I did:

ls -la /proc/29434/fd/18

The result was

/tmp/drfront_sessions/<session file>

This meant that the session file of a specific user was locked forever. Any request this user made after the first process hung would result in a system call that would wait forever. The process that hung first was requested using AJAX. The logged in user did not see this and made a request to some other page in Dr. Front. This request would then hang forever since it didn’t get the exclusive lock on the session file. The user then had to login once more to get a new session id. The problem with this was that the processes that sat there waiting for the file to get unlocked took up a lot of memory. After a while there was no more available memory and the machine stopped responding.

Let’s go back to the first process and find out some more about the file descriptor it wants data from.

ls -la /proc/926/fd/25

The output this time was

lrwx------ 1 root root 64 Jul 22 12:21 /proc/926/fd/25 -> socket:[21172073]

This meant that the file descriptor was in fact a socket. I actually have no idea what the number in the brackets mean, but I figured out that this number could be used to find out some more about the socket connection using netstat. I ran the following command on appfront:

netstat -ae | grep 21172073

The output I got was:

tcp        0      0 appfront.m323.vgnett.:37333 www.vg.no:http           ESTABLISHED apache     21172073

This meant that Dr. Front made an outgoing connection to www.vg.no on port 80. I don’t know how all the bits and pieces are connected in Dr. Front so I didn’t know why it was doing outgoing connections at all. I fired up tshark to take a look at these connections:

tshark -N ntC -d tcp.port==80,http src 10.0.0.106 and dst port 80 and dst host www.vg.no

This gave me all connections made from 10.0.0.106 (appfront) to the www.vg.no host on port 80 (http). Here is a small part of the output from tshark:

0.106286 appfront.m323.vgnett.no -> www.vg.no    HTTP HEAD /drfront/images/front-25-image-c801b964.jpeg?2203138e92f24003197c4af8f7185aa8 HTTP/1.1
0.107729 appfront.m323.vgnett.no -> www.vg.no    HTTP HEAD /drfront/images/front-25-image-dddc9b55.jpeg?0d6d9fce05cf80b34d00c518ce11c8f8 HTTP/1.1

There was a whole lot of HTTP HEAD going on. According to the developer that wrote Dr. Front, these outgoing requests are made using cURL in PHP. Why would this cause so much trouble? I decided to take a look at the output from these requests on my own computer. I ran the following command on my laptop:

HEAD http://www.vg.no/drfront/images/front-25-image-c801b964.jpeg?2203138e92f24003197c4af8f7185aa8

First I hit joanie (Varnish) that used wildbill (Apache) as backend. Everything looked fine:

200 OK
Cache-Control: must-revalidate
Connection: close
...
X-VG-WebCache: joanie
X-VG-WebServer: wildbill

I tried some more to see if I could hit fritz. When it finally hit fritz I could see a small difference in the output from HEAD:

200 OK
Cache-Control: must-revalidate
...
X-VG-WebCache: fritz
X-VG-WebServer: wildbill
Xonnection: close

As you can see the Connection: close header that joanie produced is now Xonnection: close. This was strange because fritz and joanie use the exact same configuration and they are the same version of Varnish. Was it the load balancers that did some funky stuff? At this point I was not aware that each load balancer only used one Varnish each. One of the sysadmins (who is on vacation) told me that one of the load balancers had just been installed, and that they each had their own Varnish. Since fritz was the only one that responded with the bogus header I logged into fritz and ran the HEAD command locally to skip the BIG-IP load balancer:

HEAD -H "Host:www.vg.no" http://localhost/drfront/images/front-25-image-c801b964.jpeg?2203138e92f24003197c4af8f7185aa8

This time I got the correct Connection: close header. It looked like the new load balancer was doing something very weird.

I’m not sure why an Xonnection: close header would cause cURL in PHP to hang forever but my guess is that when cURL does not get a Connection: close header it keeps the connection open. Another thing I haven’t figured out is why appfront waits forever for some data to read on some sockets. It might be that wildbill decides to drop the connection for some reason, and since appfront never gets a Connection: close it just keeps on waiting for data on the socket that wildbill has already closed.

To fix this problem temporarily I edited the /etc/hosts file on appfront so that whenever it makes a request to www.vg.no it will only hit joanie. It has been running for some time now without any processes hanging, so hopefully I have managed to figure out the source of the problem was.

I have not yet figured out why a request through BIG-IP results in an Xonnection: close header though. It might be a software bug or simply a spelling error in a config file. The problem is not really solved until both load balancers respond with the correct header, so I’m not done yet!

The process of finding this problem was new to me and I feel that I picked up some neat tricks along the way. If any of my readers know of a better way to hunt down these kind of problems, or could shed some more light on the problems I found feel free to leave a comment!

Some good resources I used along the way:

• mod_status – http://httpd.apache.org/docs/2.2/mod/mod_status.html
• An Awk Primer – http://www.vectorsite.net/tsawk.html
• tshark man page – http://www.wireshark.org/docs/man-pages/tshark.html

Who implemented this method?!

It’s been a while since I wrote anything on this blog, so I thought it was about time to get back on the scene! Today I’ll write a post about how to figure out which class actually implemented a method in PHP.

Consider the following classes:

<?php
abstract class My_Plugin_Abstract {
    public function onLoad() {}
    public function onConnect() {}
    public function onTick() {}
    public function onExit() {}
}

class My_Plugin_Something extends My_Plugin_Abstract {
    public function onLoad() {
        // Do something magic on load
    }
}

class My_Plugin_Magic extends My_Plugin_Abstract {
    public function onTick() {
        // Do some magic on every tick
    }
}

class My_Plugin_End extends My_Plugin_Abstract {
    public function onExit() {
        // Do something fun on exit
    }
}

As you can see we have an abstract class with some non-abstract methods. The plugins can then choose which of the methods to implement.

Now, let’s imagine that we have an application that runs in a loop. In each iteration of the loop we will fetch an event that the plugins can hook onto. As you can see from the code above the plugins only implement one method each, and not all four that the abstract plugin has. Since we might only want to run the methods that is actually implemented in a plugin we need to figure out which class that implemented it; the plugin itself or the abstract plugin.

PHP has a couple of functions that does something similar: method_exists() and get_class_methods(). The problem is that whenever a class extends another one, all methods of the parent class is available in the child class (with some exceptions), so method_exists() for instance will return boolean true even if the child did not implement the method.

To be able to do what we want we have to use the reflection API. By using this we can “reverse-engineer” our classes and find out some more about which methods to execute.

Back to our application.

<?php
// Register plugins and use the class names as keys
$plugins = array();
$plugins['My_Plugin_Something'] = new My_Plugin_Something();
$plugins['My_Plugin_Magic'] = new My_Plugin_Magic();
$plugins['My_Plugin_End'] = new My_Plugin_End();

// Main loop
while (true) {
    // Fetch an event
    $event = magicFunctionThatReturnsAnEvent(); // Imagine this returns "Load", "Connect", "Tick" or "Exit"

    // Method to execute
    $method = 'on' . $event;

    // Loop through the plugins and run the current method
    foreach ($plugins as $className => $plugin) {
        $plugin->$method();
    }
}

Imagine that the loop runs 4 times, and each time we get a different event. This means that the following function calls have been generated:

My_Plugin_Something::onLoad
My_Plugin_Magic::onLoad // This is implemented
My_Plugin_End::onLoad
My_Plugin_Something::onConnect
My_Plugin_Magic::onConnect
My_Plugin_End::onConnect
My_Plugin_Something::onTick // This is implemented
My_Plugin_Magic::onTick
My_Plugin_End::onTick
My_Plugin_Something::onExit
My_Plugin_Magic::onExit // This is implemented
My_Plugin_End::onExit

Out of 12 function calls, only three are of use. If we extend the foreach loop a little, we can do something like this:

foreach ($plugins as $className => $plugin) {
    $ref = new ReflectionMethod($className, $method);
    if ($ref->getDeclaringClass()->name === $className) {
        $plugin->$method();
    }
}

First we make a ReflectionMethod object by specifying the name of the plugin class and then the method. The reflection object gives us loads of methods to use, but the one we are interested in is the one called getDeclaringClass() which gives us a ReflectionClass object that is a reflection of the class that declared the method. In most cases above the declaring class is My_Plugin_Abstract which only holds placeholders for the methods that the plugins can implement.

If we run our application again the following methods are run:

My_Plugin_Magic::onLoad // This is implemented
My_Plugin_Something::onTick // This is implemented
My_Plugin_Magic::onExit // This is implemented

Looks good! But (there had to be one right?), using the reflection API costs quite a bit. In most cases it’s faster just to make all the function calls above instead of using reflection to check the declaring class and so forth. In some cases it might be worth using reflection for this, and if you have such a case at least you know how to do it. You can thank me later!

Problems with svndumpfilter

As many others I have had some problems with svndumpfilter in the last couple of days. Yesterday I decided to move yet another directory from our old subversion repository to a fresh repository of its own. As I have mentioned earlier I simply tried to issue the following command:

svnadmin dump /path/to/repos | svndumpfilter include rss > rss.dump

This failed with the dreaded svndumpfilter: Invalid copy source path error message. After asking my friend google I found a replacement for svndumpfilter made by my hero of the day Simon Tatham.

After issuing the following command:

svnadmin dump /path/to/repos | ./svndumpfilter2 /path/to/repos rss > rss.dump

everything seemed to be working as expected.

More unit testing

On the way to work today I wrote some more unit tests for our internal framework. This time it was for a component that we use to fetch remote content. The component is called VG_Wget and one of its features is something called availability caching. What that does is that it caches the content of an URL in case it is unavailable the next time the content is fetched. There is a “regular” caching layer as well that can be enabled but the availability cache is only used when the external resource is unavailable.

I came up with a nice solution on how to test this, and since I’m such a nice guy I’ll share it with the lot of you.

Here is how we use the component:

<?php
require_once 'VG/Wget.php';
$wget = new VG_Wget();
$content = $wget->get('http://localhost/file.php'');

Since I know the contents of file.php I can easily check that the get method works as expected. But how can I make a file unavailable on the next request automagically? I’m glad you asked!

First I wrote a script that dynamically creates another php script, which in turn create some output, and then deletes itself.

The create.php script looks like this:

<?php
// Generate a filename and create the file
$filename = microtime(true) . '.php';
$fp = fopen($filename, 'w');

// String to write to the file
$contents = '<?php print(\'please cache me\'); unlink(__FILE__);';
fwrite($fp, $contents);

// Close the file
fclose($fp);

// Output the filename of the tmp file
print($filename);

First we generate a unique filename, and then create a file with the following content:

<?php
print('please cache me');
unlink(__FILE__);

After the file is created we output the unique filename so we know which file to fetch when testing the availability cache mechanism. When we access that file in a test the file will output “please cache me” and then self destruct. The next time we access that file the VG_Wget component will receive an HTTP 404, and the availability cache will be used instead.

The test method looks something like this:

<?php
class VG_Wget_WgetTest extends PHPUnit_Framework_TestCase {
    /**
     * Wget object
     *
     * @var VG_Wget
     */
    protected $wget = null;

    /**
     * Setup method
     */
    public function setUp() {
        $this->wget = new VG_Wget();
    }

    /**
     * Teardown method
     */
    public function tearDown() {
        $this->wget = null;
    }

    /**
     * Test method
     */
    public function testGetFileUsingAvailabilityCache() {
        // Base url
        $url = 'http://localhost/';

        // Call an url that creates a tmp file that is deleted after first request
        $tmpFilename = trim(file_get_contents($url . 'create.php'));

        $url .= $tmpFilename;

        // Fetch the file effectively deleting it so that the next request fails
        $content = trim($this->wget->get($url));

        // Fetch it again (this time the cache will be used)
        $content2 = trim($this->wget->get($url));

        $this->assertSame('please cache me', $content);
        $this->assertSame($content, $content2);
    }
}

I have removed some logic from the test method but you can see roughly how I test the component. Comments anyone?