Inspired by I-57

We use NoCat extensively, and we have peak usages of > 1100 users actively logged in at the same time, with another good 3000 or so people actively using wireless and holding a dhcp lease.

We recently started experiencing slowdowns, where it would take minutes for NoCat gateways to actually return a page (status, capture, actually authenticate and redirect, etc). This was caused by a simple mistake in their child reaping, documented below.

It’s important to note that in a captive portal solution, the gateway is MUCH busier than the number of authenticated users suggests; authenticated users usually don’t impact the cpu much (they’re just permit rules in the underlying firewall). It’s unauthenticated users who hammer the system. Every weather widget, RSS request, twitter status update, etc, etc that’s happening automatically on port 80 is being captured (silently) by the gateway. So we need to be sure our forking is smart.

The root cause of the slowdown was that it was reaping children using the wait() call. wait() is a blocking call. This means that the software, if it had any children forked out, would block until a child (any child) exited, and then processing would continue. The server would fork off enough threads to handle any children waiting, and then come back and block again for a child to exit. This is ok at low loads, but not good when you’re at high loads (or there’s a packet loss problem, common in wireless, causing a thread to run slow, etc).

The fix: Change wait() to waitpid(-1,NOHANG). Now, it’s non blocking. Performance is much, much better.

--- Gateway.pm  (revision 667)
+++ Gateway.pm  (working copy)
@@ -3,6 +3,7 @@
 use IO::Socket;
 use IO::Select;
 use IO::Pipe;
+use POSIX ":sys_wait_h";
 use NoCat qw( PERMIT DENY PUBLIC MEMBER OWNER LOGIN ANY );
 use vars qw( @ISA @REQUIRED @EXPORT_OK *FILE );
 use strict;
@@ -93,6 +94,7 @@
     my $self   = shift;
     my $kids   = 0;
     my $hup = 0;
+    my $childPid;
 
     return unless $self->bind_socket;
 
@@ -143,10 +145,10 @@
         }
 
        # See if any kids have expired, reap zombies
-       if ( $kids ) {
-           1 until ( wait == -1 );
-           $kids = 0;
-       }
+        do {
+            $childPid = waitpid(-1, WNOHANG);
+           $kids-- if $childPid>0;
+        } until $childPid<=0;
 
     } # loop forever
 }