Mongrel Rails "Application Error". Reaping threads. The Fix and a Hack.

Part of my reason for writing this is due to the frustration of having come on early this morning to find so many of our readers having sent e-mails telling me there was "Application errors" on the front-page of the site! The other part of my reason is to help people with similar situations work-around it. Some of you who use mongrel may have found that it's ease of use and setup far exceed its counterparts (lighttpd and apache), especially for rails applications. However, some of you may have also noticed a bumpy road when it comes to Applications errors. I can count on two hands the number of mongrel run sites i've witnessed with Application errors in the last two weeks. I wan't to explain what I've been finding in my logs and hope that this issue is looked into further. But first an explanation. Pre-rails conf, Zed Shaw had been receiving reports of problems with his mongrel software by users who had been experiencing too many open threads not being cleaned up. This problem seems to still exist, in 0.3.13.3, that latest released version and the version that we have been using here on the blog. For a very hacked solution, go to the very bottom of this post to see a way to avoid this "Application error" for most cases. For a proper solution, read on. In mongrel, you can control the number of processor threads by doing the following (ie. ActiveRecord database connections): h = Mongrel::HttpServer.new("0.0.0.0", "3000", 40) Which will make 40 thread processors although Zed Shaw claims 20 is the sweet spot for him. If you are running mongrel_rails from a non-custom ruby script, you'll want to likely expand the number of processors running. You can do so by changing --num-procs INT. ie: mongrel_rails -n 50 start From the help file: -n, --num-procs INT Number of processors active before clients denied As an example of the usage provided in his docs, he shows: The examples/simpletest.rb file has the following code as the simplest example:

require 'mongrel'

 class SimpleHandler < Mongrel::HttpHandler
    def process(request, response)
      response.start(200) do |head,out|
        head["Content-Type"] = "text/plain"
        out.write("hello!\n")
      end
    end
 end

 h = Mongrel::HttpServer.new("0.0.0.0", "3000")
 h.register("/test", SimpleHandler.new)
 h.register("/files", Mongrel::DirHandler.new("."))
 h.run.join

If you run this and access port 3000 with a browser it will say "hello!". If you access it with any url other than "/test" it will give a simple 404. Check out the Mongrel::Error404Handler for a basic way to give a more complex 404 message. Here are the error messages: Error sending file /Ruby_on_Rails_Blog/public/robots.txt: Too many open files - /Ruby_on_Rails_Blog/public/robots.txt Error sending file /Ruby_on_Rails_Blog/public/images/code_assist.png: Too many open files - /Ruby_on_Rails_Blog/public/images/code_assist.png Error sending file /Ruby_on_Rails_Blog/public/images/multi_language.png: Too many open files - /Ruby_on_Rails_Blog/public/images/multi_language.png Error sending file /Ruby_on_Rails_Blog/public/images/td_tower02.jpg: Too many open files - /Ruby_on_Rails_Blog/public/images/td_tower02.jpg Error sending file /Ruby_on_Rails_Blog/public/images/alertmax.jpg: Too many open files - /Ruby_on_Rails_Blog/public/images/alertmax.jpg Error sending file /Ruby_on_Rails_Blog/public/images/code_assist.png: Too many open files - /Ruby_on_Rails_Blog/public/images/code_assist.png Error sending file /Ruby_on_Rails_Blog/public/images/multi_language.png: Too many open files - /Ruby_on_Rails_Blog/public/images/multi_language.png Error sending file /Ruby_on_Rails_Blog/public/favicon.ico: Too many open files - /Ruby_on_Rails_Blog/public/favicon.ico Error sending file /Ruby_on_Rails_Blog/public/images/td_tower02.jpg: Too many open files - /Ruby_on_Rails_Blog/public/images/td_tower02.jpg Error sending file /Ruby_on_Rails_Blog/public/images/code_assist.png: Too many open files - /Ruby_on_Rails_Blog/public/images/code_assist.png Error sending file /Ruby_on_Rails_Blog/public/robots.txt: Too many open files - /Ruby_on_Rails_Blog/public/robots.txt Error sending file /Ruby_on_Rails_Blog/public/favicon.ico: Too many open files - /Ruby_on_Rails_Blog/public/favicon.ico --- And for those still reading, here is a very hackish way of solving this problem. Create a crontab restart script that restarts your site every 10 minutes. This is not the suggested method but it will prevent you from having a likelyhood of ever exceeding your 'too many open files' situation. As long as your thread processors are at a decent count and the problem was just occasional, this would restart mongrel and most likely prevent you from having it be a continual problem. This may however interrupt threads currently connected (streaming/downloading/etc) of files and so it remains a very dangerous solution to employ for certain types of sites. If you have a very static site though, it is not likely to be a major burden. In crontab (restarts every 10 minutes)

*/10 * * * * /home/user/myscript

and then the myscript file:

#!/bin/sh
cd /Ruby_On_Rails_Blog/
mongrel_rails restart