unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: "Mulvaney, Mike" <MMulvaney@bna.com>
To: "unicorn-public@bogomips.org" <unicorn-public@bogomips.org>
Subject: TeeInput leaks file handles/space
Date: Wed, 22 Apr 2015 16:42:51 +0000	[thread overview]
Message-ID: <CY1PR0301MB078011EB5A22B733EB222A45A4EE0@CY1PR0301MB0780.namprd03.prod.outlook.com> (raw)

When I upload large files to my unicorn process, TeeInput is streaming them through a @tmp instance variable which writes to /tmp/0.123456789 (some random number).  The file is immediately deleted but the file handle is not closed, so the files are not really deleted by the file system.

The files are eventually deleted when GC runs, but before GC runs they continue to take up space.  You can see this in lsof output, for example:

ruby       2783   webuser  23u      REG               0,17  6128086    3556917 /tmp/0.04249158625633187 (deleted)

This can cause problems if you have big files and a small /tmp, such as a tmpfs disk mounted in ram.  If someone sends in several 100MB files, you could easily get 2-3 open files for each of 6 unicorn processes, which would take up 1200MB of disk space until GC decides to run.

I looked into fixing this but it doesn't look easy.  I can reach into the TeeInput variable and close out the @tmp instance variable in my application, and that does fix the problem.  But obviously that is not a good solution.  I think there would have to be some kind of "close" method on http_request that would close out all the open resources such as these files.

-Mike

             reply	other threads:[~2015-04-22 16:42 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-04-22 16:42 Mulvaney, Mike [this message]
2015-04-22 18:38 ` TeeInput leaks file handles/space Eric Wong
2015-04-22 19:10   ` Mulvaney, Mike
2015-04-22 19:16     ` Eric Wong
2015-04-22 19:24       ` Mulvaney, Mike
2015-04-24  3:08         ` Eric Wong
2015-04-24 11:56           ` Mulvaney, Mike

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CY1PR0301MB078011EB5A22B733EB222A45A4EE0@CY1PR0301MB0780.namprd03.prod.outlook.com \
    --to=mmulvaney@bna.com \
    --cc=unicorn-public@bogomips.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).