From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: * X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=1.0 required=3.0 tests=AWL,HK_RANDOM_FROM, MSGID_FROM_MTA_HEADER shortcircuit=no autolearn=no version=3.3.2 Path: news.gmane.org!not-for-mail From: Eric Wong Newsgroups: gmane.comp.lang.ruby.rainbows.general Subject: Re: streaming input for large requests Date: Tue, 10 Aug 2010 16:25:27 -0700 Message-ID: <20100810232527.GA14486@dcvr.yhbt.net> References: <1281478992.8692.15.camel@dogen> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1281483035 665 80.91.229.12 (10 Aug 2010 23:30:35 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Tue, 10 Aug 2010 23:30:35 +0000 (UTC) To: Rainbows! list Original-X-From: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Wed Aug 11 01:30:31 2010 Return-path: Envelope-to: gclrrg-rainbows-talk@m.gmane.org X-Original-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Delivered-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Content-Disposition: inline In-Reply-To: <1281478992.8692.15.camel@dogen> User-Agent: Mutt/1.5.18 (2008-05-17) X-BeenThere: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Errors-To: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Xref: news.gmane.org gmane.comp.lang.ruby.rainbows.general:117 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1OiyH8-00059M-4D for gclrrg-rainbows-talk@m.gmane.org; Wed, 11 Aug 2010 01:30:30 +0200 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 461B81858375; Tue, 10 Aug 2010 19:30:29 -0400 (EDT) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by rubyforge.org (Postfix) with ESMTP id DDED41858366 for ; Tue, 10 Aug 2010 19:25:28 -0400 (EDT) Received: from localhost (unknown [127.0.2.5]) by dcvr.yhbt.net (Postfix) with ESMTP id 1AD3D1F769; Tue, 10 Aug 2010 23:25:28 +0000 (UTC) John Leach wrote: > Hi, > > I'm looking to be able to get access to the request body as it is > available on the socket, so I can process uploads on the fly, as they > stream in. Hi John, Cool! If you need some example code, you should check out upr, http://upr.bogomips.org/ Sadly the demo machine is down, but the one application I helped somebody write (on a private LAN somewhere) still works well :) There's also the test/examples in the Rainbows! source tree: t/sha1*.ru t/content-md5.ru > But to be rewindable, I'm assuming they're being stored somewhere? I'd > like to be able to handle huge request bodies bit by bit without having > them written to disk (or worse, stored in ram!). Is there some way to > do this? Yes, we store uploads to an unlinked temporary file if the body is larger than 112 Kbytes (this threshold was established by Mongrel back in the day). Rack currently requires rewindability, but this requirement will most likely be optional in Rack 2.x, and we'll update our code to match, then. Meanwhile, you can either: 1. Write a module to disable writes to tmp for the Unicorn::TeeInput class (or monkey patch it) it. 2. Without loading Rack::Lint (or anything that wraps env["rack.input"]): Redirect the temporary file to /dev/null: input = env["rack.input"] if input.respond_to?(:tmp) tmp = input.tmp # StringIO is used for bodies <112K, can't reopen those tmp.respond_to?(:reopen) and tmp.reopen('/dev/null', 'wb') end -- Eric Wong _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying