From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-2.9 required=3.0 tests=ALL_TRUSTED,BAYES_00, URIBL_BLOCKED shortcircuit=no autolearn=unavailable version=3.3.2 X-Original-To: yahns-public@yhbt.net Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id DA04D633811; Thu, 21 May 2015 00:10:43 +0000 (UTC) Date: Thu, 21 May 2015 00:10:43 +0000 From: Eric Wong To: yahns-public@yhbt.net Subject: beware of lowering TCP buffer sizes under Linux Message-ID: <20150521001043.GA25809@dcvr.yhbt.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline List-Id: On Linux, the (currently) excessively large memory buffers size defaults hide some bugs in the TCP implementation regarding edge-triggered epoll wakeups. Setting low :sndbuf values (or lowering tcp_wmem/tcp_mem) can cause problems) for servers using edge-triggered I/O. yahns uses one-shot epoll (a special case of edge-triggering), so is also affected; nginx uses edge-triggering, so it would also be affected. Most other epoll users I know about use level-triggering, so are not affected. For now, leaving buffers at defaults should be sufficient and I've yet to encounter this problem in the wild with default settings on current kernels. These problems are being worked on in netdev@vger.kernel.org: http://mid.gmane.org/20150520155253.86202203D@prod-mail-relay06.akamai.com http://mid.gmane.org/20150506155223.9743A2027@prod-mail-relay06.akamai.com http://mid.gmane.org/1431718770-3815-1-git-send-email-edumazet@google.com Fwiw, I have encountered and helped fix missing wakeup bugs in Linux before, and I'm sure there'll be new bugs found in the future when handling millions of sockets.