unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
* Master Process Reaping Worker with No message
@ 2017-05-23 13:20 Aakash Gupta
  2017-05-23 16:22 ` Eric Wong
  0 siblings, 1 reply; 4+ messages in thread
From: Aakash Gupta @ 2017-05-23 13:20 UTC (permalink / raw)
  To: unicorn-public@bogomips.org

I am using unicorn server with Nginx for running my Rails app. I don't know why but my master server is reaping workers continously while file uploads. I have raised the timeout limit of unicorn to 20 minutes for giving sufficient time for uploading file. This happends mostly during file uploads on server. For file uploads, I am using Carrierwave gem with Fog to upload the files to S3. Server is running on AWS EC2 instance of 1GB RAM with 2GB Swap space. 
 Master process kills worker with message "ERROR : reaped ... SIGKILL (signal 9)> worker=1 " without any reason or message such as timeout or memory overflow.

Whenever a worker is reaped, Nginx also logs an error for each reaping instance with message:
" upstream prematurely closed connection while reading response header from upstream ..."

I need help to figure out the problem and the reason why this is happening. I don't think this is happening because of timeout issue because whenever I try to test my server by uploading a file, this error appears even before 20 minutes which is the timeout limit. Any help regarding this would be appreciated. 

^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: Master Process Reaping Worker with No message
@ 2017-05-23 19:29 Aakash Gupta
  2017-05-23 20:40 ` Eric Wong
  0 siblings, 1 reply; 4+ messages in thread
From: Aakash Gupta @ 2017-05-23 19:29 UTC (permalink / raw)
  To: unicorn-public@bogomips.org; +Cc: unicorn-public@bogomips.org

Thank a lot Eric. 
I am using Ruby 2.3.3 and Unicorn 5.3. 

"It sounds like somebody else is sending SIGKILL to a worker..." This line of yours saved my server. 

Actually when I uploads a file, it first goes to /tmp directory and then one of the worker processes of unicorn uploads it to S3 from there. So, earlier when I was trying to fix timeout connection error of unicorn, I increased it to 20 minutes and the problem was solved temporarily. 

But the problem was that the files that were uploaded temporarily to /tmp directory, those files do not get deleted from /tmp after they are uploaded to S3. So, I wrote a script using tmpreaper that cleans the /tmp directory recursively and removes the files that are 30 min older and added that to cron service. 

After this, as a result of this script, all the files that were required to be removed, were removed from /tmp directory but again a problem occurred. When I used "df -h" command to check whether the files are actually deleted or not, it showed that the space that was taken by uploaded files on my disk has not been freed :( . So, I searched internet and got to know that any other process was still using those files, that was why the space on the disk was not freed. 

I used ' lsof +L1 | grep "deleted" ' command and found that a process named 'bundle' is still using these deleted files. So, I again wrote a script that was killing the process that was using a deleted file.  Script was using this command

" lsof +L1 | grep 'deleted' | awk '{print $2}' | xargs kill -9 "

Today, after reading your words, I finally realised that my script was killing the unicorn worker itself because the command that I was using was killing the parent of "bundle" process. As "bundle" was a child of Unicorn worker, unicorn worker was being killed.

So, thanks a lot Eric :) 

I was finally able to solve this error because of the direction and hint that you gave me. But the only thing that I need to find is that what is causing uploaded files to persist in /tmp directory.   

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2017-05-23 20:40 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-23 13:20 Master Process Reaping Worker with No message Aakash Gupta
2017-05-23 16:22 ` Eric Wong
  -- strict thread matches above, loose matches on Subject: below --
2017-05-23 19:29 Aakash Gupta
2017-05-23 20:40 ` Eric Wong

Code repositories for project(s) associated with this public inbox


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).