* rainbows for 3rd party api @ 2009-11-04 18:49 Giovanni Lion [not found] ` <2007122a0911041049u2b4376dbpd3b1f727e315ea88-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Giovanni Lion @ 2009-11-04 18:49 UTC (permalink / raw) To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw Hi all, I came across rainbows while I was looking for a smart solution for handling 3rd party api calls for my rails app. I would like to know a little more about how to achieve efficency in the following context: 1 user requests a page 2 page content requires xml to be retrieved from 3rd party server through http call 3 page is rendered, without the 3rd party data but with an onload ajax request back to the app to retrieve 3rd party data 4 app generates an http call to 3rd party api 5 app waits for 3rd party response 6 app responds to ajax call rendering html out of the xml response from 3rd party api Right now my current setup is apache + passenger, no constraints on switching to anything else. This setup is not optimal of course because if i receive many concurrent requests that need 3rd party response passenger app pool is full and sleepy. From what i read in the documentation rainbows should come handy in this situation. I had a look at unicorn and i think i got more or less how it works. Can anyone suggest me how to set up the app deployment in order to reduce waste on step 5? My guessing is should create a rack app to handle these calls using DevFdResponse and run it with rainbows. Only problem is can i have the rails environment in there? Thanks in advance, Giovanni ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <2007122a0911041049u2b4376dbpd3b1f727e315ea88-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: rainbows for 3rd party api [not found] ` <2007122a0911041049u2b4376dbpd3b1f727e315ea88-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2009-11-04 21:40 ` Eric Wong [not found] ` <20091104214018.GA25942-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Eric Wong @ 2009-11-04 21:40 UTC (permalink / raw) To: Rainbows! list Giovanni Lion <giovanni.lion-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > Hi all, > > I came across rainbows while I was looking for a smart solution for > handling 3rd party api calls for my rails app. I would like to know a > little more about how to achieve efficency in the following context: > > 1 user requests a page > 2 page content requires xml to be retrieved from 3rd party server > through http call > 3 page is rendered, without the 3rd party data but with an onload ajax > request back to the app to retrieve 3rd party data > 4 app generates an http call to 3rd party api > 5 app waits for 3rd party response > 6 app responds to ajax call rendering html out of the xml response > from 3rd party api > > Right now my current setup is apache + passenger, no constraints on > switching to anything else. This setup is not optimal of course > because if i receive many concurrent requests that need 3rd party > response passenger app pool is full and sleepy. From what i read in > the documentation rainbows should come handy in this situation. I had > a look at unicorn and i think i got more or less how it works. Can > anyone suggest me how to set up the app deployment in order to reduce > waste on step 5? My guessing is should create a rack app to handle > these calls using DevFdResponse and run it with rainbows. Only problem > is can i have the rails environment in there? Hi Giovanni, 3rd party API responses are exactly one of the uses Rainbows! was built for. You really only want DevFdResponse if you're doing a straight proxy between the 3rd party and your client without modifying the data. Since you seem to be getting XML and rendering HTML, you probably can't use DevFdResponse efficiently. Don't despair, though, Rainbows! still gives you plenty of options :) You can build a Rack config.ru to use with Rails, too. In fact, you'll have to for now since we're unsure if we want to support a "rainbows_rails" wrapper like I do with "unicorn_rails". Using config.ru gives you much more flexibility to route around/outside of Rails. Your config.ru can be something like this: ---------------- 8< ------------------ # this example is totally untested and may have syntax errors require 'config/boot' # might not be necessary with newer Rails require 'config/environment' # you only need one of these: dispatcher = if $old_rails require 'unicorn/app/old_rails' Unicorn::App::OldRails.new else ActionController::Dispatcher.new end # send all 3rd party API requests to "/3rd_party" through this block: map("/3rd_party") do use Rack::ContentLength run lambda { |env| # error-checking is left as an exercise to the reader :) body = if env['rainbows.model'] == :Revactor url = "http://example.com/#{ENV['PATH_INFO']}" Revactor::HttpClient.request("GET", url).body else Net::HTTP.get("api.example.com", env["PATH_INFO"]) end # render_to_html(body) # define your own function here [ 200, { "Content-Type" => "text/html" }, [ body ] ] } end # send normal Rails requests here: map("/") do use Rack::Lock # only needed in case your Rails app is not thread-safe # you can also use Rainbows::AppPool to limit Rails concurrency here # independently of "/3rd_party" requests if your Rails app is # thread-safe but not happy with too many threads. run dispatcher end ---------------------------------- 8< -------------------------------- The above example will work best with the ThreadPool or ThreadSpawn or Revactor concurrency models. I hope to have time to work on hybrid concurrency models with Rev and EventMachine to mix threads into them so the application dispatch can be concurrent for those, not just client <-> server I/O too. -- Eric Wong ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <20091104214018.GA25942-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>]
* Re: rainbows for 3rd party api [not found] ` <20091104214018.GA25942-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> @ 2009-11-05 13:03 ` Giovanni Lion [not found] ` <2007122a0911050503x5740cf3ei4f1185b4cb895298-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Giovanni Lion @ 2009-11-05 13:03 UTC (permalink / raw) To: Rainbows! list > Hi Giovanni, > > 3rd party API responses are exactly one of the uses Rainbows! was built > for. > > You really only want DevFdResponse if you're doing a straight proxy > between the 3rd party and your client without modifying the data. Since > you seem to be getting XML and rendering HTML, you probably can't use > DevFdResponse efficiently. Don't despair, though, Rainbows! still > gives you plenty of options :) > > You can build a Rack config.ru to use with Rails, too. In fact, you'll > have to for now since we're unsure if we want to support a > "rainbows_rails" wrapper like I do with "unicorn_rails". Using > config.ru gives you much more flexibility to route around/outside > of Rails. Ok i think i got most of it. Now i was just thinking about the best way to get this going. The issue now is that processing the xml into the html is something I prefer keeping insde the app for consistency. My idea was to do something like this, i use an example this time: 1) I get a request for a friend list html partial 2) I intercept it and using revactor 3) Wait for the response (It shouldn't be called waiting with revactor and fibers, right?) 4) I write the response to memcached 5) I call the rails app who now fetches from cache the friend list 6) The rails app renders the partial and everybody is happy Do you think this is a good flow? Should I create a specific method instead? Also, another issue is not clear to me yet. I need to make api requests with oauth, hence there is a logic layer on top of the http request. I would probably need to brake it down and replace Net:HTTP with Revactor::HttpClient, am i correct? Thanks for your help Giovanni _______________________________________________ rainbows-talk mailing list rainbows-talk@rubyforge.org http://rubyforge.org/mailman/listinfo/rainbows-talk ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <2007122a0911050503x5740cf3ei4f1185b4cb895298-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: rainbows for 3rd party api [not found] ` <2007122a0911050503x5740cf3ei4f1185b4cb895298-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2009-11-05 23:06 ` Eric Wong [not found] ` <20091105230638.GA7131-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Eric Wong @ 2009-11-05 23:06 UTC (permalink / raw) To: Rainbows! list Giovanni Lion <giovanni.lion-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > Hi Giovanni, > > > > 3rd party API responses are exactly one of the uses Rainbows! was built > > for. > > > > You really only want DevFdResponse if you're doing a straight proxy > > between the 3rd party and your client without modifying the data. Since > > you seem to be getting XML and rendering HTML, you probably can't use > > DevFdResponse efficiently. Don't despair, though, Rainbows! still > > gives you plenty of options :) > > > > You can build a Rack config.ru to use with Rails, too. In fact, you'll > > have to for now since we're unsure if we want to support a > > "rainbows_rails" wrapper like I do with "unicorn_rails". Using > > config.ru gives you much more flexibility to route around/outside > > of Rails. > > Ok i think i got most of it. Now i was just thinking about the best > way to get this going. The issue now is that processing the xml into > the html is something I prefer keeping insde the app for consistency. > My idea was to do something like this, i use an example this time: > > 1) I get a request for a friend list html partial > 2) I intercept it and using revactor > 3) Wait for the response (It shouldn't be called waiting with revactor > and fibers, right?) Well, from the caller's point of view, it is waiting :) > 4) I write the response to memcached > 5) I call the rails app who now fetches from cache the friend list > 6) The rails app renders the partial and everybody is happy > > Do you think this is a good flow? Should I create a specific method instead? Depends on the rest of your app, I guess. Is your Rails app reentrant? If so, definitely go for it. If you're dealing with DB connections in there, compatibility will probably be better with the ThreadPool or ThreadSpawn models unless somebody writes Revactor-enabled DB libraries. > Also, another issue is not clear to me yet. I need to make api > requests with oauth, hence there is a logic layer on top of the http > request. I would probably need to brake it down and replace Net:HTTP > with Revactor::HttpClient, am i correct? Yes. To effectively use Revactor you pretty much have to change any parts of your app/libraries to use Revactor's networking API. Fortunately it's not too hard since program logic is still linear with the Actor model. You might even want to do it for memcached, too, but then again memcached is pretty fast on a LAN and you might not notice it block. -- Eric Wong ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <20091105230638.GA7131-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>]
* Re: rainbows for 3rd party api [not found] ` <20091105230638.GA7131-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> @ 2009-11-06 10:40 ` Giovanni Lion [not found] ` <2007122a0911060240j105c1fcfgfebb2c5757cf7fd1-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Giovanni Lion @ 2009-11-06 10:40 UTC (permalink / raw) To: Rainbows! list On Fri, Nov 6, 2009 at 12:06 AM, Eric Wong <normalperson@yhbt.net> wrote: > Giovanni Lion <giovanni.lion@gmail.com> wrote: >> > Hi Giovanni, >> > >> > 3rd party API responses are exactly one of the uses Rainbows! was built >> > for. >> > >> > You really only want DevFdResponse if you're doing a straight proxy >> > between the 3rd party and your client without modifying the data. Since >> > you seem to be getting XML and rendering HTML, you probably can't use >> > DevFdResponse efficiently. Don't despair, though, Rainbows! still >> > gives you plenty of options :) >> > >> > You can build a Rack config.ru to use with Rails, too. In fact, you'll >> > have to for now since we're unsure if we want to support a >> > "rainbows_rails" wrapper like I do with "unicorn_rails". Using >> > config.ru gives you much more flexibility to route around/outside >> > of Rails. >> >> Ok i think i got most of it. Now i was just thinking about the best >> way to get this going. The issue now is that processing the xml into >> the html is something I prefer keeping insde the app for consistency. >> My idea was to do something like this, i use an example this time: >> >> 1) I get a request for a friend list html partial >> 2) I intercept it and using revactor >> 3) Wait for the response (It shouldn't be called waiting with revactor >> and fibers, right?) > > Well, from the caller's point of view, it is waiting :) #=> true >> 4) I write the response to memcached >> 5) I call the rails app who now fetches from cache the friend list >> 6) The rails app renders the partial and everybody is happy >> >> Do you think this is a good flow? Should I create a specific method instead? > > Depends on the rest of your app, I guess. Is your Rails app > reentrant? If so, definitely go for it. If you're dealing > with DB connections in there, compatibility will probably be > better with the ThreadPool or ThreadSpawn models unless somebody > writes Revactor-enabled DB libraries. Well, I do have a db in there but it's just one users table, which i really need only to pair the facebook id to the oauth tokens to make the api calls. Unfortunately it's a 3-legged oauth so i'm kinda stuck with having my own db. I'm testing now something like this: map("/3rd_party/friends") do use Rack::Facebook #checking the fb signature run lambda { |env| request = Rack::Request.new(env) return Rack::Response.new(["Invalid Facebook signature"], 400).finish unless request.POST['fb_sig'] user = User.find_by_fb_id(request.POST['fb_sig_user']) #db query ... oauth_stuff_i_am_still_working_on .... } end Will rails connection pool mess things up if used in an actor? Right now I'm really much more worried about the app being blocked in the http call as the third party is acting wonky lately. Db load is not a worry for now. Let me know your thoughts while I get the oauth - revactor http going. Giovanni _______________________________________________ rainbows-talk mailing list rainbows-talk@rubyforge.org http://rubyforge.org/mailman/listinfo/rainbows-talk ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <2007122a0911060240j105c1fcfgfebb2c5757cf7fd1-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: rainbows for 3rd party api [not found] ` <2007122a0911060240j105c1fcfgfebb2c5757cf7fd1-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2009-11-06 19:20 ` Eric Wong 0 siblings, 0 replies; 6+ messages in thread From: Eric Wong @ 2009-11-06 19:20 UTC (permalink / raw) To: Rainbows! list Giovanni Lion <giovanni.lion-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > On Fri, Nov 6, 2009 at 12:06 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > > Giovanni Lion <giovanni.lion-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >> 4) I write the response to memcached > >> 5) I call the rails app who now fetches from cache the friend list > >> 6) The rails app renders the partial and everybody is happy > >> > >> Do you think this is a good flow? Should I create a specific method instead? > > > > Depends on the rest of your app, I guess. Is your Rails app > > reentrant? If so, definitely go for it. If you're dealing > > with DB connections in there, compatibility will probably be > > better with the ThreadPool or ThreadSpawn models unless somebody > > writes Revactor-enabled DB libraries. > > Well, I do have a db in there but it's just one users table, which i > really need only to pair the facebook id to the oauth tokens to make > the api calls. Unfortunately it's a 3-legged oauth so i'm kinda stuck > with having my own db. I'm testing now something like this: > > map("/3rd_party/friends") do > use Rack::Facebook #checking the fb signature > run lambda { |env| > request = Rack::Request.new(env) > return Rack::Response.new(["Invalid Facebook signature"], > 400).finish unless request.POST['fb_sig'] > user = User.find_by_fb_id(request.POST['fb_sig_user']) #db query > ... > oauth_stuff_i_am_still_working_on > .... > } > end > > Will rails connection pool mess things up if used in an actor? Right > now I'm really much more worried about the app being blocked in the > http call as the third party is acting wonky lately. Db load is not a > worry for now. > > Let me know your thoughts while I get the oauth - revactor http going. I don't think there's any need to worry if your DB queries are fast and predictable in performance. Let us know how everything goes. 3rd-party API calls often pose a scalability issue that Rainbows! is designed to solve. -- Eric Wong ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-11-06 19:26 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2009-11-04 18:49 rainbows for 3rd party api Giovanni Lion [not found] ` <2007122a0911041049u2b4376dbpd3b1f727e315ea88-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2009-11-04 21:40 ` Eric Wong [not found] ` <20091104214018.GA25942-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2009-11-05 13:03 ` Giovanni Lion [not found] ` <2007122a0911050503x5740cf3ei4f1185b4cb895298-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2009-11-05 23:06 ` Eric Wong [not found] ` <20091105230638.GA7131-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2009-11-06 10:40 ` Giovanni Lion [not found] ` <2007122a0911060240j105c1fcfgfebb2c5757cf7fd1-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2009-11-06 19:20 ` Eric Wong
Code repositories for project(s) associated with this public inbox https://yhbt.net/rainbows.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).