From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dagaen Golomb Subject: Re: [PATCH] Modified RTDS scheduler to use an event-driven model instead of polling. Date: Mon, 22 Jun 2015 07:58:45 -0400 Message-ID: References: <1433764019-2194-1-git-send-email-dgolomb@seas.upenn.edu> <1433854393.2403.141.camel@citrix.com> <1433975412.2482.116.camel@citrix.com> <1434500436.2420.154.camel@citrix.com> <1434529644.2375.129.camel@citrix.com> <1434964300.25170.65.camel@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1434964300.25170.65.camel@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Dario Faggioli Cc: Wei Liu , Sisu Xi , George Dunlap , xen-devel@lists.xen.org, Meng Xu , Jan Beulich , Chong Li List-Id: xen-devel@lists.xenproject.org Thank you. This clears things up a bit. I will work on this model and make (or ignore) any corner case decisions I feel adequate for now. We can focus on them after the main structure is in place. ~Dagaen On Mon, Jun 22, 2015 at 5:11 AM, Dario Faggioli wrote: > On Thu, 2015-06-18 at 14:07 -0400, Dagaen Golomb wrote: > >> > Anyway, I've zero interest in turning this into a fight over >> > terminology... If you want to call runq_tickle() "the scheduler", go >> > ahead, it would just make communication a bit more difficult, but I'm up >> > for the challenge! :-) >> > >> > Oh, BTW, while we're here, __runq_pick() being called from a bunch of >> > places, outside of rt_schedule(), is also something I never liked (you >> > can go check that in the archives), and I also blame the confusion >> > between scheduling and replenishmets, for the fact that such a thing is >> > necessary. I seriously hope this can be fixed too. >> >> I have no interest in a terminology war either. I just figured that if we want >> the replenishment function to only tickle when necessary, then it should >> check that the replenished vCPU is a greater priority than current ones. >> > Exactly, we need a preemption check, which is part of the 'scheduling > logic'... but everything in the file is part of the scheduling logic (or > it will be in another one), and we don't want to put everything that is > in the file in one only function! :-) > > An in fact, a preemption check is not the full scheduling logic it's... > well... a preemption check. For example, if there are idle pCPUs, it's > something very quick. > > Note that, right now, this preemption check is done by acquiring the > global lock and checking the deadlines of currently running vCPUs on all > pCPUs. In future, this can be modified/improved, by using a dedicate > data structure, and a locking scheme that would reduce the pressure on > the global lock. I'm not saying that you should do this as part of the > work you're doing now. Rather, I'm saying that the work being done now > should go in the direction of making this easier, not harder. > >> > And in fact, I want __runq_pick() and related logic to be in >> > rt_schedule(), and nowhere else, while I want runq_tickle() to be done >> > from replenishment (and wakeup), and nowhere else. >> >> I guess the last remaining question is this: when the scheduler is >> enforcing a budget and times out, should it check for replenishment >> before kicking the vCPU? Or assume that any relevant replenishments >> have occurred? >> > This is an implementation detail that is quite hard to discuss without > seeing the code. In theory, if there wasn't any overhead, etc., you > shouldn't, because it should not be necessary. In practise, yes, it is > possible that various sources of overhead prevent a replenishment to be > notified in time, and that you end up with a vCPU running, and consuming > the last bits of its budget _after_ a scheduled replenishment instant as > it is, I guess, possible that you figure during a replenishment that the > budget was exhausted and the vCPU had overrun a bit, I guess (or not, it > again depends on the actual implementation). > > So, yes, you certainly need to take care of this corner cases, and you > can do it in the way you think it's best, basing on how the code ends up > looking like. The important thing is that they're treated for what they > are, i.e., we should handle them, not design the code around them. > >> This is the issue I mentioned before about two timers >> armed at the same time - I'm not sure what convention Xen uses to >> order them. >> > No, I don't think this has much to do with the internals of timers' > implementation, it's a corner case that, due to overhead, long critical > section (or, in general, interrupts-off sections), etc., will always be > present and will need to be taken care of, no mater how the scheduler is > implemented. > >> I would assume from your very reason for mentioning this >> change that you don't want any replenishment in rt_schedule, but then >> it may kick a vCPU who at that very instant is supposed to be replenished >> as well, and should actually stay on the pCPU. >> > That depends of the vCPU, on it's budget and deadline, and on what's > running on other pCPUs. And that is something unlikely, although > possible, and it should be the exception rather than the rule. > > Anyway, I think it would be correct, for instance, to check during > rt_schedule() whether the vCPU running on the pCPU is already beyond > it's deadline/replenishment time. If yes, log/count a deadline miss > (because that's what just happened!), replenish it and continue with > rt_schedule(). > >> This is assuming you >> still want rt_schedule to be timer-triggered to enforce budget, is this >> correct? >> > Of course. > >> However, I think this is a more minor issue that we can sort out via >> inspecting the default Xen behavior, or allowing a single replenishment >> call before kicking (I don't expect you to like that option :P), or some >> other method. >> > Exactly. > > Thanks and Regards, > Dario > > -- > <> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)