From: "Wu, Feng" <feng.wu@intel.com>
To: "Jan Beulich (JBeulich@suse.com)" <JBeulich@suse.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
"keir@xen.org" <keir@xen.org>,
"george.dunlap@eu.citrix.com" <george.dunlap@eu.citrix.com>,
Andrew Cooper <andrew.cooper3@citrix.com>,
"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>,
"Zhang, Yang Z" <yang.z.zhang@intel.com>,
"Wu, Feng" <feng.wu@intel.com>
Subject: Re: [v3 12/15] vmx: posted-interrupt handling when vCPU is blocked
Date: Wed, 8 Jul 2015 10:36:35 +0000 [thread overview]
Message-ID: <E959C4978C3B6342920538CF579893F0025F8BAA@SHSMSX104.ccr.corp.intel.com> (raw)
In-Reply-To: <55917B35.2070706@citrix.com>
> -----Original Message-----
> From: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
> Sent: Tuesday, June 30, 2015 1:07 AM
> To: Wu, Feng; xen-devel@lists.xen.org
> Cc: keir@xen.org; jbeulich@suse.com; Tian, Kevin; Zhang, Yang Z;
> george.dunlap@eu.citrix.com
> Subject: Re: [v3 12/15] vmx: posted-interrupt handling when vCPU is blocked
>
> On 24/06/15 06:18, Feng Wu wrote:
> > This patch includes the following aspects:
> > - Add a global vector to wake up the blocked vCPU
> > when an interrupt is being posted to it (This
> > part was sugguested by Yang Zhang <yang.z.zhang@intel.com>).
> > - Adds a new per-vCPU tasklet to wakeup the blocked
> > vCPU. It can be used in the case vcpu_unblock
> > cannot be called directly.
> > - Define two per-cpu variables:
> > * pi_blocked_vcpu:
> > A list storing the vCPUs which were blocked on this pCPU.
> >
> > * pi_blocked_vcpu_lock:
> > The spinlock to protect pi_blocked_vcpu.
> >
> > Signed-off-by: Feng Wu <feng.wu@intel.com>
> > ---
> > v3:
> > - This patch is generated by merging the following three patches in v2:
> > [RFC v2 09/15] Add a new per-vCPU tasklet to wakeup the blocked vCPU
> > [RFC v2 10/15] vmx: Define two per-cpu variables
> > [RFC v2 11/15] vmx: Add a global wake-up vector for VT-d
> Posted-Interrupts
> > - rename 'vcpu_wakeup_tasklet' to 'pi_vcpu_wakeup_tasklet'
> > - Move the definition of 'pi_vcpu_wakeup_tasklet' to 'struct arch_vmx_struct'
> > - rename 'vcpu_wakeup_tasklet_handler' to
> 'pi_vcpu_wakeup_tasklet_handler'
> > - Make pi_wakeup_interrupt() static
> > - Rename 'blocked_vcpu_list' to 'pi_blocked_vcpu_list'
> > - move 'pi_blocked_vcpu_list' to 'struct arch_vmx_struct'
> > - Rename 'blocked_vcpu' to 'pi_blocked_vcpu'
> > - Rename 'blocked_vcpu_lock' to 'pi_blocked_vcpu_lock'
> >
> > xen/arch/x86/hvm/vmx/vmcs.c | 3 +++
> > xen/arch/x86/hvm/vmx/vmx.c | 54
> ++++++++++++++++++++++++++++++++++++++
> > xen/include/asm-x86/hvm/hvm.h | 1 +
> > xen/include/asm-x86/hvm/vmx/vmcs.h | 5 ++++
> > xen/include/asm-x86/hvm/vmx/vmx.h | 5 ++++
> > 5 files changed, 68 insertions(+)
> >
> > diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
> > index 11dc1b5..0c5ce3f 100644
> > --- a/xen/arch/x86/hvm/vmx/vmcs.c
> > +++ b/xen/arch/x86/hvm/vmx/vmcs.c
> > @@ -631,6 +631,9 @@ int vmx_cpu_up(void)
> > if ( cpu_has_vmx_vpid )
> > vpid_sync_all();
> >
> > + INIT_LIST_HEAD(&per_cpu(pi_blocked_vcpu, cpu));
> > + spin_lock_init(&per_cpu(pi_blocked_vcpu_lock, cpu));
> > +
> > return 0;
> > }
> >
> > diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c
> > index b94ef6a..7db6009 100644
> > --- a/xen/arch/x86/hvm/vmx/vmx.c
> > +++ b/xen/arch/x86/hvm/vmx/vmx.c
> > @@ -82,7 +82,20 @@ static int vmx_msr_read_intercept(unsigned int msr,
> uint64_t *msr_content);
> > static int vmx_msr_write_intercept(unsigned int msr, uint64_t
> msr_content);
> > static void vmx_invlpg_intercept(unsigned long vaddr);
> >
> > +/*
> > + * We maintian a per-CPU linked-list of vCPU, so in PI wakeup handler we
> > + * can find which vCPU should be waken up.
> > + */
> > +DEFINE_PER_CPU(struct list_head, pi_blocked_vcpu);
> > +DEFINE_PER_CPU(spinlock_t, pi_blocked_vcpu_lock);
> > +
> > uint8_t __read_mostly posted_intr_vector;
> > +uint8_t __read_mostly pi_wakeup_vector;
> > +
> > +static void pi_vcpu_wakeup_tasklet_handler(unsigned long arg)
> > +{
> > + vcpu_unblock((struct vcpu *)arg);
> > +}
> >
> > static int vmx_domain_initialise(struct domain *d)
> > {
> > @@ -148,11 +161,19 @@ static int vmx_vcpu_initialise(struct vcpu *v)
> > if ( v->vcpu_id == 0 )
> > v->arch.user_regs.eax = 1;
> >
> > + tasklet_init(
> > + &v->arch.hvm_vmx.pi_vcpu_wakeup_tasklet,
> > + pi_vcpu_wakeup_tasklet_handler,
> > + (unsigned long)v);
>
> c/s f6dd295 indicates that the global tasklet lock causes a bottleneck
> when injecting interrupts, and replaced a tasklet with a softirq to fix
> the scalability issue.
>
> I would expect exactly the bottleneck to exist here.
I am still considering this comments. Jan, what is your opinion about this?
Thanks,
Feng
>
> > +
> > + INIT_LIST_HEAD(&v->arch.hvm_vmx.pi_blocked_vcpu_list);
> > +
> > return 0;
> > }
> >
> > static void vmx_vcpu_destroy(struct vcpu *v)
> > {
> > + tasklet_kill(&v->arch.hvm_vmx.pi_vcpu_wakeup_tasklet);
> > /*
> > * There are cases that domain still remains in log-dirty mode when it
> is
> > * about to be destroyed (ex, user types 'xl destroy <dom>'), in which
> case
> > @@ -1848,6 +1869,33 @@ static struct hvm_function_table __initdata
> vmx_function_table = {
> > .enable_msr_exit_interception = vmx_enable_msr_exit_interception,
> > };
> >
> > +/*
> > + * Handle VT-d posted-interrupt when VCPU is blocked.
> > + */
> > +static void pi_wakeup_interrupt(struct cpu_user_regs *regs)
> > +{
> > + struct arch_vmx_struct *vmx;
> > + unsigned int cpu = smp_processor_id();
> > +
> > + spin_lock(&per_cpu(pi_blocked_vcpu_lock, cpu));
>
> this_cpu($foo) should be used in preference to per_cpu($foo, $myself).
>
> However, always hoist repeated uses of this/per_cpu into local
> variables, as the compiler is unable to elide repeated accesses (because
> of a deliberate anti-optimisation behind the scenes).
>
> spinlock_t *lock = &this_cpu(pi_blocked_vcpu_lock);
> list_head *blocked_vcpus = &this_cpu(ps_blocked_vcpu);
>
> ~Andrew
next prev parent reply other threads:[~2015-07-08 10:36 UTC|newest]
Thread overview: 155+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-06-24 5:18 [v3 00/15] Add VT-d Posted-Interrupts support Feng Wu
2015-06-24 5:18 ` [v3 01/15] Vt-d Posted-intterrupt (PI) design Feng Wu
2015-06-24 6:15 ` Meng Xu
2015-06-24 6:19 ` Wu, Feng
2015-07-08 7:21 ` Tian, Kevin
2015-07-08 7:29 ` Wu, Feng
2015-06-24 5:18 ` [v3 02/15] Add helper macro for X86_FEATURE_CX16 feature detection Feng Wu
2015-06-24 17:31 ` Andrew Cooper
2015-07-08 7:23 ` Tian, Kevin
2015-06-24 5:18 ` [v3 03/15] Add cmpxchg16b support for x86-64 Feng Wu
2015-06-24 18:35 ` Andrew Cooper
2015-07-08 7:06 ` Wu, Feng
2015-07-08 8:12 ` Jan Beulich
2015-07-08 8:33 ` Wu, Feng
2015-07-08 8:43 ` Jan Beulich
2015-07-08 8:50 ` Wu, Feng
2015-07-08 8:50 ` Andrew Cooper
2015-07-10 12:57 ` Jan Beulich
2015-06-24 5:18 ` [v3 04/15] iommu: Add iommu_intpost to control VT-d Posted-Interrupts feature Feng Wu
2015-06-25 9:06 ` Andrew Cooper
2015-06-25 9:47 ` Wu, Feng
2015-06-25 10:16 ` Andrew Cooper
2015-06-25 12:47 ` Wu, Feng
2015-07-08 7:30 ` Tian, Kevin
2015-06-24 5:18 ` [v3 05/15] vt-d: VT-d Posted-Interrupts feature detection Feng Wu
2015-06-25 10:21 ` Andrew Cooper
2015-06-25 13:02 ` Wu, Feng
2015-07-08 7:32 ` Tian, Kevin
2015-07-08 8:00 ` Wu, Feng
2015-06-24 5:18 ` [v3 06/15] vmx: Extend struct pi_desc to support VT-d Posted-Interrupts Feng Wu
2015-06-29 15:04 ` Andrew Cooper
2015-07-08 7:48 ` Tian, Kevin
2015-07-10 13:08 ` Jan Beulich
2015-07-15 2:40 ` Wu, Feng
2015-07-15 8:20 ` Jan Beulich
2015-07-15 8:26 ` Wu, Feng
2015-07-15 8:36 ` Jan Beulich
2015-07-15 8:43 ` Wu, Feng
2015-07-15 9:28 ` Jan Beulich
2015-07-15 9:30 ` Wu, Feng
2015-07-15 3:13 ` Wu, Feng
2015-06-24 5:18 ` [v3 07/15] vmx: Initialize VT-d Posted-Interrupts Descriptor Feng Wu
2015-06-29 15:32 ` Andrew Cooper
2015-06-30 1:46 ` Wu, Feng
2015-06-30 2:32 ` Dario Faggioli
2015-07-08 7:53 ` Tian, Kevin
2015-06-24 5:18 ` [v3 08/15] Suppress posting interrupts when 'SN' is set Feng Wu
2015-06-29 15:41 ` Andrew Cooper
2015-06-30 1:48 ` Wu, Feng
2015-07-08 9:06 ` Tian, Kevin
2015-07-08 10:11 ` Wu, Feng
2015-07-08 11:31 ` Tian, Kevin
2015-07-08 11:58 ` Wu, Feng
2015-07-10 13:20 ` Jan Beulich
2015-06-24 5:18 ` [v3 09/15] vt-d: Extend struct iremap_entry to support VT-d Posted-Interrupts Feng Wu
2015-06-29 16:04 ` Andrew Cooper
2015-06-30 1:52 ` Wu, Feng
2015-07-08 9:10 ` Tian, Kevin
2015-07-10 13:27 ` Jan Beulich
2015-06-24 5:18 ` [v3 10/15] vt-d: Add API to update IRTE when VT-d PI is used Feng Wu
2015-06-29 16:22 ` Andrew Cooper
2015-07-08 9:59 ` Tian, Kevin
2015-07-08 10:12 ` Wu, Feng
2015-07-10 14:01 ` Jan Beulich
2015-07-15 6:04 ` Wu, Feng
2015-07-15 8:24 ` Jan Beulich
2015-07-15 8:38 ` Wu, Feng
2015-07-15 8:46 ` Jan Beulich
2015-07-15 8:55 ` Wu, Feng
2015-07-15 9:32 ` Jan Beulich
2015-06-24 5:18 ` [v3 11/15] Update IRTE according to guest interrupt config changes Feng Wu
2015-06-29 16:46 ` Andrew Cooper
2015-07-08 10:22 ` Tian, Kevin
2015-07-08 10:31 ` Wu, Feng
2015-07-08 11:46 ` Tian, Kevin
2015-07-08 11:52 ` Wu, Feng
2015-07-08 11:54 ` Tian, Kevin
2015-07-10 14:23 ` Jan Beulich
2015-06-24 5:18 ` [v3 12/15] vmx: posted-interrupt handling when vCPU is blocked Feng Wu
2015-06-29 17:07 ` Andrew Cooper
2015-07-08 10:36 ` Wu, Feng [this message]
2015-07-08 10:48 ` Jan Beulich
[not found] ` <559181F9.6020106@citrix.com>
2015-06-30 2:51 ` Fwd: " Dario Faggioli
2015-06-30 2:59 ` Wu, Feng
2015-06-30 9:46 ` Dario Faggioli
2015-06-30 10:11 ` Andrew Cooper
2015-07-01 13:26 ` Dario Faggioli
2015-07-02 4:27 ` Wu, Feng
2015-07-02 8:30 ` Dario Faggioli
2015-07-02 8:58 ` Wu, Feng
2015-07-02 10:09 ` Dario Faggioli
2015-07-02 10:41 ` Wu, Feng
2015-07-02 10:30 ` Andrew Cooper
2015-07-02 10:56 ` Wu, Feng
2015-07-02 12:04 ` Dario Faggioli
2015-07-02 12:10 ` Wu, Feng
2015-07-02 12:16 ` Andrew Cooper
2015-07-02 12:38 ` Dario Faggioli
2015-07-02 12:59 ` Andrew Cooper
2015-07-03 1:33 ` Wu, Feng
2015-07-02 4:25 ` Wu, Feng
2015-07-08 11:00 ` Tian, Kevin
2015-07-08 11:02 ` Wu, Feng
2015-07-08 12:46 ` Jan Beulich
2015-07-08 13:09 ` Andrew Cooper
2015-07-08 22:49 ` Tian, Kevin
2015-07-09 7:25 ` Jan Beulich
2015-07-10 6:21 ` Wu, Feng
2015-07-10 6:32 ` Jan Beulich
2015-07-10 7:29 ` Wu, Feng
2015-07-10 8:49 ` Jan Beulich
2015-07-10 8:57 ` Wu, Feng
2015-07-08 22:31 ` Tian, Kevin
2015-06-24 5:18 ` [v3 13/15] vmx: Properly handle notification event when vCPU is running Feng Wu
2015-07-08 11:03 ` Tian, Kevin
2015-07-10 14:40 ` Jan Beulich
2015-06-24 5:18 ` [v3 14/15] Update Posted-Interrupts Descriptor during vCPU scheduling Feng Wu
[not found] ` <55918214.4030102@citrix.com>
2015-06-30 2:58 ` Fwd: " Dario Faggioli
2015-07-02 4:32 ` Wu, Feng
2015-07-02 4:34 ` Wu, Feng
2015-07-02 8:20 ` Dario Faggioli
2015-07-09 3:09 ` Wu, Feng
2015-07-09 8:18 ` Dario Faggioli
2015-07-09 11:19 ` George Dunlap
2015-07-09 11:29 ` George Dunlap
2015-07-09 11:38 ` Wu, Feng
2015-07-09 12:42 ` Dario Faggioli
2015-07-10 0:07 ` Wu, Feng
2015-07-10 12:40 ` Dario Faggioli
2015-07-10 13:47 ` Konrad Rzeszutek Wilk
2015-07-10 13:59 ` Dario Faggioli
2015-07-09 12:53 ` George Dunlap
2015-07-09 13:44 ` Jan Beulich
2015-07-09 14:18 ` Dario Faggioli
2015-07-09 14:27 ` George Dunlap
2015-07-09 14:47 ` Dario Faggioli
2015-07-10 5:59 ` Wu, Feng
2015-07-10 6:22 ` Jan Beulich
2015-07-10 11:05 ` Dario Faggioli
2015-07-14 5:44 ` Wu, Feng
2015-07-14 14:08 ` Wu, Feng
2015-07-14 14:54 ` Jan Beulich
2015-07-14 15:20 ` Dario Faggioli
2015-07-14 16:41 ` George Dunlap
2015-07-14 16:02 ` Dario Faggioli
2015-07-15 0:54 ` Wu, Feng
2015-07-17 7:46 ` Wu, Feng
2015-07-17 10:13 ` Dario Faggioli
2015-07-17 22:57 ` Wu, Feng
2015-07-18 13:43 ` Dario Faggioli
2015-07-10 0:15 ` Wu, Feng
2015-07-08 11:24 ` Tian, Kevin
2015-07-10 14:48 ` Jan Beulich
2015-06-24 5:18 ` [v3 15/15] Add a command line parameter for VT-d posted-interrupts Feng Wu
2015-07-08 11:25 ` Tian, Kevin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=E959C4978C3B6342920538CF579893F0025F8BAA@SHSMSX104.ccr.corp.intel.com \
--to=feng.wu@intel.com \
--cc=JBeulich@suse.com \
--cc=andrew.cooper3@citrix.com \
--cc=george.dunlap@eu.citrix.com \
--cc=keir@xen.org \
--cc=kevin.tian@intel.com \
--cc=xen-devel@lists.xen.org \
--cc=yang.z.zhang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.