* IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
@ 2015-06-10 8:31 ` Pavel Fedin
0 siblings, 0 replies; 15+ messages in thread
From: Pavel Fedin @ 2015-06-10 8:31 UTC (permalink / raw)
To: linux-arm-kernel
Hello guys!
> Currently on ARM, irqfd supports routing an host eventfd towards a
> virtual SPI:
> eventfd -> vSPI = gsi+32
> parameters of irqfd are the eventfd and the gsi.
Yes, but this works only with GICv2m, because it actually turns MSI data into SPI number.
ITS works in a completely different way.
> 2) now we have virtual msi injection, we could use msi routing to inject
> virtual LPI's. But is it what you need for your qemu integration?
Actually this is what i wanted to discuss here...
I have studied a little bit IRQ routing mechanism... And it comes to a question what is
'GSI'. As far as i could understand, on x86 GSI is a completely virtual entity, which can
be tied either to irqchip's pin (physical IRQ) or MSI event. There is totally no
correspondence between GSI numbers and guest IRQ numbers. GSIs are just allocated by the
userspace starting from 0 and on. Is my understanding correct?
On ARM, i see, completely different approach is used. For KVM_IRQ_LINE ioctl GSI is
actually a raw GIC IRQ number plus some extra bits for target and type. For KVM_IRQFD with
GICv2m GSI is actually SPI number (starting from zero, so that IRQ = GSI + 32).
First of all, i would say that we already have an inconsistence in ARM API. The same
thing called GSI has two different meanings for different functions.
I think it would be a bad idea to introduce a third, separate meaning for MSIs. However,
this is what we could do:
Approach 1: GICv2m way.
We could add one more IOCTL which would decode MSI data into IRQ (in our case it's LPI)
number. What it would return is LPI - 32, to keep in line with existing convention.
Pros: does not bring any more inconsistence into KVM API.
Cons: requires adding one more IOCTL and one more MSI handling mechanism. Isn't there too
many of them already?
Approach 2: IRQ routing.
We could implement MSI routing using virtual GSI numbers. In order to stay compatible
with what we have, we could say that GSI numbers below 8192 are SPI GSIs, and everything
starting from 8192 is MSI. Then we could use KVM_SET_GSI_ROUTING ioctl to assign these
GSIs to actual MSIs which then will go full-cycle through ITS.
Pros: Does not introduce any new APIs.
Cons:
- Introduces third meaning for GSI on ARM.
- Slower than approach 1 because in that case every interrupt is pre-translated while
here we engage ITS every time.
Personally i have already tried approach 1 and i can say that it works. There is no
problem with target specification because current ITS code stores everything in a single
bunch so that i anyway have to locate a particular ITTE corresponding to an LPI and get
collection ID from there. However, yes, i agree, this approach has the same performance
drawback as my suggested approach 2.
Any thoughts / ideas ?
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
^ permalink raw reply [flat|nested] 15+ messages in thread
* IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
@ 2015-06-10 8:31 ` Pavel Fedin
0 siblings, 0 replies; 15+ messages in thread
From: Pavel Fedin @ 2015-06-10 8:31 UTC (permalink / raw)
To: 'Eric Auger', 'Marc Zyngier',
'Andre Przywara', christoffer.dall
Cc: kvmarm, linux-arm-kernel, kvm
Hello guys!
> Currently on ARM, irqfd supports routing an host eventfd towards a
> virtual SPI:
> eventfd -> vSPI = gsi+32
> parameters of irqfd are the eventfd and the gsi.
Yes, but this works only with GICv2m, because it actually turns MSI data into SPI number.
ITS works in a completely different way.
> 2) now we have virtual msi injection, we could use msi routing to inject
> virtual LPI's. But is it what you need for your qemu integration?
Actually this is what i wanted to discuss here...
I have studied a little bit IRQ routing mechanism... And it comes to a question what is
'GSI'. As far as i could understand, on x86 GSI is a completely virtual entity, which can
be tied either to irqchip's pin (physical IRQ) or MSI event. There is totally no
correspondence between GSI numbers and guest IRQ numbers. GSIs are just allocated by the
userspace starting from 0 and on. Is my understanding correct?
On ARM, i see, completely different approach is used. For KVM_IRQ_LINE ioctl GSI is
actually a raw GIC IRQ number plus some extra bits for target and type. For KVM_IRQFD with
GICv2m GSI is actually SPI number (starting from zero, so that IRQ = GSI + 32).
First of all, i would say that we already have an inconsistence in ARM API. The same
thing called GSI has two different meanings for different functions.
I think it would be a bad idea to introduce a third, separate meaning for MSIs. However,
this is what we could do:
Approach 1: GICv2m way.
We could add one more IOCTL which would decode MSI data into IRQ (in our case it's LPI)
number. What it would return is LPI - 32, to keep in line with existing convention.
Pros: does not bring any more inconsistence into KVM API.
Cons: requires adding one more IOCTL and one more MSI handling mechanism. Isn't there too
many of them already?
Approach 2: IRQ routing.
We could implement MSI routing using virtual GSI numbers. In order to stay compatible
with what we have, we could say that GSI numbers below 8192 are SPI GSIs, and everything
starting from 8192 is MSI. Then we could use KVM_SET_GSI_ROUTING ioctl to assign these
GSIs to actual MSIs which then will go full-cycle through ITS.
Pros: Does not introduce any new APIs.
Cons:
- Introduces third meaning for GSI on ARM.
- Slower than approach 1 because in that case every interrupt is pre-translated while
here we engage ITS every time.
Personally i have already tried approach 1 and i can say that it works. There is no
problem with target specification because current ITS code stores everything in a single
bunch so that i anyway have to locate a particular ITTE corresponding to an LPI and get
collection ID from there. However, yes, i agree, this approach has the same performance
drawback as my suggested approach 2.
Any thoughts / ideas ?
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
^ permalink raw reply [flat|nested] 15+ messages in thread
* IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-10 8:31 ` Pavel Fedin
@ 2015-06-10 13:58 ` Eric Auger
-1 siblings, 0 replies; 15+ messages in thread
From: Eric Auger @ 2015-06-10 13:58 UTC (permalink / raw)
To: linux-arm-kernel
Hi,
On 06/10/2015 10:31 AM, Pavel Fedin wrote:
> Hello guys!
>
>> Currently on ARM, irqfd supports routing an host eventfd towards a
>> virtual SPI:
>> eventfd -> vSPI = gsi+32
>> parameters of irqfd are the eventfd and the gsi.
>
> Yes, but this works only with GICv2m, because it actually turns MSI data into SPI number.
> ITS works in a completely different way.
>
>> 2) now we have virtual msi injection, we could use msi routing to inject
>> virtual LPI's. But is it what you need for your qemu integration?
>
> Actually this is what i wanted to discuss here...
> I have studied a little bit IRQ routing mechanism... And it comes to a question what is
> 'GSI'. As far as i could understand, on x86 GSI is a completely virtual entity, which can
> be tied either to irqchip's pin (physical IRQ) or MSI event. There is totally no
> correspondence between GSI numbers and guest IRQ numbers. GSIs are just allocated by the
> userspace starting from 0 and on. Is my understanding correct?
Well I think as long as you use irqchip routing, gsi is not random. When
looking at arch/x86/kvm/irq_comm.c and in kvm_set_routing_entry you can
see there is an offset applied on the gsi and irqchip/pin depending on
type of irqchip (pic_master, pic_slave, ioapic). originally done by
BIOS? This is the default routing. Now in qemu the irqchip routing
entries are built by kvm_irqchip_add_irq_route in hw/intc/openpic_kvm.c,
hw/i386/kvm/ioapic.c, ... and then committed.
When looking at MSI routing and qemu integration, in case
kvm_gsi_direct_mapping is NOT used, kvm_irqchp_get_virq indeed finds out
a gsi belonging to the gsi range and not mapped with irqchip entries.
if kvm_gsi_direct_mapping is used, an irqchip mapped gsi is used
instead. At least this is my understanding
> On ARM, i see, completely different approach is used. For KVM_IRQ_LINE ioctl GSI is
> actually a raw GIC IRQ number plus some extra bits for target and type. For KVM_IRQFD with
> GICv2m GSI is actually SPI number (starting from zero, so that IRQ = GSI + 32).
> First of all, i would say that we already have an inconsistence in ARM API. The same
> thing called GSI has two different meanings for different functions.
well that's true. This avoided to have ARM archi specific adaptations
for VFIO and MMIO VHOST-NET proto at that time. Now with the MSI advent,
those adaptations becomes needed anyway. Also we concluded it was not
meaningful to inject PPIs. gsi irqfd argument directly matches the IRQ
number found in guest device tree.
> I think it would be a bad idea to introduce a third, separate meaning for MSIs. However,
> this is what we could do:
>
> Approach 1: GICv2m way.
> We could add one more IOCTL which would decode MSI data into IRQ (in our case it's LPI)
> number. What it would return is LPI - 32, to keep in line with existing convention.
> Pros: does not bring any more inconsistence into KVM API.
> Cons: requires adding one more IOCTL and one more MSI handling mechanism. Isn't there too
> many of them already?
indeed in newly added qemu kvm-all.c kvm_arch_msi_data_to_gsi we could
call a new ioctl that translates the data + deviceid? into an LPI and
program irqfd with that LPI. This is done once when setting irqfd up.
This also means extending irqfd support to lpi injection, gsi being the
LPI index if gsi >= 8192. in that case we continue using
kvm_gsi_direct_mapping and gsi still is an IRQ index.
>
> Approach 2: IRQ routing.
> We could implement MSI routing using virtual GSI numbers. In order to stay compatible
> with what we have, we could say that GSI numbers below 8192 are SPI GSIs, and everything
> starting from 8192 is MSI.
I think the gsi can be considered as an index:
0 - 1020: SPI index
>= 8192: LPI index
Then we could use KVM_SET_GSI_ROUTING ioctl to assign these
> GSIs to actual MSIs which then will go full-cycle through ITS.
> Pros: Does not introduce any new APIs.
> Cons:
> - Introduces third meaning for GSI on ARM.
> - Slower than approach 1 because in that case every interrupt is pre-translated while
> here we engage ITS every time.
KVM GSI routing, even if only used for MSI routing then mandates to
build entries for non MSI IRQs, using irqchip routing entries. Then you
draw the irqchip.c kvm_irq_routing_table
chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS] static allocation issue. I
guess this code would need to be revisited to accomodate large space and
variable pin number of GIC.
Hope it helps
Best Regards
Eric
>
> Personally i have already tried approach 1 and i can say that it works. There is no
> problem with target specification because current ITS code stores everything in a single
> bunch so that i anyway have to locate a particular ITTE corresponding to an LPI and get
> collection ID from there. However, yes, i agree, this approach has the same performance
> drawback as my suggested approach 2.
>
> Any thoughts / ideas ?
>
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
@ 2015-06-10 13:58 ` Eric Auger
0 siblings, 0 replies; 15+ messages in thread
From: Eric Auger @ 2015-06-10 13:58 UTC (permalink / raw)
To: Pavel Fedin, 'Marc Zyngier', 'Andre Przywara',
christoffer.dall
Cc: kvmarm, linux-arm-kernel, kvm
Hi,
On 06/10/2015 10:31 AM, Pavel Fedin wrote:
> Hello guys!
>
>> Currently on ARM, irqfd supports routing an host eventfd towards a
>> virtual SPI:
>> eventfd -> vSPI = gsi+32
>> parameters of irqfd are the eventfd and the gsi.
>
> Yes, but this works only with GICv2m, because it actually turns MSI data into SPI number.
> ITS works in a completely different way.
>
>> 2) now we have virtual msi injection, we could use msi routing to inject
>> virtual LPI's. But is it what you need for your qemu integration?
>
> Actually this is what i wanted to discuss here...
> I have studied a little bit IRQ routing mechanism... And it comes to a question what is
> 'GSI'. As far as i could understand, on x86 GSI is a completely virtual entity, which can
> be tied either to irqchip's pin (physical IRQ) or MSI event. There is totally no
> correspondence between GSI numbers and guest IRQ numbers. GSIs are just allocated by the
> userspace starting from 0 and on. Is my understanding correct?
Well I think as long as you use irqchip routing, gsi is not random. When
looking at arch/x86/kvm/irq_comm.c and in kvm_set_routing_entry you can
see there is an offset applied on the gsi and irqchip/pin depending on
type of irqchip (pic_master, pic_slave, ioapic). originally done by
BIOS? This is the default routing. Now in qemu the irqchip routing
entries are built by kvm_irqchip_add_irq_route in hw/intc/openpic_kvm.c,
hw/i386/kvm/ioapic.c, ... and then committed.
When looking at MSI routing and qemu integration, in case
kvm_gsi_direct_mapping is NOT used, kvm_irqchp_get_virq indeed finds out
a gsi belonging to the gsi range and not mapped with irqchip entries.
if kvm_gsi_direct_mapping is used, an irqchip mapped gsi is used
instead. At least this is my understanding
> On ARM, i see, completely different approach is used. For KVM_IRQ_LINE ioctl GSI is
> actually a raw GIC IRQ number plus some extra bits for target and type. For KVM_IRQFD with
> GICv2m GSI is actually SPI number (starting from zero, so that IRQ = GSI + 32).
> First of all, i would say that we already have an inconsistence in ARM API. The same
> thing called GSI has two different meanings for different functions.
well that's true. This avoided to have ARM archi specific adaptations
for VFIO and MMIO VHOST-NET proto at that time. Now with the MSI advent,
those adaptations becomes needed anyway. Also we concluded it was not
meaningful to inject PPIs. gsi irqfd argument directly matches the IRQ
number found in guest device tree.
> I think it would be a bad idea to introduce a third, separate meaning for MSIs. However,
> this is what we could do:
>
> Approach 1: GICv2m way.
> We could add one more IOCTL which would decode MSI data into IRQ (in our case it's LPI)
> number. What it would return is LPI - 32, to keep in line with existing convention.
> Pros: does not bring any more inconsistence into KVM API.
> Cons: requires adding one more IOCTL and one more MSI handling mechanism. Isn't there too
> many of them already?
indeed in newly added qemu kvm-all.c kvm_arch_msi_data_to_gsi we could
call a new ioctl that translates the data + deviceid? into an LPI and
program irqfd with that LPI. This is done once when setting irqfd up.
This also means extending irqfd support to lpi injection, gsi being the
LPI index if gsi >= 8192. in that case we continue using
kvm_gsi_direct_mapping and gsi still is an IRQ index.
>
> Approach 2: IRQ routing.
> We could implement MSI routing using virtual GSI numbers. In order to stay compatible
> with what we have, we could say that GSI numbers below 8192 are SPI GSIs, and everything
> starting from 8192 is MSI.
I think the gsi can be considered as an index:
0 - 1020: SPI index
>= 8192: LPI index
Then we could use KVM_SET_GSI_ROUTING ioctl to assign these
> GSIs to actual MSIs which then will go full-cycle through ITS.
> Pros: Does not introduce any new APIs.
> Cons:
> - Introduces third meaning for GSI on ARM.
> - Slower than approach 1 because in that case every interrupt is pre-translated while
> here we engage ITS every time.
KVM GSI routing, even if only used for MSI routing then mandates to
build entries for non MSI IRQs, using irqchip routing entries. Then you
draw the irqchip.c kvm_irq_routing_table
chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS] static allocation issue. I
guess this code would need to be revisited to accomodate large space and
variable pin number of GIC.
Hope it helps
Best Regards
Eric
>
> Personally i have already tried approach 1 and i can say that it works. There is no
> problem with target specification because current ITS code stores everything in a single
> bunch so that i anyway have to locate a particular ITTE corresponding to an LPI and get
> collection ID from there. However, yes, i agree, this approach has the same performance
> drawback as my suggested approach 2.
>
> Any thoughts / ideas ?
>
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-10 13:58 ` Eric Auger
@ 2015-06-10 15:30 ` Pavel Fedin
-1 siblings, 0 replies; 15+ messages in thread
From: Pavel Fedin @ 2015-06-10 15:30 UTC (permalink / raw)
To: linux-arm-kernel
Hi!
> indeed in newly added qemu kvm-all.c kvm_arch_msi_data_to_gsi we could
> call a new ioctl that translates the data + deviceid? into an LPI and
> program irqfd with that LPI. This is done once when setting irqfd up.
> This also means extending irqfd support to lpi injection, gsi being the
> LPI index if gsi >= 8192. in that case we continue using
> kvm_gsi_direct_mapping and gsi still is an IRQ index.
This is exactly what i have done in my kernel + qemu. I have added a new KVM capability
and then in qemu i do this:
--- cut ---
if (kvm_gsi_kernel_mapping()) {
struct kvm_msi msi;
msi.address_lo = (uint32_t)msg.address;
msi.address_hi = msg.address >> 32;
msi.data = le32_to_cpu(msg.data);
memset(msi.pad, 0, sizeof(msi.pad));
if (dev) {
msi.devid = (pci_bus_num(dev->bus) << 8) | dev->devfn;
msi.flags = KVM_MSI_VALID_DEVID;
} else {
msi.devid = 0;
msi.flags = 0;
}
return kvm_vm_ioctl(s, KVM_TRANSLATE_MSI, &msi);
}
--- cut ---
KVM_TRANSLATE_MSI returns an LPI number. This seemed to be the simplest and fastest thing
to do.
If someone is interested, i could prepare an RFC patch series for this, which would apply
on top of Andre's ITS implementation.
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
@ 2015-06-10 15:30 ` Pavel Fedin
0 siblings, 0 replies; 15+ messages in thread
From: Pavel Fedin @ 2015-06-10 15:30 UTC (permalink / raw)
To: 'Eric Auger', 'Marc Zyngier',
'Andre Przywara', christoffer.dall
Cc: kvmarm, linux-arm-kernel, kvm
Hi!
> indeed in newly added qemu kvm-all.c kvm_arch_msi_data_to_gsi we could
> call a new ioctl that translates the data + deviceid? into an LPI and
> program irqfd with that LPI. This is done once when setting irqfd up.
> This also means extending irqfd support to lpi injection, gsi being the
> LPI index if gsi >= 8192. in that case we continue using
> kvm_gsi_direct_mapping and gsi still is an IRQ index.
This is exactly what i have done in my kernel + qemu. I have added a new KVM capability
and then in qemu i do this:
--- cut ---
if (kvm_gsi_kernel_mapping()) {
struct kvm_msi msi;
msi.address_lo = (uint32_t)msg.address;
msi.address_hi = msg.address >> 32;
msi.data = le32_to_cpu(msg.data);
memset(msi.pad, 0, sizeof(msi.pad));
if (dev) {
msi.devid = (pci_bus_num(dev->bus) << 8) | dev->devfn;
msi.flags = KVM_MSI_VALID_DEVID;
} else {
msi.devid = 0;
msi.flags = 0;
}
return kvm_vm_ioctl(s, KVM_TRANSLATE_MSI, &msi);
}
--- cut ---
KVM_TRANSLATE_MSI returns an LPI number. This seemed to be the simplest and fastest thing
to do.
If someone is interested, i could prepare an RFC patch series for this, which would apply
on top of Andre's ITS implementation.
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
^ permalink raw reply [flat|nested] 15+ messages in thread
* IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-10 13:58 ` Eric Auger
@ 2015-06-10 15:38 ` Pavel Fedin
-1 siblings, 0 replies; 15+ messages in thread
From: Pavel Fedin @ 2015-06-10 15:38 UTC (permalink / raw)
To: linux-arm-kernel
Hello!
> KVM GSI routing, even if only used for MSI routing then mandates to
> build entries for non MSI IRQs, using irqchip routing entries. Then you
> draw the irqchip.c kvm_irq_routing_table
> chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS] static allocation issue.
Sorry for this add-on, needed time to look at the code.
Actually, if we don't use this code at all, and implement our own kvm_set_irq_routing()
and kvm_free_irq_routing(), we don't have to bother about all these limitations.
The simplest thing to do there would be to store GSI number in struct its_itte. In this
case raising an MSI by GSI would not differ from what i currently do.
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
@ 2015-06-10 15:38 ` Pavel Fedin
0 siblings, 0 replies; 15+ messages in thread
From: Pavel Fedin @ 2015-06-10 15:38 UTC (permalink / raw)
To: 'Eric Auger', 'Marc Zyngier',
'Andre Przywara', christoffer.dall
Cc: kvmarm, linux-arm-kernel, kvm
Hello!
> KVM GSI routing, even if only used for MSI routing then mandates to
> build entries for non MSI IRQs, using irqchip routing entries. Then you
> draw the irqchip.c kvm_irq_routing_table
> chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS] static allocation issue.
Sorry for this add-on, needed time to look at the code.
Actually, if we don't use this code at all, and implement our own kvm_set_irq_routing()
and kvm_free_irq_routing(), we don't have to bother about all these limitations.
The simplest thing to do there would be to store GSI number in struct its_itte. In this
case raising an MSI by GSI would not differ from what i currently do.
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-10 15:30 ` Pavel Fedin
(?)
@ 2015-06-17 9:21 ` Pavel Fedin
2015-06-17 9:34 ` Marc Zyngier
2015-06-17 11:02 ` Andre Przywara
-1 siblings, 2 replies; 15+ messages in thread
From: Pavel Fedin @ 2015-06-17 9:21 UTC (permalink / raw)
To: 'Eric Auger', 'Marc Zyngier',
'Andre Przywara', christoffer.dall
Cc: kvm
PING!
The discussion has suddenly stopped... What is our status? Is ITS v2 patch being
developed, or what? And do we have some conclusion on irqfd ?
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
> -----Original Message-----
> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On Behalf Of Pavel
> Fedin
> Sent: Wednesday, June 10, 2015 6:30 PM
> To: 'Eric Auger'; 'Marc Zyngier'; 'Andre Przywara'; christoffer.dall@linaro.org
> Cc: kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org;
kvm@vger.kernel.org
> Subject: RE: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS
> emulation)
>
> Hi!
>
> > indeed in newly added qemu kvm-all.c kvm_arch_msi_data_to_gsi we could
> > call a new ioctl that translates the data + deviceid? into an LPI and
> > program irqfd with that LPI. This is done once when setting irqfd up.
> > This also means extending irqfd support to lpi injection, gsi being the
> > LPI index if gsi >= 8192. in that case we continue using
> > kvm_gsi_direct_mapping and gsi still is an IRQ index.
>
> This is exactly what i have done in my kernel + qemu. I have added a new KVM capability
> and then in qemu i do this:
> --- cut ---
> if (kvm_gsi_kernel_mapping()) {
> struct kvm_msi msi;
>
> msi.address_lo = (uint32_t)msg.address;
> msi.address_hi = msg.address >> 32;
> msi.data = le32_to_cpu(msg.data);
> memset(msi.pad, 0, sizeof(msi.pad));
>
> if (dev) {
> msi.devid = (pci_bus_num(dev->bus) << 8) | dev->devfn;
> msi.flags = KVM_MSI_VALID_DEVID;
> } else {
> msi.devid = 0;
> msi.flags = 0;
> }
>
> return kvm_vm_ioctl(s, KVM_TRANSLATE_MSI, &msi);
> }
> --- cut ---
> KVM_TRANSLATE_MSI returns an LPI number. This seemed to be the simplest and fastest
thing
> to do.
> If someone is interested, i could prepare an RFC patch series for this, which would
apply
> on top of Andre's ITS implementation.
>
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-17 9:21 ` Pavel Fedin
@ 2015-06-17 9:34 ` Marc Zyngier
2015-06-17 10:00 ` Pavel Fedin
2015-06-17 11:02 ` Andre Przywara
1 sibling, 1 reply; 15+ messages in thread
From: Marc Zyngier @ 2015-06-17 9:34 UTC (permalink / raw)
To: Pavel Fedin, 'Eric Auger', Andre Przywara,
christoffer.dall@linaro.org
Cc: kvm@vger.kernel.org
On 17/06/15 10:21, Pavel Fedin wrote:
> PING!
> The discussion has suddenly stopped... What is our status? Is ITS v2 patch being
> developed, or what? And do we have some conclusion on irqfd ?
Hmmm. You may not have noticed it, but we're actually all are quite busy
at the moment (hint, we're at -rc8, and the next merge window is about
to open).
As for the state of the ITS, it is still in review, and I expect Andre
will post an updated series after the merge window.
As for your irqfd proposal, see below:
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
>
>
>> -----Original Message-----
>> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On Behalf Of Pavel
>> Fedin
>> Sent: Wednesday, June 10, 2015 6:30 PM
>> To: 'Eric Auger'; 'Marc Zyngier'; 'Andre Przywara'; christoffer.dall@linaro.org
>> Cc: kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org;
> kvm@vger.kernel.org
>> Subject: RE: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS
>> emulation)
>>
>> Hi!
>>
>>> indeed in newly added qemu kvm-all.c kvm_arch_msi_data_to_gsi we could
>>> call a new ioctl that translates the data + deviceid? into an LPI and
>>> program irqfd with that LPI. This is done once when setting irqfd up.
>>> This also means extending irqfd support to lpi injection, gsi being the
>>> LPI index if gsi >= 8192. in that case we continue using
>>> kvm_gsi_direct_mapping and gsi still is an IRQ index.
>>
>> This is exactly what i have done in my kernel + qemu. I have added a new KVM capability
>> and then in qemu i do this:
>> --- cut ---
>> if (kvm_gsi_kernel_mapping()) {
>> struct kvm_msi msi;
>>
>> msi.address_lo = (uint32_t)msg.address;
>> msi.address_hi = msg.address >> 32;
>> msi.data = le32_to_cpu(msg.data);
>> memset(msi.pad, 0, sizeof(msi.pad));
>>
>> if (dev) {
>> msi.devid = (pci_bus_num(dev->bus) << 8) | dev->devfn;
>> msi.flags = KVM_MSI_VALID_DEVID;
>> } else {
>> msi.devid = 0;
>> msi.flags = 0;
>> }
>>
>> return kvm_vm_ioctl(s, KVM_TRANSLATE_MSI, &msi);
>> }
>> --- cut ---
>> KVM_TRANSLATE_MSI returns an LPI number. This seemed to be the simplest and fastest
> thing
>> to do.
>> If someone is interested, i could prepare an RFC patch series for this, which would
> apply
>> on top of Andre's ITS implementation.
This feels just wrong. The LPI number is under complete control of the
guest, and can be changed at any time. You can never rely on it to be
stable.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-17 9:34 ` Marc Zyngier
@ 2015-06-17 10:00 ` Pavel Fedin
0 siblings, 0 replies; 15+ messages in thread
From: Pavel Fedin @ 2015-06-17 10:00 UTC (permalink / raw)
To: 'Marc Zyngier', 'Eric Auger',
'Andre Przywara', christoffer.dall
Cc: kvm
Hello!
> Hmmm. You may not have noticed it, but we're actually all are quite busy
> at the moment (hint, we're at -rc8, and the next merge window is about
> to open).
Ok ok, i do not mind of course. :) Just i expected at least some, quick reply. It's like
talking to a person while he/she suddenly starts ignoring you and turns away without any
ACK/NAK. I simply do not know what is wrong.
> This feels just wrong. The LPI number is under complete control of the
> guest, and can be changed at any time. You can never rely on it to be
> stable.
Heh... Then i'm afraid the only option is the second one: GSI routing. It would allow to
associate an irqfd with MSI bunch (data + address + devID) as it is.
I'm also currently busy with some strange vhost-net performance issues, so i'll make
another RFC later, after i redo my implementation using routing.
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-17 9:21 ` Pavel Fedin
2015-06-17 9:34 ` Marc Zyngier
@ 2015-06-17 11:02 ` Andre Przywara
2015-06-17 11:46 ` Pavel Fedin
1 sibling, 1 reply; 15+ messages in thread
From: Andre Przywara @ 2015-06-17 11:02 UTC (permalink / raw)
To: Pavel Fedin, 'Eric Auger', Marc Zyngier,
christoffer.dall@linaro.org
Cc: kvm@vger.kernel.org
Здравствуй Pavel,
On 06/17/2015 10:21 AM, Pavel Fedin wrote:
> PING!
> The discussion has suddenly stopped... What is our status? Is ITS v2 patch being
> developed, or what?
Yes, I am about to get a v2 ready, but mostly with some fixes. If you
want to work on top of it, I can push a WIP branch to my repo.
As Marc mentioned before, this whole irqfd story does not go together
well with KVM and the ITS architecture, so that needs some more
investigation (which I am planning to do in the next days).
Cheers,
Andre.
And do we have some conclusion on irqfd ?
>
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
>
>
>> -----Original Message-----
>> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On Behalf Of Pavel
>> Fedin
>> Sent: Wednesday, June 10, 2015 6:30 PM
>> To: 'Eric Auger'; 'Marc Zyngier'; 'Andre Przywara'; christoffer.dall@linaro.org
>> Cc: kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org;
> kvm@vger.kernel.org
>> Subject: RE: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS
>> emulation)
>>
>> Hi!
>>
>>> indeed in newly added qemu kvm-all.c kvm_arch_msi_data_to_gsi we could
>>> call a new ioctl that translates the data + deviceid? into an LPI and
>>> program irqfd with that LPI. This is done once when setting irqfd up.
>>> This also means extending irqfd support to lpi injection, gsi being the
>>> LPI index if gsi >= 8192. in that case we continue using
>>> kvm_gsi_direct_mapping and gsi still is an IRQ index.
>>
>> This is exactly what i have done in my kernel + qemu. I have added a new KVM capability
>> and then in qemu i do this:
>> --- cut ---
>> if (kvm_gsi_kernel_mapping()) {
>> struct kvm_msi msi;
>>
>> msi.address_lo = (uint32_t)msg.address;
>> msi.address_hi = msg.address >> 32;
>> msi.data = le32_to_cpu(msg.data);
>> memset(msi.pad, 0, sizeof(msi.pad));
>>
>> if (dev) {
>> msi.devid = (pci_bus_num(dev->bus) << 8) | dev->devfn;
>> msi.flags = KVM_MSI_VALID_DEVID;
>> } else {
>> msi.devid = 0;
>> msi.flags = 0;
>> }
>>
>> return kvm_vm_ioctl(s, KVM_TRANSLATE_MSI, &msi);
>> }
>> --- cut ---
>> KVM_TRANSLATE_MSI returns an LPI number. This seemed to be the simplest and fastest
> thing
>> to do.
>> If someone is interested, i could prepare an RFC patch series for this, which would
> apply
>> on top of Andre's ITS implementation.
>>
>> Kind regards,
>> Pavel Fedin
>> Expert Engineer
>> Samsung Electronics Research center Russia
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-17 11:02 ` Andre Przywara
@ 2015-06-17 11:46 ` Pavel Fedin
2015-06-18 6:35 ` Eric Auger
0 siblings, 1 reply; 15+ messages in thread
From: Pavel Fedin @ 2015-06-17 11:46 UTC (permalink / raw)
To: 'Andre Przywara', 'Eric Auger',
'Marc Zyngier', christoffer.dall
Cc: kvm
Hello!
> Yes, I am about to get a v2 ready, but mostly with some fixes. If you
> want to work on top of it, I can push a WIP branch to my repo.
Thank you but no need to hurry up. I am busy with other things too. And, anyway, i work on top of my own branch here.
> As Marc mentioned before, this whole irqfd story does not go together
> well with KVM and the ITS architecture,
It actually does with GSI routing (see http://www.spinics.net/lists/kvm/msg117475.html, approach 2).
> so that needs some more
> investigation (which I am planning to do in the next days).
Ok.
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-17 11:46 ` Pavel Fedin
@ 2015-06-18 6:35 ` Eric Auger
2015-06-18 14:19 ` Pavel Fedin
0 siblings, 1 reply; 15+ messages in thread
From: Eric Auger @ 2015-06-18 6:35 UTC (permalink / raw)
To: Pavel Fedin, 'Andre Przywara', 'Marc Zyngier',
christoffer.dall
Cc: kvm
On 06/17/2015 01:46 PM, Pavel Fedin wrote:
> Hello!
>
>> Yes, I am about to get a v2 ready, but mostly with some fixes. If you
>> want to work on top of it, I can push a WIP branch to my repo.
>
> Thank you but no need to hurry up. I am busy with other things too. And, anyway, i work on top of my own branch here.
>
>> As Marc mentioned before, this whole irqfd story does not go together
>> well with KVM and the ITS architecture,
>
> It actually does with GSI routing (see http://www.spinics.net/lists/kvm/msg117475.html, approach 2).
Hi Pavel,
I also have an implementation of GSI routing on ARM, basically a rebase
of my old/first implementation of irqfd
(https://patches.linaro.org/32261/) based on irqchip gsi routing & qemu
part (https://lists.gnu.org/archive/html/qemu-devel/2014-07/msg01090.html).
I will repost today as a base of discussion for evolutions.
Best Regards
Eric
>
>> so that needs some more
>> investigation (which I am planning to do in the next days).
>
> Ok.
>
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation)
2015-06-18 6:35 ` Eric Auger
@ 2015-06-18 14:19 ` Pavel Fedin
0 siblings, 0 replies; 15+ messages in thread
From: Pavel Fedin @ 2015-06-18 14:19 UTC (permalink / raw)
To: 'Eric Auger', 'Andre Przywara',
'Marc Zyngier', christoffer.dall
Cc: kvm
Hello!
> I also have an implementation of GSI routing on ARM, basically a rebase
> of my old/first implementation of irqfd
> (https://patches.linaro.org/32261/) based on irqchip gsi routing & qemu
> part (https://lists.gnu.org/archive/html/qemu-devel/2014-07/msg01090.html).
I took a glance at it, and looks like it's already obsolete. We already have a convention of GSI number == SPI number. Kind of hardcoded default routing table which cannot be changed. It is used at least by GICv2m emulation.
I think we should maintain backwards compatibility with it. I thought about something like:
a) GSI < 8192 - correspond to SPIs and cannot be re-routed.
b) GSI >= 8192 - correspond to MSI and need to be routed before use.
During routing setup we could use either GSI with offset (starting from 8192), or raw number (starting from 0). In case of raw number we would have some complex structure of GSI field in KVM_CAP_IRQFD ioctl, similar to KVM_IRQ_LINE. Something like:
bits: | 31 ... 24 | 23 ... 0 |
field: | irq_type | irq_id |
irq_type[0]: irq_id = SPI
irq_type[3]: irq_id = GSI number routed to MSI
Consequently, we have to implement only KVM_IRQ_ROUTING_MSI type and completely ignore KVM_IRQ_ROUTING_IRQCHIP.
I hope i am clear enough...
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2015-06-18 14:19 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-10 8:31 IRQFD support with GICv3 ITS (WAS: RE: [PATCH 00/13] arm64: KVM: GICv3 ITS emulation) Pavel Fedin
2015-06-10 8:31 ` Pavel Fedin
2015-06-10 13:58 ` Eric Auger
2015-06-10 13:58 ` Eric Auger
2015-06-10 15:30 ` Pavel Fedin
2015-06-10 15:30 ` Pavel Fedin
2015-06-17 9:21 ` Pavel Fedin
2015-06-17 9:34 ` Marc Zyngier
2015-06-17 10:00 ` Pavel Fedin
2015-06-17 11:02 ` Andre Przywara
2015-06-17 11:46 ` Pavel Fedin
2015-06-18 6:35 ` Eric Auger
2015-06-18 14:19 ` Pavel Fedin
2015-06-10 15:38 ` Pavel Fedin
2015-06-10 15:38 ` Pavel Fedin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.