* [PATCH] KVM: s390: Enable specification exception interpretation
@ 2021-07-06 11:47 Janis Schoetterl-Glausch
2021-07-06 11:52 ` Cornelia Huck
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Janis Schoetterl-Glausch @ 2021-07-06 11:47 UTC (permalink / raw
To: Christian Borntraeger, Janosch Frank, Heiko Carstens,
Vasily Gorbik
Cc: Janis Schoetterl-Glausch, David Hildenbrand, Cornelia Huck,
Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
When this feature is enabled the hardware is free to interpret
specification exceptions generated by the guest, instead of causing
program interruption interceptions.
This benefits (test) programs that generate a lot of specification
exceptions (roughly 4x increase in exceptions/sec).
Interceptions will occur as before if ICTL_PINT is set,
i.e. if guest debug is enabled.
Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
---
I'll additionally send kvm-unit-tests for testing this feature.
arch/s390/include/asm/kvm_host.h | 1 +
arch/s390/kvm/kvm-s390.c | 2 ++
arch/s390/kvm/vsie.c | 2 ++
3 files changed, 5 insertions(+)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 9b4473f76e56..3a5b5084cdbe 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -244,6 +244,7 @@ struct kvm_s390_sie_block {
__u8 fpf; /* 0x0060 */
#define ECB_GS 0x40
#define ECB_TE 0x10
+#define ECB_SPECI 0x08
#define ECB_SRSI 0x04
#define ECB_HOSTPROTINT 0x02
__u8 ecb; /* 0x0061 */
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index b655a7d82bf0..aadd589a3755 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
vcpu->arch.sie_block->ecb |= ECB_SRSI;
if (test_kvm_facility(vcpu->kvm, 73))
vcpu->arch.sie_block->ecb |= ECB_TE;
+ if (!kvm_is_ucontrol(vcpu->kvm))
+ vcpu->arch.sie_block->ecb |= ECB_SPECI;
if (test_kvm_facility(vcpu->kvm, 8) && vcpu->kvm->arch.use_pfmfi)
vcpu->arch.sie_block->ecb2 |= ECB2_PFMFI;
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 4002a24bc43a..acda4b6fc851 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -510,6 +510,8 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
prefix_unmapped(vsie_page);
scb_s->ecb |= ECB_TE;
}
+ /* specification exception interpretation */
+ scb_s->ecb |= scb_o->ecb & ECB_SPECI;
/* branch prediction */
if (test_kvm_facility(vcpu->kvm, 82))
scb_s->fpf |= scb_o->fpf & FPF_BPBC;
--
2.25.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-06 11:47 [PATCH] KVM: s390: Enable specification exception interpretation Janis Schoetterl-Glausch
@ 2021-07-06 11:52 ` Cornelia Huck
2021-07-06 11:56 ` Christian Borntraeger
2021-07-06 11:55 ` David Hildenbrand
2021-07-07 8:30 ` Christian Borntraeger
2 siblings, 1 reply; 15+ messages in thread
From: Cornelia Huck @ 2021-07-06 11:52 UTC (permalink / raw
To: Janis Schoetterl-Glausch, Christian Borntraeger, Janosch Frank,
Heiko Carstens, Vasily Gorbik
Cc: Janis Schoetterl-Glausch, David Hildenbrand, Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On Tue, Jul 06 2021, Janis Schoetterl-Glausch <scgl@linux.ibm.com> wrote:
> When this feature is enabled the hardware is free to interpret
> specification exceptions generated by the guest, instead of causing
> program interruption interceptions.
>
> This benefits (test) programs that generate a lot of specification
> exceptions (roughly 4x increase in exceptions/sec).
>
> Interceptions will occur as before if ICTL_PINT is set,
> i.e. if guest debug is enabled.
>
> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
> ---
> I'll additionally send kvm-unit-tests for testing this feature.
>
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/s390/kvm/kvm-s390.c | 2 ++
> arch/s390/kvm/vsie.c | 2 ++
> 3 files changed, 5 insertions(+)
(...)
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index b655a7d82bf0..aadd589a3755 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
> vcpu->arch.sie_block->ecb |= ECB_SRSI;
> if (test_kvm_facility(vcpu->kvm, 73))
> vcpu->arch.sie_block->ecb |= ECB_TE;
> + if (!kvm_is_ucontrol(vcpu->kvm))
> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
Does this exist for any hardware version (i.e. not guarded by a cpu
feature?)
>
> if (test_kvm_facility(vcpu->kvm, 8) && vcpu->kvm->arch.use_pfmfi)
> vcpu->arch.sie_block->ecb2 |= ECB2_PFMFI;
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-06 11:47 [PATCH] KVM: s390: Enable specification exception interpretation Janis Schoetterl-Glausch
2021-07-06 11:52 ` Cornelia Huck
@ 2021-07-06 11:55 ` David Hildenbrand
2021-07-07 8:30 ` Christian Borntraeger
2 siblings, 0 replies; 15+ messages in thread
From: David Hildenbrand @ 2021-07-06 11:55 UTC (permalink / raw
To: Janis Schoetterl-Glausch, Christian Borntraeger, Janosch Frank,
Heiko Carstens, Vasily Gorbik
Cc: Cornelia Huck, Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 06.07.21 13:47, Janis Schoetterl-Glausch wrote:
> When this feature is enabled the hardware is free to interpret
> specification exceptions generated by the guest, instead of causing
> program interruption interceptions.
>
> This benefits (test) programs that generate a lot of specification
> exceptions (roughly 4x increase in exceptions/sec).
>
> Interceptions will occur as before if ICTL_PINT is set,
> i.e. if guest debug is enabled.
>
> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
> ---
> I'll additionally send kvm-unit-tests for testing this feature.
>
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/s390/kvm/kvm-s390.c | 2 ++
> arch/s390/kvm/vsie.c | 2 ++
> 3 files changed, 5 insertions(+)
>
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 9b4473f76e56..3a5b5084cdbe 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -244,6 +244,7 @@ struct kvm_s390_sie_block {
> __u8 fpf; /* 0x0060 */
> #define ECB_GS 0x40
> #define ECB_TE 0x10
> +#define ECB_SPECI 0x08
> #define ECB_SRSI 0x04
> #define ECB_HOSTPROTINT 0x02
> __u8 ecb; /* 0x0061 */
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index b655a7d82bf0..aadd589a3755 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
> vcpu->arch.sie_block->ecb |= ECB_SRSI;
> if (test_kvm_facility(vcpu->kvm, 73))
> vcpu->arch.sie_block->ecb |= ECB_TE;
> + if (!kvm_is_ucontrol(vcpu->kvm))
> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>
> if (test_kvm_facility(vcpu->kvm, 8) && vcpu->kvm->arch.use_pfmfi)
> vcpu->arch.sie_block->ecb2 |= ECB2_PFMFI;
> diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
> index 4002a24bc43a..acda4b6fc851 100644
> --- a/arch/s390/kvm/vsie.c
> +++ b/arch/s390/kvm/vsie.c
> @@ -510,6 +510,8 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> prefix_unmapped(vsie_page);
> scb_s->ecb |= ECB_TE;
> }
> + /* specification exception interpretation */
> + scb_s->ecb |= scb_o->ecb & ECB_SPECI;
> /* branch prediction */
> if (test_kvm_facility(vcpu->kvm, 82))
> scb_s->fpf |= scb_o->fpf & FPF_BPBC;
>
I assume this is a new CPU feature, right? If so
a) How can we check whether we can actually safely enable it. (which
facility do we have to check)
b) Do we have to handle vSIE? Do we have to indicate a CPU feature that
unlocks this feature?
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-06 11:52 ` Cornelia Huck
@ 2021-07-06 11:56 ` Christian Borntraeger
2021-07-06 11:59 ` David Hildenbrand
0 siblings, 1 reply; 15+ messages in thread
From: Christian Borntraeger @ 2021-07-06 11:56 UTC (permalink / raw
To: Cornelia Huck, Janis Schoetterl-Glausch, Janosch Frank,
Heiko Carstens, Vasily Gorbik
Cc: David Hildenbrand, Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 06.07.21 13:52, Cornelia Huck wrote:
> On Tue, Jul 06 2021, Janis Schoetterl-Glausch <scgl@linux.ibm.com> wrote:
>
>> When this feature is enabled the hardware is free to interpret
>> specification exceptions generated by the guest, instead of causing
>> program interruption interceptions.
>>
>> This benefits (test) programs that generate a lot of specification
>> exceptions (roughly 4x increase in exceptions/sec).
>>
>> Interceptions will occur as before if ICTL_PINT is set,
>> i.e. if guest debug is enabled.
>>
>> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
>> ---
>> I'll additionally send kvm-unit-tests for testing this feature.
>>
>> arch/s390/include/asm/kvm_host.h | 1 +
>> arch/s390/kvm/kvm-s390.c | 2 ++
>> arch/s390/kvm/vsie.c | 2 ++
>> 3 files changed, 5 insertions(+)
>
> (...)
>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index b655a7d82bf0..aadd589a3755 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>> vcpu->arch.sie_block->ecb |= ECB_SRSI;
>> if (test_kvm_facility(vcpu->kvm, 73))
>> vcpu->arch.sie_block->ecb |= ECB_TE;
>> + if (!kvm_is_ucontrol(vcpu->kvm))
>> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>
> Does this exist for any hardware version (i.e. not guarded by a cpu
> feature?)
Not for all hardware versions, but also no indication. The architecture
says that the HW is free to do this or not. (which makes the vsie code
simpler).
>
>>
>> if (test_kvm_facility(vcpu->kvm, 8) && vcpu->kvm->arch.use_pfmfi)
>> vcpu->arch.sie_block->ecb2 |= ECB2_PFMFI;
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-06 11:56 ` Christian Borntraeger
@ 2021-07-06 11:59 ` David Hildenbrand
2021-07-06 12:02 ` Christian Borntraeger
0 siblings, 1 reply; 15+ messages in thread
From: David Hildenbrand @ 2021-07-06 11:59 UTC (permalink / raw
To: Christian Borntraeger, Cornelia Huck, Janis Schoetterl-Glausch,
Janosch Frank, Heiko Carstens, Vasily Gorbik
Cc: Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 06.07.21 13:56, Christian Borntraeger wrote:
>
>
> On 06.07.21 13:52, Cornelia Huck wrote:
>> On Tue, Jul 06 2021, Janis Schoetterl-Glausch <scgl@linux.ibm.com> wrote:
>>
>>> When this feature is enabled the hardware is free to interpret
>>> specification exceptions generated by the guest, instead of causing
>>> program interruption interceptions.
>>>
>>> This benefits (test) programs that generate a lot of specification
>>> exceptions (roughly 4x increase in exceptions/sec).
>>>
>>> Interceptions will occur as before if ICTL_PINT is set,
>>> i.e. if guest debug is enabled.
>>>
>>> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
>>> ---
>>> I'll additionally send kvm-unit-tests for testing this feature.
>>>
>>> arch/s390/include/asm/kvm_host.h | 1 +
>>> arch/s390/kvm/kvm-s390.c | 2 ++
>>> arch/s390/kvm/vsie.c | 2 ++
>>> 3 files changed, 5 insertions(+)
>>
>> (...)
>>
>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>> index b655a7d82bf0..aadd589a3755 100644
>>> --- a/arch/s390/kvm/kvm-s390.c
>>> +++ b/arch/s390/kvm/kvm-s390.c
>>> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>>> vcpu->arch.sie_block->ecb |= ECB_SRSI;
>>> if (test_kvm_facility(vcpu->kvm, 73))
>>> vcpu->arch.sie_block->ecb |= ECB_TE;
>>> + if (!kvm_is_ucontrol(vcpu->kvm))
>>> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>>
>> Does this exist for any hardware version (i.e. not guarded by a cpu
>> feature?)
>
> Not for all hardware versions, but also no indication. The architecture
> says that the HW is free to do this or not. (which makes the vsie code
> simpler).
I remember the architecture said at some point to never set undefined
bits - and this bit is undefined on older HW generations. I might be
wrong, though.
(I though HW learned the lesson to always use proper feature indications
along with new features)
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-06 11:59 ` David Hildenbrand
@ 2021-07-06 12:02 ` Christian Borntraeger
2021-07-06 15:16 ` David Hildenbrand
0 siblings, 1 reply; 15+ messages in thread
From: Christian Borntraeger @ 2021-07-06 12:02 UTC (permalink / raw
To: David Hildenbrand, Cornelia Huck, Janis Schoetterl-Glausch,
Janosch Frank, Heiko Carstens, Vasily Gorbik
Cc: Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 06.07.21 13:59, David Hildenbrand wrote:
> On 06.07.21 13:56, Christian Borntraeger wrote:
>>
>>
>> On 06.07.21 13:52, Cornelia Huck wrote:
>>> On Tue, Jul 06 2021, Janis Schoetterl-Glausch <scgl@linux.ibm.com> wrote:
>>>
>>>> When this feature is enabled the hardware is free to interpret
>>>> specification exceptions generated by the guest, instead of causing
>>>> program interruption interceptions.
>>>>
>>>> This benefits (test) programs that generate a lot of specification
>>>> exceptions (roughly 4x increase in exceptions/sec).
>>>>
>>>> Interceptions will occur as before if ICTL_PINT is set,
>>>> i.e. if guest debug is enabled.
>>>>
>>>> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
>>>> ---
>>>> I'll additionally send kvm-unit-tests for testing this feature.
>>>>
>>>> arch/s390/include/asm/kvm_host.h | 1 +
>>>> arch/s390/kvm/kvm-s390.c | 2 ++
>>>> arch/s390/kvm/vsie.c | 2 ++
>>>> 3 files changed, 5 insertions(+)
>>>
>>> (...)
>>>
>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>>> index b655a7d82bf0..aadd589a3755 100644
>>>> --- a/arch/s390/kvm/kvm-s390.c
>>>> +++ b/arch/s390/kvm/kvm-s390.c
>>>> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>>>> vcpu->arch.sie_block->ecb |= ECB_SRSI;
>>>> if (test_kvm_facility(vcpu->kvm, 73))
>>>> vcpu->arch.sie_block->ecb |= ECB_TE;
>>>> + if (!kvm_is_ucontrol(vcpu->kvm))
>>>> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>>>
>>> Does this exist for any hardware version (i.e. not guarded by a cpu
>>> feature?)
>>
>> Not for all hardware versions, but also no indication. The architecture
>> says that the HW is free to do this or not. (which makes the vsie code
>> simpler).
>
> I remember the architecture said at some point to never set undefined bits - and this bit is undefined on older HW generations. I might be wrong, though.
I can confirm that this bit will be ignored on older machines. The notion of
never setting undefined bits comes from "you never know what this bit will
change in future machines". Now we know :-)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-06 12:02 ` Christian Borntraeger
@ 2021-07-06 15:16 ` David Hildenbrand
2021-07-06 15:27 ` Janis Schoetterl-Glausch
0 siblings, 1 reply; 15+ messages in thread
From: David Hildenbrand @ 2021-07-06 15:16 UTC (permalink / raw
To: Christian Borntraeger, Cornelia Huck, Janis Schoetterl-Glausch,
Janosch Frank, Heiko Carstens, Vasily Gorbik
Cc: Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 06.07.21 14:02, Christian Borntraeger wrote:
>
>
> On 06.07.21 13:59, David Hildenbrand wrote:
>> On 06.07.21 13:56, Christian Borntraeger wrote:
>>>
>>>
>>> On 06.07.21 13:52, Cornelia Huck wrote:
>>>> On Tue, Jul 06 2021, Janis Schoetterl-Glausch <scgl@linux.ibm.com> wrote:
>>>>
>>>>> When this feature is enabled the hardware is free to interpret
>>>>> specification exceptions generated by the guest, instead of causing
>>>>> program interruption interceptions.
>>>>>
>>>>> This benefits (test) programs that generate a lot of specification
>>>>> exceptions (roughly 4x increase in exceptions/sec).
>>>>>
>>>>> Interceptions will occur as before if ICTL_PINT is set,
>>>>> i.e. if guest debug is enabled.
>>>>>
>>>>> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
>>>>> ---
>>>>> I'll additionally send kvm-unit-tests for testing this feature.
>>>>>
>>>>> arch/s390/include/asm/kvm_host.h | 1 +
>>>>> arch/s390/kvm/kvm-s390.c | 2 ++
>>>>> arch/s390/kvm/vsie.c | 2 ++
>>>>> 3 files changed, 5 insertions(+)
>>>>
>>>> (...)
>>>>
>>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>>>> index b655a7d82bf0..aadd589a3755 100644
>>>>> --- a/arch/s390/kvm/kvm-s390.c
>>>>> +++ b/arch/s390/kvm/kvm-s390.c
>>>>> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>>>>> vcpu->arch.sie_block->ecb |= ECB_SRSI;
>>>>> if (test_kvm_facility(vcpu->kvm, 73))
>>>>> vcpu->arch.sie_block->ecb |= ECB_TE;
>>>>> + if (!kvm_is_ucontrol(vcpu->kvm))
>>>>> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>>>>
>>>> Does this exist for any hardware version (i.e. not guarded by a cpu
>>>> feature?)
>>>
>>> Not for all hardware versions, but also no indication. The architecture
>>> says that the HW is free to do this or not. (which makes the vsie code
>>> simpler).
>>
>> I remember the architecture said at some point to never set undefined bits - and this bit is undefined on older HW generations. I might be wrong, though.
>
> I can confirm that this bit will be ignored on older machines. The notion of
> never setting undefined bits comes from "you never know what this bit will
> change in future machines". Now we know :-)
Well, okay then :)
So the plan for vSIE is to always keep it disabled? IIUC, one could
similarly always forward the bit of set.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-06 15:16 ` David Hildenbrand
@ 2021-07-06 15:27 ` Janis Schoetterl-Glausch
2021-07-07 7:26 ` David Hildenbrand
0 siblings, 1 reply; 15+ messages in thread
From: Janis Schoetterl-Glausch @ 2021-07-06 15:27 UTC (permalink / raw
To: David Hildenbrand, Christian Borntraeger, Cornelia Huck,
Janis Schoetterl-Glausch, Janosch Frank, Heiko Carstens,
Vasily Gorbik
Cc: Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 7/6/21 5:16 PM, David Hildenbrand wrote:
> On 06.07.21 14:02, Christian Borntraeger wrote:
>>
>>
>> On 06.07.21 13:59, David Hildenbrand wrote:
>>> On 06.07.21 13:56, Christian Borntraeger wrote:
>>>>
>>>>
>>>> On 06.07.21 13:52, Cornelia Huck wrote:
>>>>> On Tue, Jul 06 2021, Janis Schoetterl-Glausch <scgl@linux.ibm.com> wrote:
>>>>>
>>>>>> When this feature is enabled the hardware is free to interpret
>>>>>> specification exceptions generated by the guest, instead of causing
>>>>>> program interruption interceptions.
>>>>>>
>>>>>> This benefits (test) programs that generate a lot of specification
>>>>>> exceptions (roughly 4x increase in exceptions/sec).
>>>>>>
>>>>>> Interceptions will occur as before if ICTL_PINT is set,
>>>>>> i.e. if guest debug is enabled.
>>>>>>
>>>>>> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
>>>>>> ---
>>>>>> I'll additionally send kvm-unit-tests for testing this feature.
>>>>>>
>>>>>> arch/s390/include/asm/kvm_host.h | 1 +
>>>>>> arch/s390/kvm/kvm-s390.c | 2 ++
>>>>>> arch/s390/kvm/vsie.c | 2 ++
>>>>>> 3 files changed, 5 insertions(+)
>>>>>
>>>>> (...)
>>>>>
>>>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>>>>> index b655a7d82bf0..aadd589a3755 100644
>>>>>> --- a/arch/s390/kvm/kvm-s390.c
>>>>>> +++ b/arch/s390/kvm/kvm-s390.c
>>>>>> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>>>>>> vcpu->arch.sie_block->ecb |= ECB_SRSI;
>>>>>> if (test_kvm_facility(vcpu->kvm, 73))
>>>>>> vcpu->arch.sie_block->ecb |= ECB_TE;
>>>>>> + if (!kvm_is_ucontrol(vcpu->kvm))
>>>>>> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>>>>>
>>>>> Does this exist for any hardware version (i.e. not guarded by a cpu
>>>>> feature?)
>>>>
>>>> Not for all hardware versions, but also no indication. The architecture
>>>> says that the HW is free to do this or not. (which makes the vsie code
>>>> simpler).
>>>
>>> I remember the architecture said at some point to never set undefined bits - and this bit is undefined on older HW generations. I might be wrong, though.
>>
>> I can confirm that this bit will be ignored on older machines. The notion of
>> never setting undefined bits comes from "you never know what this bit will
>> change in future machines". Now we know :-)
>
> Well, okay then :)
>
> So the plan for vSIE is to always keep it disabled? IIUC, one could similarly always forward the bit of set.
The bit does get copied for vSIE.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-06 15:27 ` Janis Schoetterl-Glausch
@ 2021-07-07 7:26 ` David Hildenbrand
0 siblings, 0 replies; 15+ messages in thread
From: David Hildenbrand @ 2021-07-07 7:26 UTC (permalink / raw
To: Janis Schoetterl-Glausch, Christian Borntraeger, Cornelia Huck,
Janis Schoetterl-Glausch, Janosch Frank, Heiko Carstens,
Vasily Gorbik
Cc: Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 06.07.21 17:27, Janis Schoetterl-Glausch wrote:
> On 7/6/21 5:16 PM, David Hildenbrand wrote:
>> On 06.07.21 14:02, Christian Borntraeger wrote:
>>>
>>>
>>> On 06.07.21 13:59, David Hildenbrand wrote:
>>>> On 06.07.21 13:56, Christian Borntraeger wrote:
>>>>>
>>>>>
>>>>> On 06.07.21 13:52, Cornelia Huck wrote:
>>>>>> On Tue, Jul 06 2021, Janis Schoetterl-Glausch <scgl@linux.ibm.com> wrote:
>>>>>>
>>>>>>> When this feature is enabled the hardware is free to interpret
>>>>>>> specification exceptions generated by the guest, instead of causing
>>>>>>> program interruption interceptions.
>>>>>>>
>>>>>>> This benefits (test) programs that generate a lot of specification
>>>>>>> exceptions (roughly 4x increase in exceptions/sec).
>>>>>>>
>>>>>>> Interceptions will occur as before if ICTL_PINT is set,
>>>>>>> i.e. if guest debug is enabled.
>>>>>>>
>>>>>>> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
>>>>>>> ---
>>>>>>> I'll additionally send kvm-unit-tests for testing this feature.
>>>>>>>
>>>>>>> arch/s390/include/asm/kvm_host.h | 1 +
>>>>>>> arch/s390/kvm/kvm-s390.c | 2 ++
>>>>>>> arch/s390/kvm/vsie.c | 2 ++
>>>>>>> 3 files changed, 5 insertions(+)
>>>>>>
>>>>>> (...)
>>>>>>
>>>>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>>>>>> index b655a7d82bf0..aadd589a3755 100644
>>>>>>> --- a/arch/s390/kvm/kvm-s390.c
>>>>>>> +++ b/arch/s390/kvm/kvm-s390.c
>>>>>>> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>>>>>>> vcpu->arch.sie_block->ecb |= ECB_SRSI;
>>>>>>> if (test_kvm_facility(vcpu->kvm, 73))
>>>>>>> vcpu->arch.sie_block->ecb |= ECB_TE;
>>>>>>> + if (!kvm_is_ucontrol(vcpu->kvm))
>>>>>>> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>>>>>>
>>>>>> Does this exist for any hardware version (i.e. not guarded by a cpu
>>>>>> feature?)
>>>>>
>>>>> Not for all hardware versions, but also no indication. The architecture
>>>>> says that the HW is free to do this or not. (which makes the vsie code
>>>>> simpler).
>>>>
>>>> I remember the architecture said at some point to never set undefined bits - and this bit is undefined on older HW generations. I might be wrong, though.
>>>
>>> I can confirm that this bit will be ignored on older machines. The notion of
>>> never setting undefined bits comes from "you never know what this bit will
>>> change in future machines". Now we know :-)
>>
>> Well, okay then :)
>>
>> So the plan for vSIE is to always keep it disabled? IIUC, one could similarly always forward the bit of set.
>
> The bit does get copied for vSIE.
... and I missed that hunk :)
LGTM then
Reviewed-by: David Hildenbrand <david@redhat.com>
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-06 11:47 [PATCH] KVM: s390: Enable specification exception interpretation Janis Schoetterl-Glausch
2021-07-06 11:52 ` Cornelia Huck
2021-07-06 11:55 ` David Hildenbrand
@ 2021-07-07 8:30 ` Christian Borntraeger
2021-07-07 8:54 ` Cornelia Huck
2021-07-07 8:56 ` Janis Schoetterl-Glausch
2 siblings, 2 replies; 15+ messages in thread
From: Christian Borntraeger @ 2021-07-07 8:30 UTC (permalink / raw
To: Janis Schoetterl-Glausch, Janosch Frank, Heiko Carstens,
Vasily Gorbik
Cc: David Hildenbrand, Cornelia Huck, Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 06.07.21 13:47, Janis Schoetterl-Glausch wrote:
> When this feature is enabled the hardware is free to interpret
> specification exceptions generated by the guest, instead of causing
> program interruption interceptions.
>
> This benefits (test) programs that generate a lot of specification
> exceptions (roughly 4x increase in exceptions/sec).
>
> Interceptions will occur as before if ICTL_PINT is set,
> i.e. if guest debug is enabled.
I think I will add
There is no indication if this feature is available or not and the hardware
is free to interpret or not. So we can simply set this bit and if the
hardware ignores it we fall back to intercept 8 handling.
With that
Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
>
> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
> ---
> I'll additionally send kvm-unit-tests for testing this feature.
>
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/s390/kvm/kvm-s390.c | 2 ++
> arch/s390/kvm/vsie.c | 2 ++
> 3 files changed, 5 insertions(+)
>
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 9b4473f76e56..3a5b5084cdbe 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -244,6 +244,7 @@ struct kvm_s390_sie_block {
> __u8 fpf; /* 0x0060 */
> #define ECB_GS 0x40
> #define ECB_TE 0x10
> +#define ECB_SPECI 0x08
> #define ECB_SRSI 0x04
> #define ECB_HOSTPROTINT 0x02
> __u8 ecb; /* 0x0061 */
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index b655a7d82bf0..aadd589a3755 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
> vcpu->arch.sie_block->ecb |= ECB_SRSI;
> if (test_kvm_facility(vcpu->kvm, 73))
> vcpu->arch.sie_block->ecb |= ECB_TE;
> + if (!kvm_is_ucontrol(vcpu->kvm))
> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>
> if (test_kvm_facility(vcpu->kvm, 8) && vcpu->kvm->arch.use_pfmfi)
> vcpu->arch.sie_block->ecb2 |= ECB2_PFMFI;
> diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
> index 4002a24bc43a..acda4b6fc851 100644
> --- a/arch/s390/kvm/vsie.c
> +++ b/arch/s390/kvm/vsie.c
> @@ -510,6 +510,8 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> prefix_unmapped(vsie_page);
> scb_s->ecb |= ECB_TE;
> }
> + /* specification exception interpretation */
> + scb_s->ecb |= scb_o->ecb & ECB_SPECI;
> /* branch prediction */
> if (test_kvm_facility(vcpu->kvm, 82))
> scb_s->fpf |= scb_o->fpf & FPF_BPBC;
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-07 8:30 ` Christian Borntraeger
@ 2021-07-07 8:54 ` Cornelia Huck
2021-07-07 9:55 ` Janis Schoetterl-Glausch
2021-07-07 8:56 ` Janis Schoetterl-Glausch
1 sibling, 1 reply; 15+ messages in thread
From: Cornelia Huck @ 2021-07-07 8:54 UTC (permalink / raw
To: Christian Borntraeger, Janis Schoetterl-Glausch, Janosch Frank,
Heiko Carstens, Vasily Gorbik
Cc: David Hildenbrand, Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On Wed, Jul 07 2021, Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> On 06.07.21 13:47, Janis Schoetterl-Glausch wrote:
>> When this feature is enabled the hardware is free to interpret
>> specification exceptions generated by the guest, instead of causing
>> program interruption interceptions.
>>
>> This benefits (test) programs that generate a lot of specification
>> exceptions (roughly 4x increase in exceptions/sec).
>>
>> Interceptions will occur as before if ICTL_PINT is set,
>> i.e. if guest debug is enabled.
>
> I think I will add
>
> There is no indication if this feature is available or not and the hardware
> is free to interpret or not. So we can simply set this bit and if the
> hardware ignores it we fall back to intercept 8 handling.
Sounds good.
>
>
> With that
>
> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>
>> Signed-off-by: Janis Schoetterl-Glausch <scgl@linux.ibm.com>
>> ---
>> I'll additionally send kvm-unit-tests for testing this feature.
>>
>> arch/s390/include/asm/kvm_host.h | 1 +
>> arch/s390/kvm/kvm-s390.c | 2 ++
>> arch/s390/kvm/vsie.c | 2 ++
>> 3 files changed, 5 insertions(+)
(...)
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index b655a7d82bf0..aadd589a3755 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>> vcpu->arch.sie_block->ecb |= ECB_SRSI;
>> if (test_kvm_facility(vcpu->kvm, 73))
>> vcpu->arch.sie_block->ecb |= ECB_TE;
Maybe add
/* no facility bit, but safe as the hardware may ignore it */
or something like that, so that we don't stumble over that in the future?
>> + if (!kvm_is_ucontrol(vcpu->kvm))
>> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>>
>> if (test_kvm_facility(vcpu->kvm, 8) && vcpu->kvm->arch.use_pfmfi)
>> vcpu->arch.sie_block->ecb2 |= ECB2_PFMFI;
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-07 8:30 ` Christian Borntraeger
2021-07-07 8:54 ` Cornelia Huck
@ 2021-07-07 8:56 ` Janis Schoetterl-Glausch
2021-07-07 8:59 ` Christian Borntraeger
1 sibling, 1 reply; 15+ messages in thread
From: Janis Schoetterl-Glausch @ 2021-07-07 8:56 UTC (permalink / raw
To: Christian Borntraeger, Janis Schoetterl-Glausch, Janosch Frank,
Heiko Carstens, Vasily Gorbik
Cc: David Hildenbrand, Cornelia Huck, Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 7/7/21 10:30 AM, Christian Borntraeger wrote:
>
>
> On 06.07.21 13:47, Janis Schoetterl-Glausch wrote:
>> When this feature is enabled the hardware is free to interpret
>> specification exceptions generated by the guest, instead of causing
>> program interruption interceptions.
>>
>> This benefits (test) programs that generate a lot of specification
>> exceptions (roughly 4x increase in exceptions/sec).
>>
>> Interceptions will occur as before if ICTL_PINT is set,
>> i.e. if guest debug is enabled.
>
> I think I will add
>
> There is no indication if this feature is available or not and the hardware
> is free to interpret or not. So we can simply set this bit and if the
> hardware ignores it we fall back to intercept 8 handling.
Might also mention vSIE and/or incorporate into first paragraph:
When this feature is enabled the hardware is free to interpret
specification exceptions generated by the guest, instead of causing
program interruption interceptions, but it is not required to.
There is no indication if this feature is available or not,
so we can simply set this bit and if the hardware ignores it
we fall back to intercept 8 handling.
The same applies to vSIE, we forward the guest hypervisor's bit
and fall back to injection if interpretation does not occur.
>
>
> With that
>
> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
[...]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-07 8:56 ` Janis Schoetterl-Glausch
@ 2021-07-07 8:59 ` Christian Borntraeger
0 siblings, 0 replies; 15+ messages in thread
From: Christian Borntraeger @ 2021-07-07 8:59 UTC (permalink / raw
To: Janis Schoetterl-Glausch, Janis Schoetterl-Glausch, Janosch Frank,
Heiko Carstens, Vasily Gorbik
Cc: David Hildenbrand, Cornelia Huck, Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 07.07.21 10:56, Janis Schoetterl-Glausch wrote:
> On 7/7/21 10:30 AM, Christian Borntraeger wrote:
>>
>>
>> On 06.07.21 13:47, Janis Schoetterl-Glausch wrote:
>>> When this feature is enabled the hardware is free to interpret
>>> specification exceptions generated by the guest, instead of causing
>>> program interruption interceptions.
>>>
>>> This benefits (test) programs that generate a lot of specification
>>> exceptions (roughly 4x increase in exceptions/sec).
>>>
>>> Interceptions will occur as before if ICTL_PINT is set,
>>> i.e. if guest debug is enabled.
>>
>> I think I will add
>>
>> There is no indication if this feature is available or not and the hardware
>> is free to interpret or not. So we can simply set this bit and if the
>> hardware ignores it we fall back to intercept 8 handling.
>
> Might also mention vSIE and/or incorporate into first paragraph:
>
> When this feature is enabled the hardware is free to interpret
> specification exceptions generated by the guest, instead of causing
> program interruption interceptions, but it is not required to.
> There is no indication if this feature is available or not,
> so we can simply set this bit and if the hardware ignores it
> we fall back to intercept 8 handling.
> The same applies to vSIE, we forward the guest hypervisor's bit
> and fall back to injection if interpretation does not occur.
Can you maybe resend a v2 with all comments (and RBs) added?
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-07 8:54 ` Cornelia Huck
@ 2021-07-07 9:55 ` Janis Schoetterl-Glausch
2021-07-07 11:42 ` Cornelia Huck
0 siblings, 1 reply; 15+ messages in thread
From: Janis Schoetterl-Glausch @ 2021-07-07 9:55 UTC (permalink / raw
To: Cornelia Huck, Christian Borntraeger, Janis Schoetterl-Glausch,
Janosch Frank, Heiko Carstens, Vasily Gorbik
Cc: David Hildenbrand, Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On 7/7/21 10:54 AM, Cornelia Huck wrote:
[...]
>
>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>> index b655a7d82bf0..aadd589a3755 100644
>>> --- a/arch/s390/kvm/kvm-s390.c
>>> +++ b/arch/s390/kvm/kvm-s390.c
>>> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>>> vcpu->arch.sie_block->ecb |= ECB_SRSI;
>>> if (test_kvm_facility(vcpu->kvm, 73))
>>> vcpu->arch.sie_block->ecb |= ECB_TE;
>
> Maybe add
>
> /* no facility bit, but safe as the hardware may ignore it */
>
> or something like that, so that we don't stumble over that in the future?
Well, the hardware being allowed to ignore the bit makes its introduction
without an indication forward compatible because it does not require vSIE to be adapted.
The reserved bits are implicitly set to 0 which means new features are disabled
by default and one observes all the interception one expects.
Maybe this:
/* no facility bit, can opt in because we do not need
to observe specification exception intercepts */
?
>
>>> + if (!kvm_is_ucontrol(vcpu->kvm))
>>> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>>>
>>> if (test_kvm_facility(vcpu->kvm, 8) && vcpu->kvm->arch.use_pfmfi)
>>> vcpu->arch.sie_block->ecb2 |= ECB2_PFMFI;
>
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] KVM: s390: Enable specification exception interpretation
2021-07-07 9:55 ` Janis Schoetterl-Glausch
@ 2021-07-07 11:42 ` Cornelia Huck
0 siblings, 0 replies; 15+ messages in thread
From: Cornelia Huck @ 2021-07-07 11:42 UTC (permalink / raw
To: Janis Schoetterl-Glausch, Christian Borntraeger,
Janis Schoetterl-Glausch, Janosch Frank, Heiko Carstens,
Vasily Gorbik
Cc: David Hildenbrand, Claudio Imbrenda,
open list:KERNEL VIRTUAL MACHINE for s390 (KVM/s390),
open list:S390, open list
On Wed, Jul 07 2021, Janis Schoetterl-Glausch <scgl@linux.vnet.ibm.com> wrote:
> On 7/7/21 10:54 AM, Cornelia Huck wrote:
>
> [...]
>
>>
>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>>> index b655a7d82bf0..aadd589a3755 100644
>>>> --- a/arch/s390/kvm/kvm-s390.c
>>>> +++ b/arch/s390/kvm/kvm-s390.c
>>>> @@ -3200,6 +3200,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>>>> vcpu->arch.sie_block->ecb |= ECB_SRSI;
>>>> if (test_kvm_facility(vcpu->kvm, 73))
>>>> vcpu->arch.sie_block->ecb |= ECB_TE;
>>
>> Maybe add
>>
>> /* no facility bit, but safe as the hardware may ignore it */
>>
>> or something like that, so that we don't stumble over that in the future?
>
> Well, the hardware being allowed to ignore the bit makes its introduction
> without an indication forward compatible because it does not require vSIE to be adapted.
> The reserved bits are implicitly set to 0 which means new features are disabled
> by default and one observes all the interception one expects.
>
> Maybe this:
>
> /* no facility bit, can opt in because we do not need
> to observe specification exception intercepts */
>
> ?
Works for me as well.
>
>>
>>>> + if (!kvm_is_ucontrol(vcpu->kvm))
>>>> + vcpu->arch.sie_block->ecb |= ECB_SPECI;
>>>>
>>>> if (test_kvm_facility(vcpu->kvm, 8) && vcpu->kvm->arch.use_pfmfi)
>>>> vcpu->arch.sie_block->ecb2 |= ECB2_PFMFI;
>>
>> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>>
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2021-07-07 11:42 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-07-06 11:47 [PATCH] KVM: s390: Enable specification exception interpretation Janis Schoetterl-Glausch
2021-07-06 11:52 ` Cornelia Huck
2021-07-06 11:56 ` Christian Borntraeger
2021-07-06 11:59 ` David Hildenbrand
2021-07-06 12:02 ` Christian Borntraeger
2021-07-06 15:16 ` David Hildenbrand
2021-07-06 15:27 ` Janis Schoetterl-Glausch
2021-07-07 7:26 ` David Hildenbrand
2021-07-06 11:55 ` David Hildenbrand
2021-07-07 8:30 ` Christian Borntraeger
2021-07-07 8:54 ` Cornelia Huck
2021-07-07 9:55 ` Janis Schoetterl-Glausch
2021-07-07 11:42 ` Cornelia Huck
2021-07-07 8:56 ` Janis Schoetterl-Glausch
2021-07-07 8:59 ` Christian Borntraeger
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.