From: Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
To: Manoj Kumar <ManojKumar08@arm.com>
Cc: "kernelci@lists.linux.dev" <kernelci@lists.linux.dev>,
Denys Fedoryshchenko <denys.f@collabora.com>
Subject: Re: KCIDB database
Date: Fri, 8 Nov 2024 15:15:22 +0200 [thread overview]
Message-ID: <4f93cd0c-5927-4734-af42-1b6c4f0f8208@redhat.com> (raw)
In-Reply-To: <DB9PR08MB68109AF608CFB742F567968DC4532@DB9PR08MB6810.eurprd08.prod.outlook.com>
Hi Manoj,
Glad to hear you managed to make it work!
It looks like KernelCI legacy is sending schema v1 indeed:
https://github.com/kernelci/kernelci-backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L149
Gosh, that's old. However, it's working fine in production.
I suspect this is because production is using an older version of KCIDB, and
in newer versions support for old schemas is broken. I'll work on a fix, which
should be available today. Meanwhile, could you please respond with a complete
JSON you're sending, as an attachment, so I could make sure I reproduce this
correctly.
Thank you!
Nick
On 11/8/24 11:59 AM, Manoj Kumar wrote:
> Hi Nikolai,
>
>
>
> I managed to create virtual environment python3.9 on kernelci backend and
> worked my way up to push a build message to the kcidb database.
>
> The backend code is generating data structure with schema version “1”.
>
> Below is the content of the message after adding a debug print.
>
> "builds": [
>
> {
>
> "origin": "arm",
>
> "config_name": "x86_64_defconfig",
>
> "architecture": "x86_64",
>
> "start_time": "2024-10-16T11:30:41.592000+00:00",
>
> "revision_origin_id": "2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2",
>
> "log_url": "",
>
> "duration": "1224.9012200832367",
>
> "origin_id": "kernelci.org:670fa3e1e5905849ba40dabe",
>
> "description": "v6.12-rc3-44-g2f87d0916ce0",
>
> "misc": {
>
> "kernel_image_size": ""
>
> },
>
> "valid": "true",
>
> "compiler": "gcc (Debian 12.2.0-14) 12.2.0",
>
> "revision_origin": "arm",
>
> "config_url": "",
>
> "output_files": []
>
> }
>
> ],
>
> *"version": "1"*
>
>
>
>
>
> In response to the above , I got a validation failure from the KCIDB server
> which is expecting structure/ schema version to be high
>
> ValidationError: '1' is not of type 'object'
>
> Failed validating 'type' in schema['properties']['version']:
>
> {'type': 'object',
>
> 'properties': {'major': {'type': 'integer',
>
> 'const': 4,
>
> 'description': 'Major number of the schema '
>
> 'version.\n'
>
> '\n'
>
> 'Increases represent '
>
> 'backward-incompatible '
>
> 'changes. E.g. deleting or '
>
> 'renaming a property, '
>
> 'changing a property type, '
>
> 'restricting values, making a '
>
> 'property required, or adding '
>
> 'a new required property.'},
>
>
>
> My question to you , what is the expected schema version on the Kernelci
> legacy backend code so that the message will be accepted by KCIDB database
>
>
>
>
>
> Br,
>
> Manoj
>
>
>
> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
> *Date: *Friday, 18 October 2024 at 16:05
> *To: *Manoj Kumar <ManojKumar08@arm.com>
> *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys
> Fedoryshchenko <denys.f@collabora.com>
> *Subject: *Re: KCIDB database
>
>> It is not that straight forward to update python version to python3.
>
> Of course, and you cannot really do that, as KernelCI Legacy needs Python 2.
> I was rather suggesting you install Python 3 alongside Python 2, if possible
> for your target.
>
>> This upgrade will be disruptive for our testing and deployment unless the
>> KCIDB results push can be done with python2.7 which we will prefer at this
>> point of time.
>
> If this is hard, one (radical?) way to do this could be running Python 3 on a
> different machine (in a container?) and executing kcidb-submit there, via e.g.
> ssh (docker run?).
>
> But perhaps others have other ideas.
>
> Nick
>
> On 10/18/24 5:04 PM, Manoj Kumar wrote:
>> We used ansible playbooks to deploy backend server https://github.com/ <https://github.com/>
>> kernelci/kernelci-backend-config <https://github.com/kernelci/kernelci-
>> backend-config>
>>
>> Which create virtual environment based on python2.7
>>
>> https://github.com/kernelci/kernelci-backend-config/ <https://github.com/
> kernelci/kernelci-backend-config/>
>> commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071 <https://github.com/kernelci/
>> kernelci-backend-config/commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071>
>>
>>
>>
>> It is not that straight forward to update python version to python3.
>>
>> This upgrade will be disruptive for our testing and deployment unless the
>> KCIDB results push can be done with python2.7 which we will prefer at this
>> point of time.
>>
>>
>>
>> br,
>>
>> Manoj
>>
>>
>>
>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
>> *Date: *Friday, 18 October 2024 at 14:32
>> *To: *Manoj Kumar <ManojKumar08@arm.com>
>> *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys
>> Fedoryshchenko <denys.f@collabora.com>
>> *Subject: *Re: KCIDB database
>>
>> Hi Manoj,
>>
>>> I looked at the below link to get installation steps for the kcidb-submit
>> command.
>>>
>>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/
> kcidb/installation/> <https://docs.kernelci.org/
>> kcidb/installation/> <https://docs.kernelci.org/
>>> kcidb/installation/>
>>>
>>> This instruction is for Python version V3.9 but our backend is using the
>>> Python 2.7.16 at the moment.
>>>
>>> Do you have instruction for python 2.7.16 to install this package
>>
>> KCIDB requires Python 3, and that's why KernelCI Legacy (which requires Python
>> 2) is invoking the kcidb-submit command-line tool in a new process instead of
>> using the KCIDB library directly.
>>
>> You will need to install Python 3 in your deployment, I assume. Perhaps Denys,
>> or someone else on the KernelCI maillist (both CC'd) could tell you how we did
>> that.
>>
>> Nick
>>
>> On 10/18/24 4:21 PM, Manoj Kumar wrote:
>>> Hi Nikolai,
>>>
>>>
>>>
>>> I am planning to push test build data to public KCIDB . I am seeing a failure
>>> because of the kcidb-submit command in a thread at backend.
>>>
>>>
>>>
>>> The class KcidbSubmit expect the command kcidb-submit installed on the backend
>>> system.
>>>
>>> /class KcidbSubmit(object):/
>>>
>>> / def __init__(self, kcidb_options):/
>>>
>>> / kcidb_path = kcidb_options.get("kcidb_path", "")/
>>>
>>> / self.kcidb_submit_cmd = os.path.join(kcidb_path, "kcidb-submit")/
>>>
>>>
>>>
>>> I looked at the below link to get installation steps for the kcidb-submit command.
>>>
>>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/
> kcidb/installation/> <https://docs.kernelci.org/
>> kcidb/installation/> <https://docs.kernelci.org/
>>> kcidb/installation/>
>>>
>>>
>>>
>>> This instruction is for Python version V3.9 but our backend is using the
>>> Python 2.7.16 at the moment.
>>>
>>> Do you have instruction for python 2.7.16 to install this package
>>>
>>>
>>>
>>> Br,
>>>
>>> Manoj
>>>
>>>
>>>
>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
>>> *Date: *Tuesday, 8 October 2024 at 11:46
>>> *To: *Manoj Kumar <ManojKumar08@arm.com>
>>> *Subject: *Re: KCIDB database
>>>
>>> Hi Manoj,
>>>
>>> On 10/8/24 1:26 PM, Manoj Kumar wrote:
>>>> One additional question regarding the credentials file.
>>>>
>>>> Attached credentials files was shared with me named “.kernelci-production-ci-
>>>> arm.json” .
>>>>
>>>> The attached file contains multiple parameters , If I understood correctly all
>>>> the contents of this file should be assigned to this variable https://
>>>> github.com/kernelci/kernelci-backend/
>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https://
>>>> github.com/kernelci/kernelci-backend/
>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>>
>>> Not the contents, but a path to the file itself.
>>>
>>>> Also , the email ID used by internal kernelCI deployment is different . They
>>>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>>>
>>>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>>”.
>>>
>>> That doesn't matter, the email in the JSON credentials is simply a user
>>> (service account) ID and is specific to Google Cloud. It identifies a
>>> particular submitter to the system.
>>>
>>> You don't really need to look into the credentials file at all.
>>>
>>> Nick
>>>
>>> On 10/8/24 1:26 PM, Manoj Kumar wrote:
>>>> Hi Nick,
>>>>
>>>>
>>>>
>>>> Thanks for your prompt responses :thumbsup
>>>>
>>>>
>>>>
>>>> One additional question regarding the credentials file.
>>>>
>>>> Attached credentials files was shared with me named “.kernelci-production-ci-
>>>> arm.json” .
>>>>
>>>> The attached file contains multiple parameters , If I understood correctly all
>>>> the contents of this file should be assigned to this variable https://
>>>> github.com/kernelci/kernelci-backend/
>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https://
>>>> github.com/kernelci/kernelci-backend/
>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>>>
>>>>
>>>>
>>>> Also , the email ID used by internal kernelCI deployment is different . They
>>>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>>>
>>>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>>”.
>>>>
>>>>
>>>>
>>>> Br,
>>>>
>>>> Manoj
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
>>>> *Date: *Tuesday, 8 October 2024 at 05:36
>>>> *To: *Manoj Kumar <ManojKumar08@arm.com>
>>>> *Subject: *Re: KCIDB database
>>>>
>>>> On 10/7/24 7:49 PM, Manoj Kumar wrote:
>>>>> I do have some questions if you can help me find answer.
>>>>>
>>>>> * Do I need to use the private key from the credential json for this
>>>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/
> kernelci-backend/> <https://github.com/kernelci/
>> kernelci-backend/> <https://github.com/kernelci/
>>> kernelci-backend/> <https://github.com/kernelci/
>>>> kernelci-backend/>
>>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38
>>>>> <https://github.com/kernelci/kernelci-backend/
>>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>>>
>>>> Yes, that needs to be pointing to the credentials JSON file.
>>>>
>>>>> * As the storage server is internal , I guess we need to open access to
>>>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->
> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>
>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>
>>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>>
>>>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/
>>>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/
>>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173>
>>>>> files so the kcidb server can access them . Can you confirm this , please
>>>>
>>>> Yes, KCIDB right now will try to download every 256th URL you submit, that is
>>>> under 5MB. The artifact caching system is in trial mode still. However, what's
>>>> more important, *people* will try to open the URLs you submit.
>>>>
>>>> Opening the access to the whole server might be problematic for you guys, as
>>>> you want to keep some results secret, IIRC. Maybe you could copy them to some
>>>> other web server and translate the URLs as you submit them?
>>>>
>>>> Nick
>>>>
>>>> On 10/7/24 7:49 PM, Manoj Kumar wrote:
>>>>> Hi Nick,
>>>>>
>>>>> Nice to meet you too mate, I am going through the code base of kcidb.py to
>>>>> push results.
>>>>>
>>>>> I will test pushing results to playground with the earlier credentials as you
>>>>> mentioned.
>>>>>
>>>>>
>>>>>
>>>>> Also , I will be configuring internal kernelci deployment to push results to
>>>>> public KCIDB for builds and tests.
>>>>>
>>>>>
>>>>>
>>>>> I do have some questions if you can help me find answer.
>>>>>
>>>>> * Do I need to use the private key from the credential json for this
>>>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/
> kernelci-backend/> <https://github.com/kernelci/
>> kernelci-backend/> <https://github.com/kernelci/
>>> kernelci-backend/> <https://github.com/kernelci/
>>>> kernelci-backend/>
>>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38
>>>>> <https://github.com/kernelci/kernelci-backend/
>>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>>>> * As the storage server is internal , I guess we need to open access to
>>>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->
> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>
>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>
>>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>>
>>>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/
>>>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/
>>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173>
>>>>> files so the kcidb server can access them . Can you confirm this , please
>>>>>
>>>>>
>>>>>
>>>>> Br,
>>>>>
>>>>> Manoj
>>>>>
>>>>>
>>>>>
>>>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
>>>>> *Date: *Monday, 7 October 2024 at 12:15
>>>>> *To: *Manoj Kumar <ManojKumar08@arm.com>
>>>>> *Subject: *Re: KCIDB database
>>>>>
>>>>> Hi Manoj,
>>>>>
>>>>> Nice to meet you!
>>>>>
>>>>> I'm glad to hear there's progress towards resuming KCIDB submissions at ARM!
>>>>>
>>>>> The access parameters are out of date, although they will work for
>>>>> submissions. Here's more up-to-date info:
>>>>>
>>>>> https://docs.kernelci.org/kcidb/submitter_guide/ <https://docs.kernelci.org/
> kcidb/submitter_guide/> <https://docs.kernelci.org/
>> kcidb/submitter_guide/> <https://docs.kernelci.org/
>>> kcidb/submitter_guide/> <https://docs.kernelci.org/
>>>> kcidb/submitter_guide/> <https://docs.kernelci.org/
>>>>> kcidb/submitter_guide/>
>>>>>
>>>>> The credentials I gave ARM before should still work. I'd recommend restarting
>>>>> submissions to the "playground" database ("playground_kcidb_new" topic) so you
>>>>> can experiment freely without worrying about bothering anyone with invalid
>>>>> data. Then when you're ready, switch to production ("kcidb_new").
>>>>>
>>>>> Don't hesitate to reach out, if you have any questions. I'll add you as the
>>>>> ARM's current contact here:
>>>>>
>>>>> https://github.com/kernelci/kcidb/issues/328 <https://github.com/kernelci/
> kcidb/issues/328> <https://github.com/kernelci/
>> kcidb/issues/328> <https://github.com/kernelci/
>>> kcidb/issues/328> <https://github.com/kernelci/
>>>> kcidb/issues/328> <https://github.com/kernelci/
>>>>> kcidb/issues/328>
>>>>>
>>>>> I think ARM doesn't let employees use Slack, but if I'm wrong, I can invite
>>>>> you to our channel, so we can have more interactive Q&A, if needed.
>>>>>
>>>>> Nick
>>>>>
>>>>> On 10/7/24 12:37 PM, Manoj Kumar wrote:
>>>>>> Hi Nikolai,
>>>>>>
>>>>>> I wanted to say hello to you and start the initial thread regarding pushing
>>>>>> the test results to KernelCI KCIDB.
>>>>>> I will be enabling the changes to push kernel test and build results to KCIDB
>>>>>> from ARM testing . I was told by Cristian and Mark that you are the POC from
>>>>>> kernelCI KCIDB side.
>>>>>>
>>>>>> The access parameters shared with me are below , can you confirm they are
>>>>>> still valid and not expired / updated .
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The access parameters are:
>>>>>>
>>>>>> Google Cloud project name: kernelci-production
>>>>>> Playground dataset name: playground_kernelci04
>>>>>> Playground submission queue topic: playground_kernelci_new
>>>>>> Production dataset name: kernelci04
>>>>>> Production submission queue topic: kernelci_new
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Manoj
>>>>>>
>>>>>>
>>>>>>
>>>>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>>>>> confidential and may also be privileged. If you are not the intended
>>>>>> recipient, please notify the sender immediately and do not disclose the
>>>>>> contents to any other person, use it for any purpose, or store or copy the
>>>>>> information in any medium. Thank you.
>>>>>
>>>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>>>> confidential and may also be privileged. If you are not the intended
>>>>> recipient, please notify the sender immediately and do not disclose the
>>>>> contents to any other person, use it for any purpose, or store or copy the
>>>>> information in any medium. Thank you.
>>>>
>>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>>> confidential and may also be privileged. If you are not the intended
>>>> recipient, please notify the sender immediately and do not disclose the
>>>> contents to any other person, use it for any purpose, or store or copy the
>>>> information in any medium. Thank you.
>>>
>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>> confidential and may also be privileged. If you are not the intended
>>> recipient, please notify the sender immediately and do not disclose the
>>> contents to any other person, use it for any purpose, or store or copy the
>>> information in any medium. Thank you.
>>
>> IMPORTANT NOTICE: The contents of this email and any attachments are
>> confidential and may also be privileged. If you are not the intended
>> recipient, please notify the sender immediately and do not disclose the
>> contents to any other person, use it for any purpose, or store or copy the
>> information in any medium. Thank you.
>
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
next prev parent reply other threads:[~2024-11-08 13:15 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <547ddf99-1b75-c463-e6c7-32389c461468@redhat.com>
[not found] ` <AS8PR08MB6407FD1D5E33918F5DA6032A9E772@AS8PR08MB6407.eurprd08.prod.outlook.com>
[not found] ` <PAXPR08MB6813993A4C70BDE277129A76C47D2@PAXPR08MB6813.eurprd08.prod.outlook.com>
[not found] ` <d9f9111e-9890-4795-9f8a-a6946a8fcbb9@redhat.com>
[not found] ` <PAXPR08MB68136B5FD6BFD45B2BC1BE2DC47D2@PAXPR08MB6813.eurprd08.prod.outlook.com>
[not found] ` <9ab1e3c8-9c7f-4657-815a-13a68e0b87f7@redhat.com>
[not found] ` <PAXPR08MB68139AC2C5DFA28EC9A4CCC6C47E2@PAXPR08MB6813.eurprd08.prod.outlook.com>
[not found] ` <2626f2ce-adb0-4aad-bbfc-c19f634a4d30@redhat.com>
[not found] ` <DB9PR08MB68103176F5638C352F410BCBC4402@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-10-18 13:32 ` KCIDB database Nikolai Kondrashov
[not found] ` <DB9PR08MB68104499218B100804567075C4402@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-10-18 15:05 ` Nikolai Kondrashov
[not found] ` <PAXPR08MB6813E7233F5ACB43D4E7FC81C44E2@PAXPR08MB6813.eurprd08.prod.outlook.com>
2024-10-24 15:56 ` Nikolai Kondrashov
[not found] ` <DB9PR08MB68109AF608CFB742F567968DC4532@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-11-08 13:15 ` Nikolai Kondrashov [this message]
[not found] ` <DB9PR08MB68105069590B2A3A065DEBC2C45D2@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-11-08 14:51 ` Nikolai Kondrashov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4f93cd0c-5927-4734-af42-1b6c4f0f8208@redhat.com \
--to=nikolai.kondrashov@redhat.com \
--cc=ManojKumar08@arm.com \
--cc=denys.f@collabora.com \
--cc=kernelci@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).