From: Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
To: Manoj Kumar <ManojKumar08@arm.com>
Cc: "kernelci@lists.linux.dev" <kernelci@lists.linux.dev>,
Denys Fedoryshchenko <denys.f@collabora.com>
Subject: Re: KCIDB database
Date: Fri, 18 Oct 2024 16:32:33 +0300 [thread overview]
Message-ID: <e90b3ccc-fd16-406b-ac8a-7d2bf66abd1f@redhat.com> (raw)
In-Reply-To: <DB9PR08MB68103176F5638C352F410BCBC4402@DB9PR08MB6810.eurprd08.prod.outlook.com>
Hi Manoj,
> I looked at the below link to get installation steps for the kcidb-submit
command.
>
> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/
> kcidb/installation/>
>
> This instruction is for Python version V3.9 but our backend is using the
> Python 2.7.16 at the moment.
>
> Do you have instruction for python 2.7.16 to install this package
KCIDB requires Python 3, and that's why KernelCI Legacy (which requires Python
2) is invoking the kcidb-submit command-line tool in a new process instead of
using the KCIDB library directly.
You will need to install Python 3 in your deployment, I assume. Perhaps Denys,
or someone else on the KernelCI maillist (both CC'd) could tell you how we did
that.
Nick
On 10/18/24 4:21 PM, Manoj Kumar wrote:
> Hi Nikolai,
>
>
>
> I am planning to push test build data to public KCIDB . I am seeing a failure
> because of the kcidb-submit command in a thread at backend.
>
>
>
> The class KcidbSubmit expect the command kcidb-submit installed on the backend
> system.
>
> /class KcidbSubmit(object):/
>
> / def __init__(self, kcidb_options):/
>
> / kcidb_path = kcidb_options.get("kcidb_path", "")/
>
> / self.kcidb_submit_cmd = os.path.join(kcidb_path, "kcidb-submit")/
>
>
>
> I looked at the below link to get installation steps for the kcidb-submit command.
>
> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/
> kcidb/installation/>
>
>
>
> This instruction is for Python version V3.9 but our backend is using the
> Python 2.7.16 at the moment.
>
> Do you have instruction for python 2.7.16 to install this package
>
>
>
> Br,
>
> Manoj
>
>
>
> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
> *Date: *Tuesday, 8 October 2024 at 11:46
> *To: *Manoj Kumar <ManojKumar08@arm.com>
> *Subject: *Re: KCIDB database
>
> Hi Manoj,
>
> On 10/8/24 1:26 PM, Manoj Kumar wrote:
>> One additional question regarding the credentials file.
>>
>> Attached credentials files was shared with me named “.kernelci-production-ci-
>> arm.json” .
>>
>> The attached file contains multiple parameters , If I understood correctly all
>> the contents of this file should be assigned to this variable https://
>> github.com/kernelci/kernelci-backend/
>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https://
>> github.com/kernelci/kernelci-backend/
>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>
> Not the contents, but a path to the file itself.
>
>> Also , the email ID used by internal kernelCI deployment is different . They
>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>
>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>”.
>
> That doesn't matter, the email in the JSON credentials is simply a user
> (service account) ID and is specific to Google Cloud. It identifies a
> particular submitter to the system.
>
> You don't really need to look into the credentials file at all.
>
> Nick
>
> On 10/8/24 1:26 PM, Manoj Kumar wrote:
>> Hi Nick,
>>
>>
>>
>> Thanks for your prompt responses :thumbsup
>>
>>
>>
>> One additional question regarding the credentials file.
>>
>> Attached credentials files was shared with me named “.kernelci-production-ci-
>> arm.json” .
>>
>> The attached file contains multiple parameters , If I understood correctly all
>> the contents of this file should be assigned to this variable https://
>> github.com/kernelci/kernelci-backend/
>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https://
>> github.com/kernelci/kernelci-backend/
>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>
>>
>>
>> Also , the email ID used by internal kernelCI deployment is different . They
>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>
>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>”.
>>
>>
>>
>> Br,
>>
>> Manoj
>>
>>
>>
>>
>>
>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
>> *Date: *Tuesday, 8 October 2024 at 05:36
>> *To: *Manoj Kumar <ManojKumar08@arm.com>
>> *Subject: *Re: KCIDB database
>>
>> On 10/7/24 7:49 PM, Manoj Kumar wrote:
>>> I do have some questions if you can help me find answer.
>>>
>>> * Do I need to use the private key from the credential json for this
>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/
> kernelci-backend/> <https://github.com/kernelci/
>> kernelci-backend/>
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38
>>> <https://github.com/kernelci/kernelci-backend/
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>
>> Yes, that needs to be pointing to the credentials JSON file.
>>
>>> * As the storage server is internal , I guess we need to open access to
>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->
> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>
>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/
>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173>
>>> files so the kcidb server can access them . Can you confirm this , please
>>
>> Yes, KCIDB right now will try to download every 256th URL you submit, that is
>> under 5MB. The artifact caching system is in trial mode still. However, what's
>> more important, *people* will try to open the URLs you submit.
>>
>> Opening the access to the whole server might be problematic for you guys, as
>> you want to keep some results secret, IIRC. Maybe you could copy them to some
>> other web server and translate the URLs as you submit them?
>>
>> Nick
>>
>> On 10/7/24 7:49 PM, Manoj Kumar wrote:
>>> Hi Nick,
>>>
>>> Nice to meet you too mate, I am going through the code base of kcidb.py to
>>> push results.
>>>
>>> I will test pushing results to playground with the earlier credentials as you
>>> mentioned.
>>>
>>>
>>>
>>> Also , I will be configuring internal kernelci deployment to push results to
>>> public KCIDB for builds and tests.
>>>
>>>
>>>
>>> I do have some questions if you can help me find answer.
>>>
>>> * Do I need to use the private key from the credential json for this
>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/
> kernelci-backend/> <https://github.com/kernelci/
>> kernelci-backend/>
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38
>>> <https://github.com/kernelci/kernelci-backend/
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>> * As the storage server is internal , I guess we need to open access to
>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->
> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>
>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/
>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173>
>>> files so the kcidb server can access them . Can you confirm this , please
>>>
>>>
>>>
>>> Br,
>>>
>>> Manoj
>>>
>>>
>>>
>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
>>> *Date: *Monday, 7 October 2024 at 12:15
>>> *To: *Manoj Kumar <ManojKumar08@arm.com>
>>> *Subject: *Re: KCIDB database
>>>
>>> Hi Manoj,
>>>
>>> Nice to meet you!
>>>
>>> I'm glad to hear there's progress towards resuming KCIDB submissions at ARM!
>>>
>>> The access parameters are out of date, although they will work for
>>> submissions. Here's more up-to-date info:
>>>
>>> https://docs.kernelci.org/kcidb/submitter_guide/ <https://docs.kernelci.org/
> kcidb/submitter_guide/> <https://docs.kernelci.org/
>> kcidb/submitter_guide/> <https://docs.kernelci.org/
>>> kcidb/submitter_guide/>
>>>
>>> The credentials I gave ARM before should still work. I'd recommend restarting
>>> submissions to the "playground" database ("playground_kcidb_new" topic) so you
>>> can experiment freely without worrying about bothering anyone with invalid
>>> data. Then when you're ready, switch to production ("kcidb_new").
>>>
>>> Don't hesitate to reach out, if you have any questions. I'll add you as the
>>> ARM's current contact here:
>>>
>>> https://github.com/kernelci/kcidb/issues/328 <https://github.com/kernelci/
> kcidb/issues/328> <https://github.com/kernelci/
>> kcidb/issues/328> <https://github.com/kernelci/
>>> kcidb/issues/328>
>>>
>>> I think ARM doesn't let employees use Slack, but if I'm wrong, I can invite
>>> you to our channel, so we can have more interactive Q&A, if needed.
>>>
>>> Nick
>>>
>>> On 10/7/24 12:37 PM, Manoj Kumar wrote:
>>>> Hi Nikolai,
>>>>
>>>> I wanted to say hello to you and start the initial thread regarding pushing
>>>> the test results to KernelCI KCIDB.
>>>> I will be enabling the changes to push kernel test and build results to KCIDB
>>>> from ARM testing . I was told by Cristian and Mark that you are the POC from
>>>> kernelCI KCIDB side.
>>>>
>>>> The access parameters shared with me are below , can you confirm they are
>>>> still valid and not expired / updated .
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The access parameters are:
>>>>
>>>> Google Cloud project name: kernelci-production
>>>> Playground dataset name: playground_kernelci04
>>>> Playground submission queue topic: playground_kernelci_new
>>>> Production dataset name: kernelci04
>>>> Production submission queue topic: kernelci_new
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Manoj
>>>>
>>>>
>>>>
>>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>>> confidential and may also be privileged. If you are not the intended
>>>> recipient, please notify the sender immediately and do not disclose the
>>>> contents to any other person, use it for any purpose, or store or copy the
>>>> information in any medium. Thank you.
>>>
>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>> confidential and may also be privileged. If you are not the intended
>>> recipient, please notify the sender immediately and do not disclose the
>>> contents to any other person, use it for any purpose, or store or copy the
>>> information in any medium. Thank you.
>>
>> IMPORTANT NOTICE: The contents of this email and any attachments are
>> confidential and may also be privileged. If you are not the intended
>> recipient, please notify the sender immediately and do not disclose the
>> contents to any other person, use it for any purpose, or store or copy the
>> information in any medium. Thank you.
>
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
next parent reply other threads:[~2024-10-18 13:32 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <547ddf99-1b75-c463-e6c7-32389c461468@redhat.com>
[not found] ` <AS8PR08MB6407FD1D5E33918F5DA6032A9E772@AS8PR08MB6407.eurprd08.prod.outlook.com>
[not found] ` <PAXPR08MB6813993A4C70BDE277129A76C47D2@PAXPR08MB6813.eurprd08.prod.outlook.com>
[not found] ` <d9f9111e-9890-4795-9f8a-a6946a8fcbb9@redhat.com>
[not found] ` <PAXPR08MB68136B5FD6BFD45B2BC1BE2DC47D2@PAXPR08MB6813.eurprd08.prod.outlook.com>
[not found] ` <9ab1e3c8-9c7f-4657-815a-13a68e0b87f7@redhat.com>
[not found] ` <PAXPR08MB68139AC2C5DFA28EC9A4CCC6C47E2@PAXPR08MB6813.eurprd08.prod.outlook.com>
[not found] ` <2626f2ce-adb0-4aad-bbfc-c19f634a4d30@redhat.com>
[not found] ` <DB9PR08MB68103176F5638C352F410BCBC4402@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-10-18 13:32 ` Nikolai Kondrashov [this message]
[not found] ` <DB9PR08MB68104499218B100804567075C4402@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-10-18 15:05 ` KCIDB database Nikolai Kondrashov
[not found] ` <PAXPR08MB6813E7233F5ACB43D4E7FC81C44E2@PAXPR08MB6813.eurprd08.prod.outlook.com>
2024-10-24 15:56 ` Nikolai Kondrashov
[not found] ` <DB9PR08MB68109AF608CFB742F567968DC4532@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-11-08 13:15 ` Nikolai Kondrashov
[not found] ` <DB9PR08MB68105069590B2A3A065DEBC2C45D2@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-11-08 14:51 ` Nikolai Kondrashov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e90b3ccc-fd16-406b-ac8a-7d2bf66abd1f@redhat.com \
--to=nikolai.kondrashov@redhat.com \
--cc=ManojKumar08@arm.com \
--cc=denys.f@collabora.com \
--cc=kernelci@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).