From: "Huang, Ying" <ying.huang@intel.com>
To: Alistair Popple <apopple@nvidia.com>
Cc: Andrew Morton <akpm@linux-foundation.org>, <linux-mm@kvack.org>,
<linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
<nvdimm@lists.linux.dev>, <linux-acpi@vger.kernel.org>,
"Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>,
Wei Xu <weixugc@google.com>,
Dan Williams <dan.j.williams@intel.com>,
Dave Hansen <dave.hansen@intel.com>,
"Davidlohr Bueso" <dave@stgolabs.net>,
Johannes Weiner <hannes@cmpxchg.org>,
"Jonathan Cameron" <Jonathan.Cameron@huawei.com>,
Michal Hocko <mhocko@kernel.org>, Yang Shi <shy828301@gmail.com>,
Rafael J Wysocki <rafael.j.wysocki@intel.com>
Subject: Re: [PATCH RESEND 4/4] dax, kmem: calculate abstract distance with general interface
Date: Tue, 25 Jul 2023 15:02:42 +0800 [thread overview]
Message-ID: <87cz0gxylp.fsf@yhuang6-desk2.ccr.corp.intel.com> (raw)
In-Reply-To: <87edkwznsf.fsf@nvdebian.thelocal> (Alistair Popple's message of "Tue, 25 Jul 2023 13:11:12 +1000")
Alistair Popple <apopple@nvidia.com> writes:
> Huang Ying <ying.huang@intel.com> writes:
>
>> Previously, a fixed abstract distance MEMTIER_DEFAULT_DAX_ADISTANCE is
>> used for slow memory type in kmem driver. This limits the usage of
>> kmem driver, for example, it cannot be used for HBM (high bandwidth
>> memory).
>>
>> So, we use the general abstract distance calculation mechanism in kmem
>> drivers to get more accurate abstract distance on systems with proper
>> support. The original MEMTIER_DEFAULT_DAX_ADISTANCE is used as
>> fallback only.
>>
>> Now, multiple memory types may be managed by kmem. These memory types
>> are put into the "kmem_memory_types" list and protected by
>> kmem_memory_type_lock.
>
> See below but I wonder if kmem_memory_types could be a common helper
> rather than kdax specific?
>
>> Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
>> Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
>> Cc: Wei Xu <weixugc@google.com>
>> Cc: Alistair Popple <apopple@nvidia.com>
>> Cc: Dan Williams <dan.j.williams@intel.com>
>> Cc: Dave Hansen <dave.hansen@intel.com>
>> Cc: Davidlohr Bueso <dave@stgolabs.net>
>> Cc: Johannes Weiner <hannes@cmpxchg.org>
>> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> Cc: Michal Hocko <mhocko@kernel.org>
>> Cc: Yang Shi <shy828301@gmail.com>
>> Cc: Rafael J Wysocki <rafael.j.wysocki@intel.com>
>> ---
>> drivers/dax/kmem.c | 54 +++++++++++++++++++++++++++---------
>> include/linux/memory-tiers.h | 2 ++
>> mm/memory-tiers.c | 2 +-
>> 3 files changed, 44 insertions(+), 14 deletions(-)
>>
>> diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
>> index 898ca9505754..837165037231 100644
>> --- a/drivers/dax/kmem.c
>> +++ b/drivers/dax/kmem.c
>> @@ -49,14 +49,40 @@ struct dax_kmem_data {
>> struct resource *res[];
>> };
>>
>> -static struct memory_dev_type *dax_slowmem_type;
>> +static DEFINE_MUTEX(kmem_memory_type_lock);
>> +static LIST_HEAD(kmem_memory_types);
>> +
>> +static struct memory_dev_type *kmem_find_alloc_memorty_type(int adist)
>> +{
>> + bool found = false;
>> + struct memory_dev_type *mtype;
>> +
>> + mutex_lock(&kmem_memory_type_lock);
>> + list_for_each_entry(mtype, &kmem_memory_types, list) {
>> + if (mtype->adistance == adist) {
>> + found = true;
>> + break;
>> + }
>> + }
>> + if (!found) {
>> + mtype = alloc_memory_type(adist);
>> + if (!IS_ERR(mtype))
>> + list_add(&mtype->list, &kmem_memory_types);
>> + }
>> + mutex_unlock(&kmem_memory_type_lock);
>> +
>> + return mtype;
>> +}
>> +
>> static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>> {
>> struct device *dev = &dev_dax->dev;
>> unsigned long total_len = 0;
>> struct dax_kmem_data *data;
>> + struct memory_dev_type *mtype;
>> int i, rc, mapped = 0;
>> int numa_node;
>> + int adist = MEMTIER_DEFAULT_DAX_ADISTANCE;
>>
>> /*
>> * Ensure good NUMA information for the persistent memory.
>> @@ -71,6 +97,11 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>> return -EINVAL;
>> }
>>
>> + mt_calc_adistance(numa_node, &adist);
>> + mtype = kmem_find_alloc_memorty_type(adist);
>> + if (IS_ERR(mtype))
>> + return PTR_ERR(mtype);
>> +
>
> I wrote my own quick and dirty module to test this and wrote basically
> the same code sequence.
>
> I notice your using a list of memory types here though. I think it would
> be nice to have a common helper that other users could call to do the
> mt_calc_adistance() / kmem_find_alloc_memory_type() /
> init_node_memory_type() sequence and cleanup as my naive approach would
> result in a new memory_dev_type per device even though adist might be
> the same. A common helper would make it easy to de-dup those.
If it's useful, we can move kmem_find_alloc_memory_type() to
memory-tier.c after some revision. But I tend to move it after we have
the second user. What do you think about that?
--
Best Regards,
Huang, Ying
>> for (i = 0; i < dev_dax->nr_range; i++) {
>> struct range range;
>>
>> @@ -88,7 +119,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>> return -EINVAL;
>> }
>>
>> - init_node_memory_type(numa_node, dax_slowmem_type);
>> + init_node_memory_type(numa_node, mtype);
>>
>> rc = -ENOMEM;
>> data = kzalloc(struct_size(data, res, dev_dax->nr_range), GFP_KERNEL);
>> @@ -167,7 +198,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>> err_res_name:
>> kfree(data);
>> err_dax_kmem_data:
>> - clear_node_memory_type(numa_node, dax_slowmem_type);
>> + clear_node_memory_type(numa_node, mtype);
>> return rc;
>> }
>>
>> @@ -219,7 +250,7 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
>> * for that. This implies this reference will be around
>> * till next reboot.
>> */
>> - clear_node_memory_type(node, dax_slowmem_type);
>> + clear_node_memory_type(node, NULL);
>> }
>> }
>> #else
>> @@ -251,12 +282,6 @@ static int __init dax_kmem_init(void)
>> if (!kmem_name)
>> return -ENOMEM;
>>
>> - dax_slowmem_type = alloc_memory_type(MEMTIER_DEFAULT_DAX_ADISTANCE);
>> - if (IS_ERR(dax_slowmem_type)) {
>> - rc = PTR_ERR(dax_slowmem_type);
>> - goto err_dax_slowmem_type;
>> - }
>> -
>> rc = dax_driver_register(&device_dax_kmem_driver);
>> if (rc)
>> goto error_dax_driver;
>> @@ -264,18 +289,21 @@ static int __init dax_kmem_init(void)
>> return rc;
>>
>> error_dax_driver:
>> - destroy_memory_type(dax_slowmem_type);
>> -err_dax_slowmem_type:
>> kfree_const(kmem_name);
>> return rc;
>> }
>>
>> static void __exit dax_kmem_exit(void)
>> {
>> + struct memory_dev_type *mtype, *mtn;
>> +
>> dax_driver_unregister(&device_dax_kmem_driver);
>> if (!any_hotremove_failed)
>> kfree_const(kmem_name);
>> - destroy_memory_type(dax_slowmem_type);
>> + list_for_each_entry_safe(mtype, mtn, &kmem_memory_types, list) {
>> + list_del(&mtype->list);
>> + destroy_memory_type(mtype);
>> + }
>> }
>>
>> MODULE_AUTHOR("Intel Corporation");
>> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
>> index 9377239c8d34..aca22220cb5c 100644
>> --- a/include/linux/memory-tiers.h
>> +++ b/include/linux/memory-tiers.h
>> @@ -24,6 +24,8 @@ struct memory_tier;
>> struct memory_dev_type {
>> /* list of memory types that are part of same tier as this type */
>> struct list_head tier_sibiling;
>> + /* list of memory types that are managed by one driver */
>> + struct list_head list;
>> /* abstract distance for this specific memory type */
>> int adistance;
>> /* Nodes of same abstract distance */
>> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
>> index 9a734ef2edfb..38005c60fa2d 100644
>> --- a/mm/memory-tiers.c
>> +++ b/mm/memory-tiers.c
>> @@ -581,7 +581,7 @@ EXPORT_SYMBOL_GPL(init_node_memory_type);
>> void clear_node_memory_type(int node, struct memory_dev_type *memtype)
>> {
>> mutex_lock(&memory_tier_lock);
>> - if (node_memory_types[node].memtype == memtype)
>> + if (node_memory_types[node].memtype == memtype || !memtype)
>> node_memory_types[node].map_count--;
>> /*
>> * If we umapped all the attached devices to this node,
next prev parent reply other threads:[~2023-07-25 7:04 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-21 1:29 [PATCH RESEND 0/4] memory tiering: calculate abstract distance based on ACPI HMAT Huang Ying
2023-07-21 1:29 ` [PATCH RESEND 1/4] memory tiering: add abstract distance calculation algorithms management Huang Ying
2023-07-25 2:13 ` Alistair Popple
2023-07-25 3:14 ` Huang, Ying
2023-07-25 8:26 ` Alistair Popple
2023-07-26 7:33 ` Huang, Ying
2023-07-27 3:42 ` Alistair Popple
2023-07-27 4:02 ` Huang, Ying
2023-07-27 4:07 ` Alistair Popple
2023-07-27 5:41 ` Huang, Ying
2023-07-28 1:20 ` Alistair Popple
2023-08-11 3:51 ` Huang, Ying
2023-08-21 11:26 ` Alistair Popple
2023-08-21 22:50 ` Huang, Ying
2023-08-21 23:52 ` Alistair Popple
2023-08-22 0:58 ` Huang, Ying
2023-08-22 7:11 ` Alistair Popple
2023-08-23 5:56 ` Huang, Ying
2023-08-25 5:41 ` Alistair Popple
2023-07-21 1:29 ` [PATCH RESEND 2/4] acpi, hmat: refactor hmat_register_target_initiators() Huang Ying
2023-07-25 2:44 ` Alistair Popple
2023-08-07 16:55 ` Jonathan Cameron
2023-08-11 1:13 ` Huang, Ying
2023-07-21 1:29 ` [PATCH RESEND 3/4] acpi, hmat: calculate abstract distance with HMAT Huang Ying
2023-07-25 2:45 ` Alistair Popple
2023-07-25 6:47 ` Huang, Ying
2023-08-21 11:53 ` Alistair Popple
2023-08-21 23:28 ` Huang, Ying
2023-07-21 1:29 ` [PATCH RESEND 4/4] dax, kmem: calculate abstract distance with general interface Huang Ying
2023-07-25 3:11 ` Alistair Popple
2023-07-25 7:02 ` Huang, Ying [this message]
2023-08-21 12:03 ` Alistair Popple
2023-08-21 23:33 ` Huang, Ying
2023-08-22 7:36 ` Alistair Popple
2023-08-23 2:13 ` Huang, Ying
2023-08-25 6:00 ` Alistair Popple
2023-07-21 4:15 ` [PATCH RESEND 0/4] memory tiering: calculate abstract distance based on ACPI HMAT Alistair Popple
2023-07-24 17:58 ` Andrew Morton
2023-08-01 2:35 ` Bharata B Rao
2023-08-11 6:26 ` Huang, Ying
2023-08-11 7:49 ` Bharata B Rao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87cz0gxylp.fsf@yhuang6-desk2.ccr.corp.intel.com \
--to=ying.huang@intel.com \
--cc=Jonathan.Cameron@huawei.com \
--cc=akpm@linux-foundation.org \
--cc=aneesh.kumar@linux.ibm.com \
--cc=apopple@nvidia.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@intel.com \
--cc=dave@stgolabs.net \
--cc=hannes@cmpxchg.org \
--cc=linux-acpi@vger.kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=nvdimm@lists.linux.dev \
--cc=rafael.j.wysocki@intel.com \
--cc=shy828301@gmail.com \
--cc=weixugc@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).