From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C36AC7EE2F for ; Mon, 12 Jun 2023 18:18:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B641A8E0005; Mon, 12 Jun 2023 14:18:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B14968E0002; Mon, 12 Jun 2023 14:18:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A040B8E0005; Mon, 12 Jun 2023 14:18:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8F6098E0002 for ; Mon, 12 Jun 2023 14:18:50 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3F906160305 for ; Mon, 12 Jun 2023 18:18:49 +0000 (UTC) X-FDA: 80894906778.21.51088DF Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by imf08.hostedemail.com (Postfix) with ESMTP id 3FF35160009 for ; Mon, 12 Jun 2023 18:18:45 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=W0PwfkbQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686593926; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sCegMuLmkgKKIjp12zoMl7g1gHWpAi0hnuD+7lSH5gI=; b=3srPE1Zz0fYwmalkc4PNs2Waa+7D1VyYLP1nl5YSZKEShyk68fJTXnvRols8TX++VpE1Pe Xy6M0lu6GLscedufP87WvWyytc69CJ3ED6Ds7aKVBIfzNe2N3oq5mby6ZN9FqoQM+IkWxN NPtema1r89sKz/r5vDFrS2vfuhq8urQ= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=W0PwfkbQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686593926; a=rsa-sha256; cv=none; b=azxQ7jj1SOKSLkNYf/yuxi3m5BhY+K2GoOGrGHer6LP+OLCAwOuh79CYZUP2hD/hfK1V6u RdVBmGkpRaaw6kGAv82K3dyC6nu7oeQll8R854nNsYDp32jfJ+UfH/6Ez92A+1hclTIJhA 3BJMIsR7yDy4pJTJ+83nNIjzCTlaHWc= Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-974638ed5c5so954195266b.1 for ; Mon, 12 Jun 2023 11:18:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686593924; x=1689185924; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=sCegMuLmkgKKIjp12zoMl7g1gHWpAi0hnuD+7lSH5gI=; b=W0PwfkbQCb8Bm4+yrRoaS10SzmZ/E6w3dBH2MJZaSU4s+TMcqUP0c70uBKSVBcXvVw He3r4YwsoZxqql+2sWns7SdCIoRl1nc1M48/bcRc7/wkcBJMKE/KeVoXJNxgobME/sXH AiyawVhAsxumUWDSyAu6Rvn348MnLPNOT59yN521h4YeTLOQYCG1NWOf218FTQuh2Yzx TYjYshVzwgiWaJoJ9fMCIw0/B5RgA5fmrdHqLk2pBO6Uo2XwcZU2HvJ6wmUOIXr9P3ku ghoZ23Jw/QVlbNDx8L3e6pIg4Ozc88Tf0CVvA9BhijHa9PNhGTP5BQ1J3r+2QQJQZqsl fWSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686593924; x=1689185924; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sCegMuLmkgKKIjp12zoMl7g1gHWpAi0hnuD+7lSH5gI=; b=ceYRfACe/zbJNY+GM71GYfZPQvFfP9bnwchkDyO+EeritSynHMnGE6YeUb/SnE/WFm wNyncCAI/6uKtzgNX0viicYSwqdIN/9b94TtT7zqDy5zFEs/MEw3qJzqA7nEpCbs8jCI veDj87Or5ND3fgoWhPhntyHR+nz7gttGg16q2GM5YFof+cSM+szajPWNGSNWM76ehq9V DIZgWdsfQqwCYurR4ybXIhTLmBFUyF5izxXpOw/VmAy05sbhmBcqeqCS/tegIybxidnC RRD9xtVE6pmDDax623PIdtlx8RuGNI7xF4lb5UCtj2g3ynaFV602Wnp/ilqylj5nxd2o lS6w== X-Gm-Message-State: AC+VfDxOwJxyPCRuw7kHv/40CE5uyi5ysKn08yYRDjtCfKS070c9hL46 vPih3k75TQyiAbj4E/ir9QHwXTY7ONVPm2RLQlXbnQ== X-Google-Smtp-Source: ACHHUZ5hiuYbW5oJPVtRwr2lj6pp5XmdmPS2jFgK2VzN0wQrX9wyQFh4mRlSLvXmrJ8K+zTu0FDI2hqQCnMq/AjMum4= X-Received: by 2002:a17:907:25ca:b0:971:55e2:82c3 with SMTP id ae10-20020a17090725ca00b0097155e282c3mr7708937ejc.20.1686593924390; Mon, 12 Jun 2023 11:18:44 -0700 (PDT) MIME-Version: 1.0 References: <20230612093815.133504-1-cerasuolodomenico@gmail.com> <20230612093815.133504-2-cerasuolodomenico@gmail.com> In-Reply-To: <20230612093815.133504-2-cerasuolodomenico@gmail.com> From: Yosry Ahmed Date: Mon, 12 Jun 2023 11:18:07 -0700 Message-ID: Subject: Re: [PATCH v3 1/7] mm: zswap: add pool shrinking mechanism To: Domenico Cerasuolo Cc: vitaly.wool@konsulko.com, minchan@kernel.org, senozhatsky@chromium.org, linux-mm@kvack.org, ddstreet@ieee.org, sjenning@redhat.com, nphamcs@gmail.com, hannes@cmpxchg.org, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 3FF35160009 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: cmn4w19fhyng5f7ioxoyftrppz88pbo9 X-HE-Tag: 1686593925-321659 X-HE-Meta: U2FsdGVkX19LU6ItsKqcL2a92tlVPPkF980e0wcn3+xY2m7OX7Q+opEJ1ao8jv6tVsC+DzeGTrvBxhmw5D1cc8Mc3v68h7Hl3eoIA87nGjELROyQth1p3i8IUIylujGSQlCyNX0LQ6hUgIvGAwkJNmlHr3Ud+7wwEFiRQvsMr+7truS4HNYrQgYub8q8PIhZZyAWZvS+1KbHHGFY6egDdruUyJuHJs2pWfGYfDT6RcvfZEhSy9c58AetgmF0feYMMBaG+zTwlLWq0lXTzWnOojCEiELL9f4+BvXEjsoKGFb+99Y1o1m0+nZdLZI4DkCJ/HzV5hmPdaPgQMYjCkqHZ5R8cLaPKYQFXHf0KSgInpLiCFOMxifVzaGl9r0+P6tJD4r5wusSeSOWQwtntGqJdIu2Ke1UPSfy0n+tnOUgQOI4cNqf2VVs2OYDzi1kohaAxpqb8CjXbD0OVr7Og4L7sPqCEmdquoACOqwWqGCIULr4wiZ0igOTSVf03RdG2vcxlKl+mUu+klnsdwfniS+RtU/OnwK4TQJZPcCDY3v+2XyKnqtPrAisdv4GgsA53Alzo5PhdzocWPSrUOSV0SE/i4vTLxcy/mM2wxh9PDAjaK+g6aRZ+ExNFNHJsNuS2dEn1X5VZ5/8ygFfa1k4WxaxQFpjdERbTv6lOk22XNEv7AtBMzCnnNSPqaZiFZgVOeZ1tS/UXjSOQee/ivWwG1NngQl1fpyHeKTGShNpTigBnFGNy0Y11UrXY7Vv/KPy+xJknR7/Y1MH5bPjzYTTfk93tkssmMDhNuQ9SNjMm3ccMFInyPfi8CsErJMP/iVw+iXHrXHwcOAZvGJ/sQ7y4xaS4t7P2UU4sxdQlCSAHBzVKYusmnx+5t375jKgVCqNKqEQMNIMPFAYy/Z/lCdSTWjSPwtqtNs1CcqLcPWKRZ6lWYdY3304A1ZwhuBo+B/pvuFkts3n52vSZF0FcY82DyR VQLAaj8l IOQ4mW5bRjWsOXFMZ2b9N5LBdyqphp9pZI5XPkl13lh+yxDDjF5IjqwiuCXRNb/YQAO08QI9la+eotpoBSERWFPC7/oosa0bxQca/5Sv4dxALMqhUei5av/DeeB5OQAIvObQMYsABVKOPOTSb07y/nXwptuTl5HLASvwVVzsLQL2U/i9vjurt5tkQsOfM7nDkSHBn03DXRjg6ameJvqUCGQYrZeDcyh8H7ghyPDMrR3KJp1Uh2C9NMye3SDqa3kRCyO+1XsbtzZEcN7HsIoMAzayqeg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 12, 2023 at 2:38=E2=80=AFAM Domenico Cerasuolo wrote: > > Each zpool driver (zbud, z3fold and zsmalloc) implements its own shrink > function, which is called from zpool_shrink. However, with this commit, > a unified shrink function is added to zswap. The ultimate goal is to > eliminate the need for zpool_shrink once all zpool implementations have > dropped their shrink code. > > To ensure the functionality of each commit, this change focuses solely > on adding the mechanism itself. No modifications are made to > the backends, meaning that functionally, there are no immediate changes. > The zswap mechanism will only come into effect once the backends have > removed their shrink code. The subsequent commits will address the > modifications needed in the backends. > > Acked-by: Nhat Pham > Tested-by: Yosry Ahmed Reviewed-by: Yosry Ahmed > Signed-off-by: Domenico Cerasuolo > --- > mm/zswap.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 92 insertions(+), 5 deletions(-) > > diff --git a/mm/zswap.c b/mm/zswap.c > index 9fa86265f6dd..0024ec5ed574 100644 > --- a/mm/zswap.c > +++ b/mm/zswap.c > @@ -154,6 +154,12 @@ struct crypto_acomp_ctx { > struct mutex *mutex; > }; > > +/* > + * The lock ordering is zswap_tree.lock -> zswap_pool.lru_lock. > + * The only case where lru_lock is not acquired while holding tree.lock = is > + * when a zswap_entry is taken off the lru for writeback, in that case i= t > + * needs to be verified that it's still valid in the tree. > + */ > struct zswap_pool { > struct zpool *zpool; > struct crypto_acomp_ctx __percpu *acomp_ctx; > @@ -163,6 +169,8 @@ struct zswap_pool { > struct work_struct shrink_work; > struct hlist_node node; > char tfm_name[CRYPTO_MAX_ALG_NAME]; > + struct list_head lru; > + spinlock_t lru_lock; > }; > > /* > @@ -180,10 +188,12 @@ struct zswap_pool { > * be held while changing the refcount. Since the lock must > * be held, there is no reason to also make refcount atomic. > * length - the length in bytes of the compressed page data. Needed dur= ing > - * decompression. For a same value filled page length is 0. > + * decompression. For a same value filled page length is 0, and= both > + * pool and lru are invalid and must be ignored. > * pool - the zswap_pool the entry's data is in > * handle - zpool allocation handle that stores the compressed page data > * value - value of the same-value filled pages which have same content > + * lru - handle to the pool's lru used to evict pages. > */ > struct zswap_entry { > struct rb_node rbnode; > @@ -196,6 +206,7 @@ struct zswap_entry { > unsigned long value; > }; > struct obj_cgroup *objcg; > + struct list_head lru; > }; > > struct zswap_header { > @@ -368,6 +379,12 @@ static void zswap_free_entry(struct zswap_entry *ent= ry) > if (!entry->length) > atomic_dec(&zswap_same_filled_pages); > else { > + /* zpool_evictable will be removed once all 3 backends ha= ve migrated */ > + if (!zpool_evictable(entry->pool->zpool)) { > + spin_lock(&entry->pool->lru_lock); > + list_del(&entry->lru); > + spin_unlock(&entry->pool->lru_lock); > + } > zpool_free(entry->pool->zpool, entry->handle); > zswap_pool_put(entry->pool); > } > @@ -588,14 +605,72 @@ static struct zswap_pool *zswap_pool_find_get(char = *type, char *compressor) > return NULL; > } > > +static int zswap_reclaim_entry(struct zswap_pool *pool) > +{ > + struct zswap_header *zhdr; > + struct zswap_entry *entry; > + struct zswap_tree *tree; > + pgoff_t swpoffset; > + int ret; > + > + /* Get an entry off the LRU */ > + spin_lock(&pool->lru_lock); > + if (list_empty(&pool->lru)) { > + spin_unlock(&pool->lru_lock); > + return -EINVAL; > + } > + entry =3D list_last_entry(&pool->lru, struct zswap_entry, lru); > + list_del_init(&entry->lru); > + zhdr =3D zpool_map_handle(pool->zpool, entry->handle, ZPOOL_MM_RO= ); > + tree =3D zswap_trees[swp_type(zhdr->swpentry)]; > + zpool_unmap_handle(pool->zpool, entry->handle); > + /* > + * Once the lru lock is dropped, the entry might get freed. The > + * swpoffset is copied to the stack, and entry isn't deref'd agai= n > + * until the entry is verified to still be alive in the tree. > + */ > + swpoffset =3D swp_offset(zhdr->swpentry); > + spin_unlock(&pool->lru_lock); > + > + /* Check for invalidate() race */ > + spin_lock(&tree->lock); > + if (entry !=3D zswap_rb_search(&tree->rbroot, swpoffset)) { > + ret =3D -EAGAIN; > + goto unlock; > + } > + /* Hold a reference to prevent a free during writeback */ > + zswap_entry_get(entry); > + spin_unlock(&tree->lock); > + > + ret =3D zswap_writeback_entry(pool->zpool, entry->handle); > + > + spin_lock(&tree->lock); > + if (ret) { > + /* Writeback failed, put entry back on LRU */ > + spin_lock(&pool->lru_lock); > + list_move(&entry->lru, &pool->lru); > + spin_unlock(&pool->lru_lock); > + } > + > + /* Drop local reference */ > + zswap_entry_put(tree, entry); > +unlock: > + spin_unlock(&tree->lock); > + return ret ? -EAGAIN : 0; > +} > + > static void shrink_worker(struct work_struct *w) > { > struct zswap_pool *pool =3D container_of(w, typeof(*pool), > shrink_work); > int ret, failures =3D 0; > > + /* zpool_evictable will be removed once all 3 backends have migra= ted */ > do { > - ret =3D zpool_shrink(pool->zpool, 1, NULL); > + if (zpool_evictable(pool->zpool)) > + ret =3D zpool_shrink(pool->zpool, 1, NULL); > + else > + ret =3D zswap_reclaim_entry(pool); > if (ret) { > zswap_reject_reclaim_fail++; > if (ret !=3D -EAGAIN) > @@ -659,6 +734,8 @@ static struct zswap_pool *zswap_pool_create(char *typ= e, char *compressor) > */ > kref_init(&pool->kref); > INIT_LIST_HEAD(&pool->list); > + INIT_LIST_HEAD(&pool->lru); > + spin_lock_init(&pool->lru_lock); > INIT_WORK(&pool->shrink_work, shrink_worker); > > zswap_pool_debug("created", pool); > @@ -1274,7 +1351,7 @@ static int zswap_frontswap_store(unsigned type, pgo= ff_t offset, > } > > /* store */ > - hlen =3D zpool_evictable(entry->pool->zpool) ? sizeof(zhdr) : 0; > + hlen =3D sizeof(zhdr); > gfp =3D __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; > if (zpool_malloc_support_movable(entry->pool->zpool)) > gfp |=3D __GFP_HIGHMEM | __GFP_MOVABLE; > @@ -1317,6 +1394,12 @@ static int zswap_frontswap_store(unsigned type, pg= off_t offset, > zswap_entry_put(tree, dupentry); > } > } while (ret =3D=3D -EEXIST); > + /* zpool_evictable will be removed once all 3 backends have migra= ted */ > + if (entry->length && !zpool_evictable(entry->pool->zpool)) { > + spin_lock(&entry->pool->lru_lock); > + list_add(&entry->lru, &entry->pool->lru); > + spin_unlock(&entry->pool->lru_lock); > + } > spin_unlock(&tree->lock); > > /* update stats */ > @@ -1398,8 +1481,7 @@ static int zswap_frontswap_load(unsigned type, pgof= f_t offset, > /* decompress */ > dlen =3D PAGE_SIZE; > src =3D zpool_map_handle(entry->pool->zpool, entry->handle, ZPOOL= _MM_RO); > - if (zpool_evictable(entry->pool->zpool)) > - src +=3D sizeof(struct zswap_header); > + src +=3D sizeof(struct zswap_header); > > if (!zpool_can_sleep_mapped(entry->pool->zpool)) { > memcpy(tmp, src, entry->length); > @@ -1432,6 +1514,11 @@ static int zswap_frontswap_load(unsigned type, pgo= ff_t offset, > if (!ret && zswap_exclusive_loads_enabled) { > zswap_invalidate_entry(tree, entry); > *exclusive =3D true; > + } else if (entry->length && !zpool_evictable(entry->pool->zpool))= { > + /* zpool_evictable will be removed once all 3 backends ha= ve migrated */ > + spin_lock(&entry->pool->lru_lock); > + list_move(&entry->lru, &entry->pool->lru); > + spin_unlock(&entry->pool->lru_lock); > } > spin_unlock(&tree->lock); > > -- > 2.34.1 >