AI Trajectory

Trajectory: (Expand/Collapse All)
Seq	Timestamp	Type	Name	Duration
0/0	2026/08/18 08:04	flow	patch-triage	5m Args: null Results: { "EnableConfigs": [ "BLK_CGROUP", "IOSCHED_BFQ", "BFQ_GROUP_IOSCHED", "BLK_CGROUP_IOCOST", "BLK_CGROUP_IOLATENCY", "BLK_CGROUP_IOPRIO", "NVME_FC", "BLK_CGROUP_FC_APPID", "BTRFS_FS" ], "FocusSymbols": [ "blkg_conf_prep", "blkcg_punt_bio_submit", "blkg_init_queue", "blkg_exit_queue", "blkcg_init_disk", "blkcg_exit_disk" ], "KMSANReasoning": "The patch refactors the block cgroup (blkcg) association for `bio` structures, replacing the direct `bi_blkg` pointer with `bi_blkcg` and dynamically looking up or pinning the queue-local `blkcg_gq` (blkg) using a hash table instead of a radix tree.\n\nThese changes involve modifying internal kernel structures (`bio`, `request_queue`, `blkcg_gq`) and their initialization/cleanup routines (`bio_init`, `bio_uninit`, `bio_clear_blkcg`). No new structures are copied to user space, nor are there any new buffers or complex data structures allocated without proper initialization. The modified fields in `bio` are explicitly initialized to `NULL` or appropriate values in `bio_init()`.\n\nAny bugs introduced by these changes would likely manifest as NULL pointer dereferences, use-after-free (e.g., incorrect reference counting of `blkcg` or `blkg`), or memory leaks, all of which are effectively caught by standard KASAN and other default debugging tools. There is no risk of uninitialized memory reads or kernel-to-user info-leaks that would necessitate a dedicated KMSAN fuzzing session.", "NeedsKMSAN": false, "Reasoning": "The patch significantly refactors the block cgroup (blkcg) subsystem by changing how `bio`s are associated with block cgroups. Instead of directly storing a queue-specific `blkcg_gq` (blkg) in `bio-\u003ebi_blkg`, it now stores the generic `blkcg` in `bio-\u003ebi_blkcg` and dynamically looks up or pins the queue-specific `blkg` using a new `BIO_BLKG_REF` flag. It also replaces the `radix_tree` used for `blkg` lookups with an `rhashtable`. These are core functional changes to the block layer and cgroup IO controllers, affecting IO scheduling, throttling, and accounting.", "WorthFuzzing": true }
1/1	2026/08/18 08:04	action	read-patch-diff	0m Args: null Results: { "PatchDiff": "commit abc0b1a65e87d979eec053ca54836b90fdec9729\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Tue Aug 18 08:04:15 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst\nindex 14b8c571c0d14..bbd79931d5ab5 100644\n--- a/Documentation/admin-guide/cgroup-v2.rst\n+++ b/Documentation/admin-guide/cgroup-v2.rst\n@@ -3235,7 +3235,7 @@ the configuration, the bio may be executed at a lower priority and if\n the writeback session is holding shared resources, e.g. a journal\n entry, may lead to priority inversion. There is no one easy solution\n for the problem. Filesystems can try to work around specific problem\n-cases by skipping wbc_init_bio() and using bio_associate_blkg()\n+cases by skipping wbc_init_bio() and using bio_associate_blkcg()\n directly.\n \n \ndiff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c\nindex ef9d63e8542ed..4a3975f9ff744 100644\n--- a/block/bfq-cgroup.c\n+++ b/block/bfq-cgroup.c\n@@ -363,11 +363,13 @@ void bfqg_and_blkg_put(struct bfq_group bfqg)\n \n void bfqg_stats_update_legacy_io(struct request_queue q, struct request rq)\n {\n-\tstruct bfq_group bfqg = blkg_to_bfqg(rq-\u003ebio-\u003ebi_blkg);\n+\tstruct blkcg_gq blkg = bio_blkg_lookup(rq-\u003ebio);\n+\tstruct bfq_group bfqg;\n \n-\tif (!bfqg)\n+\tif (!blkg)\n \t\treturn;\n \n+\tbfqg = blkg_to_bfqg(blkg);\n \tblkg_rwstat_add(\u0026bfqg-\u003estats.bytes, rq-\u003ecmd_flags, blk_rq_bytes(rq));\n \tblkg_rwstat_add(\u0026bfqg-\u003estats.ios, rq-\u003ecmd_flags, 1);\n }\n@@ -606,7 +608,7 @@ static void bfq_link_bfqg(struct bfq_data bfqd, struct bfq_group bfqg)\n \n struct bfq_group bfq_bio_bfqg(struct bfq_data bfqd, struct bio bio)\n {\n-\tstruct blkcg_gq blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg_gq blkg = bio_blkg_lookup(bio);\n \tstruct bfq_group bfqg;\n \n \twhile (blkg) {\n@@ -614,15 +616,17 @@ struct bfq_group bfq_bio_bfqg(struct bfq_data bfqd, struct bio bio)\n \t\t\tblkg = blkg-\u003eparent;\n \t\t\tcontinue;\n \t\t}\n+\n \t\tbfqg = blkg_to_bfqg(blkg);\n \t\tif (bfqg-\u003epd.online) {\n-\t\t\tbio_associate_blkg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\n+\t\t\tbio_associate_blkcg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\n \t\t\treturn bfqg;\n \t\t}\n \t\tblkg = blkg-\u003eparent;\n \t}\n-\tbio_associate_blkg_from_css(bio,\n-\t\t\t\t\u0026bfqg_to_blkg(bfqd-\u003eroot_group)-\u003eblkcg-\u003ecss);\n+\n+\tblkg = bfqg_to_blkg(bfqd-\u003eroot_group);\n+\tbio_associate_blkcg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\n \treturn bfqd-\u003eroot_group;\n }\n \ndiff --git a/block/bfq-iosched.c b/block/bfq-iosched.c\nindex 0f75301b3115f..d4f657bc7a4e5 100644\n--- a/block/bfq-iosched.c\n+++ b/block/bfq-iosched.c\n@@ -128,6 +128,7 @@\n \n #include \"elevator.h\"\n #include \"blk.h\"\n+#include \"blk-cgroup.h\"\n #include \"blk-mq.h\"\n #include \"blk-mq-sched.h\"\n #include \"bfq-iosched.h\"\n@@ -2452,15 +2453,16 @@ static bool bfq_bio_merge(struct request_queue q, struct bio bio,\n \tstruct request free = NULL;\n \tbool ret;\n \n+#ifdef CONFIG_BFQ_GROUP_IOSCHED\n+\t/* blkg creation takes q-\u003equeue_lock, so do it before bfqd-\u003elock. /\n+\tif (bic)\n+\t\tbio_blkg(bio);\n+#endif\n+\n \tspin_lock_irq(\u0026bfqd-\u003elock);\n \n \tif (bic) {\n-\t\t/\n-\t\t * Make sure cgroup info is uptodate for current process before\n-\t\t * considering the merge.\n-\t\t /\n \t\tbfq_bic_update_cgroup(bic, bio);\n-\n \t\tbfqd-\u003ebio_bfqq = bic_to_bfqq(bic, op_is_sync(bio-\u003ebi_opf),\n \t\t\t\t\t bfq_actuator_index(bfqd, bio));\n \t} else {\n@@ -6245,6 +6247,13 @@ static void bfq_insert_request(struct blk_mq_hw_ctx hctx, struct request rq,\n \tLIST_HEAD(free);\n \n #ifdef CONFIG_BFQ_GROUP_IOSCHED\n+\t/\n+\t * Pin the blkg used to look up bfqg. If this is the first IO for\n+\t * the blkcg on this queue, create the bfqg before holding bfqd-\u003elock.\n+\t /\n+\tif (rq-\u003ebio \u0026\u0026 !bio_flagged(rq-\u003ebio, BIO_BLKG_REF))\n+\t\tbio_blkg(rq-\u003ebio);\n+\n \tif (!cgroup_subsys_on_dfl(io_cgrp_subsys) \u0026\u0026 rq-\u003ebio)\n \t\tbfqg_stats_update_legacy_io(q, rq);\n #endif\ndiff --git a/block/bio.c b/block/bio.c\nindex 898b2f5ef8c81..6c84042adf5dd 100644\n--- a/block/bio.c\n+++ b/block/bio.c\n@@ -179,12 +179,7 @@ static inline gfp_t try_alloc_gfp(gfp_t gfp)\n \n void bio_uninit(struct bio bio)\n {\n-#ifdef CONFIG_BLK_CGROUP\n-\tif (bio-\u003ebi_blkg) {\n-\t\tblkg_put(bio-\u003ebi_blkg);\n-\t\tbio-\u003ebi_blkg = NULL;\n-\t}\n-#endif\n+\tbio_clear_blkcg(bio);\n \tif (bio_integrity(bio))\n \t\tbio_integrity_free(bio);\n \n@@ -233,10 +228,10 @@ void bio_init(struct bio bio, struct block_device bdev, struct bio_vec table,\n \tbio-\u003ebi_end_io = NULL;\n \tbio-\u003ebi_private = NULL;\n #ifdef CONFIG_BLK_CGROUP\n-\tbio-\u003ebi_blkg = NULL;\n+\tbio-\u003ebi_blkcg = NULL;\n \tbio-\u003eissue_time_ns = 0;\n \tif (bdev)\n-\t\tbio_associate_blkg(bio);\n+\t\tbio_associate_blkcg(bio);\n #ifdef CONFIG_BLK_CGROUP_IOCOST\n \tbio-\u003ebi_iocost_cost = 0;\n #endif\n@@ -281,7 +276,7 @@ void bio_reset(struct bio bio, struct block_device bdev, blk_opf_t opf)\n \tbio-\u003ebi_io_vec = bv;\n \tbio-\u003ebi_bdev = bdev;\n \tif (bio-\u003ebi_bdev)\n-\t\tbio_associate_blkg(bio);\n+\t\tbio_associate_blkcg(bio);\n \tbio-\u003ebi_opf = opf;\n }\n EXPORT_SYMBOL(bio_reset);\n@@ -866,7 +861,7 @@ static int __bio_clone(struct bio bio, struct bio bio_src, gfp_t gfp)\n \t\tif (bio-\u003ebi_bdev == bio_src-\u003ebi_bdev \u0026\u0026\n \t\t bio_flagged(bio_src, BIO_REMAPPED))\n \t\t\tbio_set_flag(bio, BIO_REMAPPED);\n-\t\tbio_clone_blkg_association(bio, bio_src);\n+\t\tbio_clone_blkcg_association(bio, bio_src);\n \t}\n \n \tif (bio_crypt_clone(bio, bio_src, gfp) \u003c 0)\n@@ -1910,17 +1905,12 @@ void bio_endio(struct bio bio)\n \t\tgoto again;\n \t}\n \n-#ifdef CONFIG_BLK_CGROUP\n \t/\n \t Release cgroup info. We shouldn't have to do this here, but quite\n \t * a few callers of bio_init fail to call bio_uninit, so we cover up\n \t * for that here at least for now.\n \t /\n-\tif (bio-\u003ebi_blkg) {\n-\t\tblkg_put(bio-\u003ebi_blkg);\n-\t\tbio-\u003ebi_blkg = NULL;\n-\t}\n-#endif\n+\tbio_clear_blkcg(bio);\n \n \tif (bio_flagged(bio, BIO_COMPLETE_IN_TASK) \u0026\u0026 bio_in_atomic())\n \t\t__bio_complete_in_task(bio);\ndiff --git a/block/blk-cgroup-fc-appid.c b/block/blk-cgroup-fc-appid.c\nindex 3ec21333f393b..7589c62099898 100644\n--- a/block/blk-cgroup-fc-appid.c\n+++ b/block/blk-cgroup-fc-appid.c\n@@ -50,8 +50,14 @@ EXPORT_SYMBOL_GPL(blkcg_set_fc_appid);\n /\n char blkcg_get_fc_appid(struct bio bio)\n {\n-\tif (!bio-\u003ebi_blkg \|\| bio-\u003ebi_blkg-\u003eblkcg-\u003efc_app_id[0] == '\\0')\n+\tstruct blkcg blkcg = bio_blkcg(bio);\n+\n+\tif (!blkcg)\n+\t\treturn NULL;\n+\n+\tif (blkcg-\u003efc_app_id[0] == '\\0')\n \t\treturn NULL;\n-\treturn bio-\u003ebi_blkg-\u003eblkcg-\u003efc_app_id;\n+\n+\treturn blkcg-\u003efc_app_id;\n }\n EXPORT_SYMBOL_GPL(blkcg_get_fc_appid);\ndiff --git a/block/blk-cgroup.c b/block/blk-cgroup.c\nindex 2b5c29434e426..2c758c26bce60 100644\n--- a/block/blk-cgroup.c\n+++ b/block/blk-cgroup.c\n@@ -66,6 +66,13 @@ static DEFINE_RAW_SPINLOCK(blkg_stat_lock);\n \n #define BLKG_DESTROY_BATCH_SIZE 64\n \n+const struct rhashtable_params blkg_hash_params = {\n+\t.key_len\t\t= sizeof_field(struct blkcg_gq, blkcg_id),\n+\t.key_offset\t\t= offsetof(struct blkcg_gq, blkcg_id),\n+\t.head_offset\t\t= offsetof(struct blkcg_gq, q_hash_node),\n+\t.automatic_shrinking\t= true,\n+};\n+\n /\n * Lockless lists for tracking IO stats update\n \n@@ -173,10 +180,6 @@ static void __blkg_release(struct rcu_head rcu)\n {\n \tstruct blkcg_gq blkg = container_of(rcu, struct blkcg_gq, rcu_head);\n \n-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n-\tWARN_ON(!bio_list_empty(\u0026blkg-\u003easync_bios));\n-#endif\n-\n \tblkg_free(blkg);\n }\n \n@@ -194,6 +197,16 @@ static void blkg_release(struct percpu_ref ref)\n \tstruct blkcg blkcg = blkg-\u003eblkcg;\n \tint cpu;\n \n+\t/\n+\t * A blkg that was never inserted into q-\u003eblkg_list has no hash\n+\t * entry. This happens when allocation or creation fails before\n+\t * rhashtable_insert_fast() succeeds.\n+\t /\n+\tif (!list_empty(\u0026blkg-\u003eq_node))\n+\t\tWARN_ON_ONCE(rhashtable_remove_fast(\u0026blkg-\u003eq-\u003eblkg_hash,\n+\t\t\t\t\t\t \u0026blkg-\u003eq_hash_node,\n+\t\t\t\t\t\t blkg_hash_params));\n+\n \t/\n \t * Flush all the non-empty percpu lockless lists before releasing\n \t * us, given these stat belongs to us.\n@@ -209,19 +222,18 @@ static void blkg_release(struct percpu_ref ref)\n #ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n static struct workqueue_struct blkcg_punt_bio_wq;\n \n-static void blkg_async_bio_workfn(struct work_struct work)\n+static void blkcg_async_bio_workfn(struct work_struct work)\n {\n-\tstruct blkcg_gq blkg = container_of(work, struct blkcg_gq,\n-\t\t\t\t\t async_bio_work);\n+\tstruct blkcg blkcg = container_of(work, struct blkcg, async_bio_work);\n \tstruct bio_list bios = BIO_EMPTY_LIST;\n \tstruct bio bio;\n \tstruct blk_plug plug;\n \tbool need_plug = false;\n \n-\t/ as long as there are pending bios, @blkg can't go away /\n-\tspin_lock(\u0026blkg-\u003easync_bio_lock);\n-\tbio_list_merge_init(\u0026bios, \u0026blkg-\u003easync_bios);\n-\tspin_unlock(\u0026blkg-\u003easync_bio_lock);\n+\t/ as long as there are pending bios, @blkcg can't go away /\n+\tspin_lock(\u0026blkcg-\u003easync_bio_lock);\n+\tbio_list_merge_init(\u0026bios, \u0026blkcg-\u003easync_bios);\n+\tspin_unlock(\u0026blkcg-\u003easync_bio_lock);\n \n \t/ start plug only when bio_list contains at least 2 bios /\n \tif (bios.head \u0026\u0026 bios.head-\u003ebi_next) {\n@@ -242,15 +254,15 @@ static void blkg_async_bio_workfn(struct work_struct work)\n /\n void blkcg_punt_bio_submit(struct bio bio)\n {\n-\tstruct blkcg_gq blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg blkcg = bio_blkcg(bio);\n \n-\tif (blkg-\u003eparent) {\n-\t\tspin_lock(\u0026blkg-\u003easync_bio_lock);\n-\t\tbio_list_add(\u0026blkg-\u003easync_bios, bio);\n-\t\tspin_unlock(\u0026blkg-\u003easync_bio_lock);\n-\t\tqueue_work(blkcg_punt_bio_wq, \u0026blkg-\u003easync_bio_work);\n+\tif (blkcg \u0026\u0026 cgroup_parent(blkcg-\u003ecss.cgroup)) {\n+\t\tspin_lock(\u0026blkcg-\u003easync_bio_lock);\n+\t\tbio_list_add(\u0026blkcg-\u003easync_bios, bio);\n+\t\tspin_unlock(\u0026blkcg-\u003easync_bio_lock);\n+\t\tqueue_work(blkcg_punt_bio_wq, \u0026blkcg-\u003easync_bio_work);\n \t} else {\n-\t\t/* never bounce for the root cgroup /\n+\t\t/ Never bounce if there is no non-root blkcg to queue on. /\n \t\tsubmit_bio(bio);\n \t}\n }\n@@ -278,9 +290,13 @@ subsys_initcall(blkcg_punt_bio_init);\n /\n struct cgroup_subsys_state bio_blkcg_css(struct bio bio)\n {\n-\tif (!bio \|\| !bio-\u003ebi_blkg)\n+\tstruct blkcg blkcg;\n+\n+\tif (!bio)\n \t\treturn NULL;\n-\treturn \u0026bio-\u003ebi_blkg-\u003eblkcg-\u003ecss;\n+\n+\tblkcg = bio_blkcg(bio);\n+\treturn blkcg ? \u0026blkcg-\u003ecss : NULL;\n }\n EXPORT_SYMBOL_GPL(bio_blkcg_css);\n \n@@ -327,12 +343,8 @@ static struct blkcg_gq blkg_alloc(struct blkcg blkcg, struct gendisk disk,\n \tblkg-\u003eq = disk-\u003equeue;\n \tINIT_LIST_HEAD(\u0026blkg-\u003eq_node);\n \tblkg-\u003eblkcg = blkcg;\n+\tblkg-\u003eblkcg_id = blkcg-\u003ecss.id;\n \tblkg-\u003eiostat.blkg = blkg;\n-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n-\tspin_lock_init(\u0026blkg-\u003easync_bio_lock);\n-\tbio_list_init(\u0026blkg-\u003easync_bios);\n-\tINIT_WORK(\u0026blkg-\u003easync_bio_work, blkg_async_bio_workfn);\n-#endif\n \n \tu64_stats_init(\u0026blkg-\u003eiostat.sync);\n \tfor_each_possible_cpu(cpu) {\n@@ -423,7 +435,8 @@ static struct blkcg_gq blkg_create(struct blkcg blkcg, struct gendisk disk,\n \n \t/ insert /\n \tspin_lock(\u0026blkcg-\u003elock);\n-\tret = radix_tree_insert(\u0026blkcg-\u003eblkg_tree, disk-\u003equeue-\u003eid, blkg);\n+\tret = rhashtable_insert_fast(\u0026disk-\u003equeue-\u003eblkg_hash,\n+\t\t\t\t \u0026blkg-\u003eq_hash_node, blkg_hash_params);\n \tif (likely(!ret)) {\n \t\thlist_add_head_rcu(\u0026blkg-\u003eblkcg_node, \u0026blkcg-\u003eblkg_list);\n \t\tlist_add(\u0026blkg-\u003eq_node, \u0026disk-\u003equeue-\u003eblkg_list);\n@@ -454,6 +467,17 @@ static struct blkcg_gq blkg_create(struct blkcg blkcg, struct gendisk disk,\n \treturn ERR_PTR(ret);\n }\n \n+/\n+ The root blkg holds a live reference while the disk is active, so walking\n+ * the parent chain always finds a blkg which can be pinned.\n+ /\n+static struct blkcg_gq blkg_lookup_tryget(struct blkcg_gq blkg)\n+{\n+\twhile (!blkg_tryget(blkg))\n+\t\tblkg = blkg-\u003eparent;\n+\treturn blkg;\n+}\n+\n /\n blkg_lookup_create - lookup blkg, try to create one if not there\n * @blkcg: blkcg of interest\n@@ -461,11 +485,13 @@ static struct blkcg_gq blkg_create(struct blkcg blkcg, struct gendisk disk,\n \n * Lookup blkg for the @blkcg - @disk pair. If it doesn't exist, try to\n * create one. blkg creation is performed recursively from blkcg_root such\n- * that all non-root blkg's have access to the parent blkg. This function\n- * should be called under RCU read lock and takes @disk-\u003equeue-\u003equeue_lock.\n+ * that all non-root blkg's have access to the parent blkg.\n \n- Returns the blkg or the closest blkg if blkg_create() fails as it walks\n- * down from root.\n+ * Must be called with @disk-\u003equeue-\u003equeue_lock held.\n+ \n+ Returns the closest blkg with an extra reference acquired. If\n+ * blkg_create() fails while walking down from root, the returned blkg may\n+ * belong to an ancestor of @blkcg. This function never returns %NULL.\n /\n static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg,\n \t\tstruct gendisk disk)\n@@ -473,12 +499,12 @@ static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg,\n \tstruct request_queue q = disk-\u003equeue;\n \tstruct blkcg_gq blkg;\n \n+\tlockdep_assert_held(\u0026q-\u003equeue_lock);\n+\n \trcu_read_lock();\n \tblkg = blkg_lookup(blkcg, q);\n \tif (blkg) {\n-\t\tif (blkcg != \u0026blkcg_root \u0026\u0026\n-\t\t blkg != rcu_dereference(blkcg-\u003eblkg_hint))\n-\t\t\trcu_assign_pointer(blkcg-\u003eblkg_hint, blkg);\n+\t\tblkg = blkg_lookup_tryget(blkg);\n \t\trcu_read_unlock();\n \t\treturn blkg;\n \t}\n@@ -514,7 +540,7 @@ static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg,\n \t\t\tbreak;\n \t}\n \n-\treturn blkg;\n+\treturn blkg_lookup_tryget(blkg);\n }\n \n static void blkg_destroy(struct blkcg_gq blkg)\n@@ -546,17 +572,8 @@ static void blkg_destroy(struct blkcg_gq blkg)\n \n \tblkg-\u003eonline = false;\n \n-\tradix_tree_delete(\u0026blkcg-\u003eblkg_tree, blkg-\u003eq-\u003eid);\n \thlist_del_init_rcu(\u0026blkg-\u003eblkcg_node);\n \n-\t/\n-\t Both setting lookup hint to and clearing it from @blkg are done\n-\t * under queue_lock. If it's not pointing to @blkg now, it never\n-\t * will. Hint assignment itself can race safely.\n-\t /\n-\tif (rcu_access_pointer(blkcg-\u003eblkg_hint) == blkg)\n-\t\trcu_assign_pointer(blkcg-\u003eblkg_hint, NULL);\n-\n \t/\n \t * Put the reference taken at the time of creation so that when all\n \t * queues are gone, group can be destroyed.\n@@ -882,18 +899,12 @@ int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol,\n \t\t\tgoto fail_exit;\n \t\t}\n \n-\t\tif (radix_tree_preload(GFP_KERNEL)) {\n-\t\t\tblkg_free(new_blkg);\n-\t\t\tret = -ENOMEM;\n-\t\t\tgoto fail_exit;\n-\t\t}\n-\n \t\tspin_lock_irq(\u0026q-\u003equeue_lock);\n \n \t\tif (!blkcg_policy_enabled(q, pol)) {\n \t\t\tblkg_free(new_blkg);\n \t\t\tret = -EOPNOTSUPP;\n-\t\t\tgoto fail_preloaded;\n+\t\t\tgoto fail_unlock;\n \t\t}\n \n \t\tblkg = blkg_lookup(pos, q);\n@@ -903,12 +914,10 @@ int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol,\n \t\t\tblkg = blkg_create(pos, disk, new_blkg);\n \t\t\tif (IS_ERR(blkg)) {\n \t\t\t\tret = PTR_ERR(blkg);\n-\t\t\t\tgoto fail_preloaded;\n+\t\t\t\tgoto fail_unlock;\n \t\t\t}\n \t\t}\n \n-\t\tradix_tree_preload_end();\n-\n \t\tif (pos == blkcg)\n \t\t\tgoto success;\n \t}\n@@ -917,8 +926,6 @@ int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol,\n \tctx-\u003eblkg = blkg;\n \treturn 0;\n \n-fail_preloaded:\n-\tradix_tree_preload_end();\n fail_unlock:\n \tspin_unlock_irq(\u0026q-\u003equeue_lock);\n fail_exit:\n@@ -1373,6 +1380,9 @@ static void blkcg_css_free(struct cgroup_subsys_state css)\n \n \tmutex_unlock(\u0026blkcg_pol_mutex);\n \n+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n+\tWARN_ON(!bio_list_empty(\u0026blkcg-\u003easync_bios));\n+#endif\n \tfree_percpu(blkcg-\u003elhead);\n \tkfree(blkcg);\n }\n@@ -1420,8 +1430,12 @@ blkcg_css_alloc(struct cgroup_subsys_state parent_css)\n \n \tspin_lock_init(\u0026blkcg-\u003elock);\n \trefcount_set(\u0026blkcg-\u003eonline_pin, 1);\n-\tINIT_RADIX_TREE(\u0026blkcg-\u003eblkg_tree, GFP_NOWAIT);\n \tINIT_HLIST_HEAD(\u0026blkcg-\u003eblkg_list);\n+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n+\tspin_lock_init(\u0026blkcg-\u003easync_bio_lock);\n+\tbio_list_init(\u0026blkcg-\u003easync_bios);\n+\tINIT_WORK(\u0026blkcg-\u003easync_bio_work, blkcg_async_bio_workfn);\n+#endif\n #ifdef CONFIG_CGROUP_WRITEBACK\n \tINIT_LIST_HEAD(\u0026blkcg-\u003ecgwb_list);\n #endif\n@@ -1457,17 +1471,22 @@ static int blkcg_css_online(struct cgroup_subsys_state css)\n \treturn 0;\n }\n \n-void blkg_init_queue(struct request_queue q)\n+int blkg_init_queue(struct request_queue q)\n {\n \tINIT_LIST_HEAD(\u0026q-\u003eblkg_list);\n \tmutex_init(\u0026q-\u003eblkcg_mutex);\n+\treturn rhashtable_init(\u0026q-\u003eblkg_hash, \u0026blkg_hash_params);\n+}\n+\n+void blkg_exit_queue(struct request_queue q)\n+{\n+\trhashtable_destroy(\u0026q-\u003eblkg_hash);\n }\n \n int blkcg_init_disk(struct gendisk disk)\n {\n \tstruct request_queue q = disk-\u003equeue;\n \tstruct blkcg_gq new_blkg, blkg;\n-\tbool preloaded;\n \n \t/\n \t If the queue is shared across disk rebind (e.g., SCSI), the\n@@ -1485,8 +1504,6 @@ int blkcg_init_disk(struct gendisk disk)\n \tif (!new_blkg)\n \t\treturn -ENOMEM;\n \n-\tpreloaded = !radix_tree_preload(GFP_KERNEL);\n-\n \t/ Make sure the root blkg exists. /\n \t/ spin_lock_irq can serve as RCU read-side critical section. /\n \tspin_lock_irq(\u0026q-\u003equeue_lock);\n@@ -1496,15 +1513,10 @@ int blkcg_init_disk(struct gendisk disk)\n \tq-\u003eroot_blkg = blkg;\n \tspin_unlock_irq(\u0026q-\u003equeue_lock);\n \n-\tif (preloaded)\n-\t\tradix_tree_preload_end();\n-\n \treturn 0;\n \n err_unlock:\n \tspin_unlock_irq(\u0026q-\u003equeue_lock);\n-\tif (preloaded)\n-\t\tradix_tree_preload_end();\n \treturn PTR_ERR(blkg);\n }\n \n@@ -2064,129 +2076,165 @@ void blkcg_add_delay(struct blkcg_gq blkg, u64 now, u64 delta)\n \tatomic64_add(delta, \u0026blkg-\u003edelay_nsec);\n }\n \n-static inline struct blkcg_gq blkg_lookup_tryget(struct blkcg_gq blkg)\n+/\n+ * Return the blkg pinned by @bio through BIO_BLKG_REF. The returned blkg is\n+ * already owned by @bio and no extra reference is acquired.\n+ /\n+static struct blkcg_gq bio_pinned_blkg(struct bio bio)\n {\n-retry:\n-\tif (blkg_tryget(blkg))\n-\t\treturn blkg;\n+\tstruct request_queue q = bdev_get_queue(bio-\u003ebi_bdev);\n+\tstruct blkcg_gq blkg;\n \n-\tblkg = blkg-\u003eparent;\n-\tif (blkg)\n-\t\tgoto retry;\n+\trcu_read_lock();\n+\tblkg = blkg_lookup_any(bio_blkcg(bio), q);\n+\trcu_read_unlock();\n \n-\treturn NULL;\n+\tWARN_ON_ONCE(!blkg);\n+\treturn blkg;\n+}\n+\n+static void bio_set_blkg_ref(struct bio bio, struct blkcg_gq blkg)\n+{\n+\tif (bio_blkcg(bio) != blkg-\u003eblkcg) {\n+\t\tcss_get(\u0026blkg-\u003eblkcg-\u003ecss);\n+\t\tbio_clear_blkcg(bio);\n+\t\tbio-\u003ebi_blkcg = blkg-\u003eblkcg;\n+\t}\n+\tbio_set_flag(bio, BIO_BLKG_REF);\n }\n+\n /\n- blkg_tryget_closest - try and get a blkg ref on the closet blkg\n+ * bio_blkg_lookup - look up a blkg associated with a bio\n * @bio: target bio\n- * @css: target css\n \n- As the failure mode here is to walk up the blkg tree, this ensure that the\n- * blkg-\u003eparent pointers are always valid. This returns the blkg that it ended\n- * up taking a reference on or %NULL if no reference was taken.\n+ * Return the blkg already pinned by @bio, or %NULL if @bio doesn't own a blkg\n+ * reference. Call bio_blkg() instead when a missing blkg should be created.\n /\n-static inline struct blkcg_gq blkg_tryget_closest(struct bio bio,\n-\t\tstruct cgroup_subsys_state css)\n+struct blkcg_gq bio_blkg_lookup(struct bio bio)\n {\n-\tstruct request_queue q = bio-\u003ebi_bdev-\u003ebd_queue;\n-\tstruct blkcg blkcg = css_to_blkcg(css);\n+\tif (!bio_flagged(bio, BIO_BLKG_REF))\n+\t\treturn NULL;\n+\treturn bio_pinned_blkg(bio);\n+}\n+EXPORT_SYMBOL_GPL(bio_blkg_lookup);\n+\n+/*\n+ bio_put_blkg_ref - drop the blkg reference pinned by a bio\n+ * @bio: target bio\n+ \n+ Drop the bio-owned blkg reference acquired by bio_blkg(), if any.\n+ /\n+void bio_put_blkg_ref(struct bio bio)\n+{\n+\tif (bio_flagged(bio, BIO_BLKG_REF)) {\n+\t\tstruct blkcg_gq blkg = bio_pinned_blkg(bio);\n+\n+\t\tif (blkg)\n+\t\t\tblkg_put(blkg);\n+\t\tbio_clear_flag(bio, BIO_BLKG_REF);\n+\t}\n+}\n+EXPORT_SYMBOL_GPL(bio_put_blkg_ref);\n+\n+/\n+ bio_blkg - look up the blkg associated with a bio\n+ * @bio: target bio\n+ \n+ Look up the queue-local blkg for @bio's current device and blkcg. If this\n+ * is the first policy use of @bio, create the missing blkg hierarchy if\n+ * necessary, pin the closest available blkg, and update @bio's blkcg\n+ * association if allocation falls back to an ancestor.\n+ /\n+struct blkcg_gq bio_blkg(struct bio bio)\n+{\n+\tstruct blkcg blkcg = bio_blkcg(bio);\n+\tstruct gendisk disk;\n+\tstruct request_queue q;\n \tstruct blkcg_gq blkg;\n \n-\trcu_read_lock();\n-\tblkg = blkg_lookup(blkcg, q);\n-\tif (likely(blkg))\n-\t\tblkg = blkg_lookup_tryget(blkg);\n-\trcu_read_unlock();\n+\tif (!blkcg \|\| !bio-\u003ebi_bdev)\n+\t\treturn NULL;\n \n-\tif (blkg)\n-\t\treturn blkg;\n+\tif (bio_flagged(bio, BIO_BLKG_REF))\n+\t\treturn bio_pinned_blkg(bio);\n+\n+\tdisk = bio-\u003ebi_bdev-\u003ebd_disk;\n+\tq = disk-\u003equeue;\n \n-\t/\n-\t * Fast path failed, we're probably issuing IO in this cgroup the first\n-\t * time, hold lock to create new blkg.\n-\t /\n \tspin_lock_irq(\u0026q-\u003equeue_lock);\n-\tblkg = blkg_lookup_create(blkcg, bio-\u003ebi_bdev-\u003ebd_disk);\n-\tif (blkg)\n-\t\tblkg = blkg_lookup_tryget(blkg);\n+\tblkg = blkg_lookup_create(blkcg, disk);\n \tspin_unlock_irq(\u0026q-\u003equeue_lock);\n \n+\tbio_set_blkg_ref(bio, blkg);\n \treturn blkg;\n }\n+EXPORT_SYMBOL_GPL(bio_blkg);\n \n /\n- bio_associate_blkg_from_css - associate a bio with a specified css\n+ * bio_associate_blkcg_from_css - associate a bio with a specified css\n * @bio: target bio\n * @css: target css\n \n- Associate @bio with the blkg found by combining the css's blkg and the\n- * request_queue of the @bio. An association failure is handled by walking up\n- * the blkg tree. Therefore, the blkg associated can be anything between @blkg\n- * and q-\u003eroot_blkg. This situation only happens when a cgroup is dying and\n- * then the remaining bios will spill to the closest alive blkg.\n+ * Associate @bio with the blkcg found from @css. The queue-local blkg is\n+ * created and pinned by bio_blkg() when blkcg policies need it. If @css\n+ * can't be referenced online, associate @bio with the root blkcg instead.\n \n- A reference will be taken on the blkg and will be released when @bio is\n+ * A reference will be taken on the blkcg and will be released when @bio is\n * freed.\n /\n-void bio_associate_blkg_from_css(struct bio bio,\n-\t\t\t\t struct cgroup_subsys_state css)\n+void bio_associate_blkcg_from_css(struct bio bio,\n+\t\t\t\t struct cgroup_subsys_state css)\n {\n-\tif (bio-\u003ebi_blkg)\n-\t\tblkg_put(bio-\u003ebi_blkg);\n+\tstruct blkcg blkcg;\n \n-\tif (css \u0026\u0026 css-\u003eparent) {\n-\t\tbio-\u003ebi_blkg = blkg_tryget_closest(bio, css);\n-\t} else {\n-\t\tblkg_get(bdev_get_queue(bio-\u003ebi_bdev)-\u003eroot_blkg);\n-\t\tbio-\u003ebi_blkg = bdev_get_queue(bio-\u003ebi_bdev)-\u003eroot_blkg;\n+\tif (!css \|\| !css_tryget_online(css)) {\n+\t\tcss = \u0026blkcg_root.css;\n+\t\tcss_get(css);\n \t}\n+\n+\tblkcg = css_to_blkcg(css);\n+\tif (bio_blkcg(bio) == blkcg) {\n+\t\tcss_put(css);\n+\t\treturn;\n+\t}\n+\n+\tbio_clear_blkcg(bio);\n+\tbio-\u003ebi_blkcg = blkcg;\n }\n-EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css);\n+EXPORT_SYMBOL_GPL(bio_associate_blkcg_from_css);\n \n /*\n- bio_associate_blkg - associate a bio with a blkg\n+ * bio_associate_blkcg - associate a bio with a blkcg\n * @bio: target bio\n \n- Associate @bio with the blkg found from the bio's css and request_queue.\n- * If one is not found, bio_lookup_blkg() creates the blkg. If a blkg is\n- * already associated, the css is reused and association redone as the\n- * request_queue may have changed.\n+ * Associate @bio with the blkcg found from the bio's css. If a blkcg is\n+ * already associated, keep it as blkcg association is not queue-local.\n /\n-void bio_associate_blkg(struct bio bio)\n+void bio_associate_blkcg(struct bio bio)\n {\n-\tstruct cgroup_subsys_state css;\n-\n \tif (blk_op_is_passthrough(bio-\u003ebi_opf))\n \t\treturn;\n \n-\tif (bio-\u003ebi_blkg) {\n-\t\tcss = bio_blkcg_css(bio);\n-\t\tbio_associate_blkg_from_css(bio, css);\n-\t} else {\n-\t\trcu_read_lock();\n-\t\tcss = blkcg_css();\n-\t\tif (!css_tryget_online(css))\n-\t\t\tcss = NULL;\n-\t\trcu_read_unlock();\n+\tif (bio_blkcg(bio))\n+\t\treturn;\n \n-\t\tbio_associate_blkg_from_css(bio, css);\n-\t\tif (css)\n-\t\t\tcss_put(css);\n-\t}\n+\trcu_read_lock();\n+\tbio_associate_blkcg_from_css(bio, blkcg_css());\n+\trcu_read_unlock();\n }\n-EXPORT_SYMBOL_GPL(bio_associate_blkg);\n+EXPORT_SYMBOL_GPL(bio_associate_blkcg);\n \n /*\n- bio_clone_blkg_association - clone blkg association from src to dst bio\n+ * bio_clone_blkcg_association - clone blkcg association from src to dst bio\n * @dst: destination bio\n * @src: source bio\n /\n-void bio_clone_blkg_association(struct bio dst, struct bio src)\n+void bio_clone_blkcg_association(struct bio dst, struct bio src)\n {\n-\tif (src-\u003ebi_blkg)\n-\t\tbio_associate_blkg_from_css(dst, bio_blkcg_css(src));\n+\tif (bio_blkcg(src))\n+\t\tbio_associate_blkcg_from_css(dst, bio_blkcg_css(src));\n }\n-EXPORT_SYMBOL_GPL(bio_clone_blkg_association);\n+EXPORT_SYMBOL_GPL(bio_clone_blkcg_association);\n \n static int blk_cgroup_io_type(struct bio bio)\n {\n@@ -2199,20 +2247,27 @@ static int blk_cgroup_io_type(struct bio bio)\n \n void blk_cgroup_bio_start(struct bio bio)\n {\n-\tstruct blkcg blkcg = bio-\u003ebi_blkg-\u003eblkcg;\n+\tstruct blkcg blkcg = bio_blkcg(bio);\n+\tstruct blkcg_gq blkg;\n \tint rwd = blk_cgroup_io_type(bio), cpu;\n \tstruct blkg_iostat_set bis;\n \tunsigned long flags;\n \n \tif (!cgroup_subsys_on_dfl(io_cgrp_subsys))\n \t\treturn;\n+\tif (!blkcg)\n+\t\treturn;\n \n \t/* Root-level stats are sourced from system-wide IO stats /\n \tif (!cgroup_parent(blkcg-\u003ecss.cgroup))\n \t\treturn;\n \n+\tblkg = bio_blkg_lookup(bio);\n+\tif (!blkg)\n+\t\treturn;\n+\n \tcpu = get_cpu();\n-\tbis = per_cpu_ptr(bio-\u003ebi_blkg-\u003eiostat_cpu, cpu);\n+\tbis = per_cpu_ptr(blkg-\u003eiostat_cpu, cpu);\n \tflags = u64_stats_update_begin_irqsave(\u0026bis-\u003esync);\n \n \t/\ndiff --git a/block/blk-cgroup.h b/block/blk-cgroup.h\nindex e67c698391294..eb76cc38b41ba 100644\n--- a/block/blk-cgroup.h\n+++ b/block/blk-cgroup.h\n@@ -19,6 +19,8 @@\n #include \u003clinux/kthread.h\u003e\n #include \u003clinux/blk-mq.h\u003e\n #include \u003clinux/llist.h\u003e\n+#include \u003clinux/rhashtable.h\u003e\n+#include \u003clinux/rcupdate.h\u003e\n #include \"blk.h\"\n \n struct blkcg_gq;\n@@ -56,9 +58,11 @@ struct blkg_iostat_set {\n struct blkcg_gq {\n \t/* Pointer to the associated request_queue /\n \tstruct request_queue\t\tq;\n+\tstruct rhash_head\t\tq_hash_node;\n \tstruct list_head\t\tq_node;\n \tstruct hlist_node\t\tblkcg_node;\n \tstruct blkcg\t\t\tblkcg;\n+\tint\t\t\t\tblkcg_id;\n \n \t/ all non-root blkcg_gq's are guaranteed to have access to parent /\n \tstruct blkcg_gq\t\t\tparent;\n@@ -73,14 +77,7 @@ struct blkcg_gq {\n \tstruct blkg_iostat_set\t\tiostat;\n \n \tstruct blkg_policy_data\t\tpd[BLKCG_MAX_POLS];\n-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n-\tspinlock_t\t\t\tasync_bio_lock;\n-\tstruct bio_list\t\t\tasync_bios;\n-#endif\n-\tunion {\n-\t\tstruct work_struct\tasync_bio_work;\n-\t\tstruct work_struct\tfree_work;\n-\t};\n+\tstruct work_struct\t\tfree_work;\n \n \tatomic_t\t\t\tuse_delay;\n \tatomic64_t\t\t\tdelay_nsec;\n@@ -98,8 +95,6 @@ struct blkcg {\n \t/ If there is block congestion on this cgroup. /\n \tatomic_t\t\t\tcongestion_count;\n \n-\tstruct radix_tree_root\t\tblkg_tree;\n-\tstruct blkcg_gq\t__rcu\t\tblkg_hint;\n \tstruct hlist_head\t\tblkg_list;\n \n \tstruct blkcg_policy_data\tcpd[BLKCG_MAX_POLS];\n@@ -111,6 +106,11 @@ struct blkcg {\n \t /\n \tstruct llist_head __percpu\tlhead;\n \n+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n+\tspinlock_t\t\t\tasync_bio_lock; / protects async_bios /\n+\tstruct bio_list\t\t\tasync_bios;\n+\tstruct work_struct\t\tasync_bio_work;\n+#endif\n #ifdef CONFIG_BLK_CGROUP_FC_APPID\n \tchar fc_app_id[FC_APPID_LEN];\n #endif\n@@ -124,6 +124,11 @@ static inline struct blkcg css_to_blkcg(struct cgroup_subsys_state css)\n \treturn css ? container_of(css, struct blkcg, css) : NULL;\n }\n \n+static inline struct blkcg bio_blkcg(struct bio bio)\n+{\n+\treturn bio-\u003ebi_blkcg;\n+}\n+\n /\n * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a\n * request_queue (q). This is used by blkcg policies which need to track\n@@ -192,8 +197,10 @@ struct blkcg_policy {\n \n extern struct blkcg blkcg_root;\n extern bool blkcg_debug_stats;\n+extern const struct rhashtable_params blkg_hash_params;\n \n-void blkg_init_queue(struct request_queue q);\n+int blkg_init_queue(struct request_queue q);\n+void blkg_exit_queue(struct request_queue q);\n int blkcg_init_disk(struct gendisk disk);\n void blkcg_exit_disk(struct gendisk disk);\n \n@@ -249,11 +256,33 @@ static inline bool bio_issue_as_root_blkg(struct bio bio)\n }\n \n /*\n- blkg_lookup - lookup blkg for the specified blkcg - q pair\n+ * blkg_lookup_any - lookup any blkg for the specified blkcg - q pair\n+ * @blkcg: blkcg of interest\n+ * @q: request_queue of interest\n+ \n+ Lookup a blkg for the @blkcg - @q pair, whether it is online or dying.\n+ \n+ Must be called in a RCU critical section.\n+ \n+ This does not acquire a reference. The caller must already hold one, or\n+ * have the blkg pinned by I/O.\n+ /\n+static inline struct blkcg_gq blkg_lookup_any(struct blkcg blkcg,\n+\t\t\t\t\t struct request_queue q)\n+{\n+\tRCU_LOCKDEP_WARN(!rcu_read_lock_held(),\n+\t\t\t \"blkg_lookup_any() requires an RCU read lock\");\n+\n+\treturn rhashtable_lookup(\u0026q-\u003eblkg_hash, \u0026blkcg-\u003ecss.id,\n+\t\t\t\t blkg_hash_params);\n+}\n+\n+/*\n+ blkg_lookup - lookup an online blkg for the specified blkcg - q pair\n * @blkcg: blkcg of interest\n * @q: request_queue of interest\n \n- Lookup blkg for the @blkcg - @q pair.\n+ * Lookup an online blkg for the @blkcg - @q pair.\n \n Must be called in a RCU critical section.\n /\n@@ -265,17 +294,15 @@ static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg,\n \tif (blkcg == \u0026blkcg_root)\n \t\treturn q-\u003eroot_blkg;\n \n-\tblkg = rcu_dereference_check(blkcg-\u003eblkg_hint,\n-\t\t\tlockdep_is_held(\u0026q-\u003equeue_lock));\n-\tif (blkg \u0026\u0026 blkg-\u003eq == q)\n-\t\treturn blkg;\n-\n-\tblkg = radix_tree_lookup(\u0026blkcg-\u003eblkg_tree, q-\u003eid);\n-\tif (blkg \u0026\u0026 blkg-\u003eq != q)\n+\tblkg = blkg_lookup_any(blkcg, q);\n+\tif (blkg \u0026\u0026 !READ_ONCE(blkg-\u003eonline))\n \t\tblkg = NULL;\n \treturn blkg;\n }\n \n+struct blkcg_gq bio_blkg_lookup(struct bio bio);\n+struct blkcg_gq bio_blkg(struct bio bio);\n+\n /\n blkg_to_pd - get policy private data\n * @blkg: blkg of interest\n@@ -343,6 +370,18 @@ static inline void blkg_put(struct blkcg_gq blkg)\n \tpercpu_ref_put(\u0026blkg-\u003erefcnt);\n }\n \n+static inline void bio_clear_blkcg(struct bio bio)\n+{\n+\tstruct blkcg blkcg = bio_blkcg(bio);\n+\n+\tbio_put_blkg_ref(bio);\n+\n+\tif (blkcg) {\n+\t\tcss_put(\u0026blkcg-\u003ecss);\n+\t\tbio-\u003ebi_blkcg = NULL;\n+\t}\n+}\n+\n /\n blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants\n * @d_blkg: loop cursor pointing to the current descendant\n@@ -472,7 +511,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq blkg)\n /\n static inline bool blk_cgroup_mergeable(struct request rq, struct bio bio)\n {\n-\treturn rq-\u003ebio-\u003ebi_blkg == bio-\u003ebi_blkg \u0026\u0026\n+\treturn bio_blkcg(rq-\u003ebio) == bio_blkcg(bio) \u0026\u0026\n \t\tbio_issue_as_root_blkg(rq-\u003ebio) == bio_issue_as_root_blkg(bio);\n }\n \n@@ -498,8 +537,13 @@ struct blkcg_policy {\n struct blkcg {\n };\n \n+static inline struct blkcg bio_blkcg(struct bio bio) { return NULL; }\n+static inline struct blkcg_gq bio_blkg_lookup(struct bio bio) { return NULL; }\n+static inline struct blkcg_gq bio_blkg(struct bio bio) { return NULL; }\n+static inline struct blkcg_gq blkg_lookup_any(struct blkcg blkcg, void key) { return NULL; }\n static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg, void key) { return NULL; }\n-static inline void blkg_init_queue(struct request_queue q) { }\n+static inline int blkg_init_queue(struct request_queue q) { return 0; }\n+static inline void blkg_exit_queue(struct request_queue q) { }\n static inline int blkcg_init_disk(struct gendisk disk) { return 0; }\n static inline void blkcg_exit_disk(struct gendisk disk) { }\n static inline int blkcg_policy_register(struct blkcg_policy pol) { return 0; }\n@@ -514,6 +558,7 @@ static inline struct blkg_policy_data blkg_to_pd(struct blkcg_gq blkg,\n static inline struct blkcg_gq pd_to_blkg(struct blkg_policy_data pd) { return NULL; }\n static inline void blkg_get(struct blkcg_gq blkg) { }\n static inline void blkg_put(struct blkcg_gq blkg) { }\n+static inline void bio_clear_blkcg(struct bio bio) { }\n static inline void blk_cgroup_bio_start(struct bio bio) { }\n static inline bool blk_cgroup_mergeable(struct request rq, struct bio bio) { return true; }\n \ndiff --git a/block/blk-core.c b/block/blk-core.c\nindex 196bccf27f58d..e396a316b8bf6 100644\n--- a/block/blk-core.c\n+++ b/block/blk-core.c\n@@ -303,6 +303,7 @@ static void blk_free_queue(struct request_queue q)\n \tif (queue_is_mq(q))\n \t\tblk_mq_release(q);\n \n+\tblkg_exit_queue(q);\n \tida_free(\u0026blk_queue_ida, q-\u003eid);\n \tlockdep_unregister_key(\u0026q-\u003eio_lock_cls_key);\n \tlockdep_unregister_key(\u0026q-\u003eq_lock_cls_key);\n@@ -481,7 +482,9 @@ struct request_queue blk_alloc_queue(struct queue_limits lim, int node_id)\n \tinit_waitqueue_head(\u0026q-\u003emq_freeze_wq);\n \tmutex_init(\u0026q-\u003emq_freeze_lock);\n \n-\tblkg_init_queue(q);\n+\terror = blkg_init_queue(q);\n+\tif (error)\n+\t\tgoto fail_stats;\n \n \t/\n \t * Init percpu_ref in atomic mode so that it's faster to shutdown.\n@@ -491,7 +494,7 @@ struct request_queue blk_alloc_queue(struct queue_limits lim, int node_id)\n \t\t\t\tblk_queue_usage_counter_release,\n \t\t\t\tPERCPU_REF_INIT_ATOMIC, GFP_KERNEL);\n \tif (error)\n-\t\tgoto fail_stats;\n+\t\tgoto fail_blkg;\n \tlockdep_register_key(\u0026q-\u003eio_lock_cls_key);\n \tlockdep_register_key(\u0026q-\u003eq_lock_cls_key);\n \tlockdep_init_map(\u0026q-\u003eio_lockdep_map, \"\u0026q-\u003eq_usage_counter(io)\",\n@@ -510,6 +513,8 @@ struct request_queue blk_alloc_queue(struct queue_limits lim, int node_id)\n \n \treturn q;\n \n+fail_blkg:\n+\tblkg_exit_queue(q);\n fail_stats:\n \tblk_free_queue_stats(q-\u003estats);\n fail_id:\ndiff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c\nindex 2a5c52ab74b4c..5ef4baab444bb 100644\n--- a/block/blk-crypto-fallback.c\n+++ b/block/blk-crypto-fallback.c\n@@ -187,7 +187,7 @@ static struct bio blk_crypto_alloc_enc_bio(struct bio bio_src,\n \tbio-\u003ebi_write_hint\t= bio_src-\u003ebi_write_hint;\n \tbio-\u003ebi_write_stream\t= bio_src-\u003ebi_write_stream;\n \tbio-\u003ebi_iter.bi_sector\t= bio_src-\u003ebi_iter.bi_sector;\n-\tbio_clone_blkg_association(bio, bio_src);\n+\tbio_clone_blkcg_association(bio, bio_src);\n \n \t/\n \t Move page array up in the allocated memory for the bio vecs as far as\ndiff --git a/block/blk-iocost.c b/block/blk-iocost.c\nindex 2745bffcd5eef..57f2b4d4af20a 100644\n--- a/block/blk-iocost.c\n+++ b/block/blk-iocost.c\n@@ -2686,7 +2686,7 @@ iocg_handle_over_budget(struct rq_qos rqos, struct ioc_gq iocg,\n \n static void ioc_rqos_throttle(struct rq_qos rqos, struct bio bio)\n {\n-\tstruct blkcg_gq blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg_gq blkg = bio_blkg(bio);\n \tstruct ioc ioc = rqos_to_ioc(rqos);\n \tstruct ioc_gq iocg = blkg_to_iocg(blkg);\n \tstruct ioc_now now;\n@@ -2775,7 +2775,7 @@ static void ioc_rqos_throttle(struct rq_qos rqos, struct bio bio)\n static void ioc_rqos_merge(struct rq_qos rqos, struct request rq,\n \t\t\t struct bio bio)\n {\n-\tstruct ioc_gq iocg = blkg_to_iocg(bio-\u003ebi_blkg);\n+\tstruct ioc_gq iocg = blkg_to_iocg(bio_blkg(bio));\n \tstruct ioc ioc = rqos_to_ioc(rqos);\n \tsector_t bio_end = bio_end_sector(bio);\n \tstruct ioc_now now;\n@@ -2833,9 +2833,13 @@ static void ioc_rqos_merge(struct rq_qos rqos, struct request rq,\n \n static void ioc_rqos_done_bio(struct rq_qos rqos, struct bio bio)\n {\n-\tstruct ioc_gq iocg = blkg_to_iocg(bio-\u003ebi_blkg);\n+\tstruct ioc_gq iocg;\n+\n+\tif (!bio-\u003ebi_iocost_cost)\n+\t\treturn;\n \n-\tif (iocg \u0026\u0026 bio-\u003ebi_iocost_cost)\n+\tiocg = blkg_to_iocg(bio_blkg_lookup(bio));\n+\tif (iocg)\n \t\tatomic64_add(bio-\u003ebi_iocost_cost, \u0026iocg-\u003edone_vtime);\n }\n \ndiff --git a/block/blk-iolatency.c b/block/blk-iolatency.c\nindex 2caa79a008ad1..7220edafd72b6 100644\n--- a/block/blk-iolatency.c\n+++ b/block/blk-iolatency.c\n@@ -463,7 +463,7 @@ static void check_scale_change(struct iolatency_grp iolat)\n static void blkcg_iolatency_throttle(struct rq_qos rqos, struct bio bio)\n {\n \tstruct blk_iolatency blkiolat = BLKIOLATENCY(rqos);\n-\tstruct blkcg_gq blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg_gq blkg = bio_blkg(bio);\n \tbool issue_as_root = bio_issue_as_root_blkg(bio);\n \n \tif (!blkiolat-\u003eenabled)\n@@ -590,11 +590,14 @@ static void blkcg_iolatency_done_bio(struct rq_qos rqos, struct bio bio)\n \tbool issue_as_root = bio_issue_as_root_blkg(bio);\n \tint inflight = 0;\n \n-\tblkg = bio-\u003ebi_blkg;\n-\tif (!blkg \|\| !bio_flagged(bio, BIO_QOS_THROTTLED))\n+\tif (!bio_flagged(bio, BIO_QOS_THROTTLED))\n \t\treturn;\n \n-\tiolat = blkg_to_lat(bio-\u003ebi_blkg);\n+\tblkg = bio_blkg_lookup(bio);\n+\tif (!blkg)\n+\t\treturn;\n+\n+\tiolat = blkg_to_lat(blkg);\n \tif (!iolat)\n \t\treturn;\n \ndiff --git a/block/blk-ioprio.c b/block/blk-ioprio.c\nindex 8fa8bca350629..5e3f8e49550da 100644\n--- a/block/blk-ioprio.c\n+++ b/block/blk-ioprio.c\n@@ -132,7 +132,7 @@ static struct blkcg_policy ioprio_policy = {\n \n void blkcg_set_ioprio(struct bio bio)\n {\n-\tstruct ioprio_blkcg blkcg = blkcg_to_ioprio_blkcg(bio-\u003ebi_blkg-\u003eblkcg);\n+\tstruct ioprio_blkcg blkcg = blkcg_to_ioprio_blkcg(bio_blkcg(bio));\n \tu16 prio;\n \n \tif (!blkcg \|\| blkcg-\u003eprio_policy == POLICY_NO_CHANGE)\ndiff --git a/block/blk-throttle.c b/block/blk-throttle.c\nindex ffc3b70065d4b..3828c38579002 100644\n--- a/block/blk-throttle.c\n+++ b/block/blk-throttle.c\n@@ -1792,7 +1792,7 @@ static bool tg_within_limit(struct throtl_grp tg, struct bio bio, bool rw)\n bool __blk_throtl_bio(struct bio bio)\n {\n \tstruct request_queue q = bdev_get_queue(bio-\u003ebi_bdev);\n-\tstruct blkcg_gq blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg_gq blkg = bio_blkg(bio);\n \tstruct throtl_qnode qn = NULL;\n \tstruct throtl_grp tg = blkg_to_tg(blkg);\n \tstruct throtl_service_queue sq;\ndiff --git a/block/blk-throttle.h b/block/blk-throttle.h\nindex 1b5775771e078..45f30179fd9c0 100644\n--- a/block/blk-throttle.h\n+++ b/block/blk-throttle.h\n@@ -171,7 +171,7 @@ static inline bool blk_should_throtl(struct bio bio)\n \tif (!blk_throtl_activated(bio-\u003ebi_bdev-\u003ebd_queue))\n \t\treturn false;\n \n-\ttg = blkg_to_tg(bio-\u003ebi_blkg);\n+\ttg = blkg_to_tg(bio_blkg(bio));\n \tif (!cgroup_subsys_on_dfl(io_cgrp_subsys)) {\n \t\tif (!bio_flagged(bio, BIO_CGROUP_ACCT)) {\n \t\t\tbio_set_flag(bio, BIO_CGROUP_ACCT);\ndiff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c\nindex 3fa3b13a410f4..c0f945b8d9413 100644\n--- a/drivers/md/bcache/request.c\n+++ b/drivers/md/bcache/request.c\n@@ -848,7 +848,7 @@ static CLOSURE_CALLBACK(cached_dev_read_done)\n \t\ts-\u003eiop.bio-\u003ebi_iter.bi_sector =\n \t\t\ts-\u003ecache_miss-\u003ebi_iter.bi_sector;\n \t\ts-\u003eiop.bio-\u003ebi_iter.bi_size = s-\u003einsert_bio_sectors \u003c\u003c 9;\n-\t\tbio_clone_blkg_association(s-\u003eiop.bio, s-\u003ecache_miss);\n+\t\tbio_clone_blkcg_association(s-\u003eiop.bio, s-\u003ecache_miss);\n \t\tbch_bio_map(s-\u003eiop.bio, NULL);\n \n \t\tbio_copy_data(s-\u003ecache_miss, s-\u003eiop.bio);\ndiff --git a/drivers/md/dm.c b/drivers/md/dm.c\nindex d413bfaf35278..cd68eec77f5a1 100644\n--- a/drivers/md/dm.c\n+++ b/drivers/md/dm.c\n@@ -1373,7 +1373,7 @@ void dm_submit_bio_remap(struct bio clone, struct bio tgt_clone)\n \tif (!tgt_clone)\n \t\ttgt_clone = clone;\n \n-\tbio_clone_blkg_association(tgt_clone, io-\u003eorig_bio);\n+\tbio_clone_blkcg_association(tgt_clone, io-\u003eorig_bio);\n \n \t/\n \t * Account io-\u003eorigin_bio to DM dev on behalf of target\ndiff --git a/drivers/md/md.c b/drivers/md/md.c\nindex 680b34a63cb3d..fd78d6cc8da17 100644\n--- a/drivers/md/md.c\n+++ b/drivers/md/md.c\n@@ -9386,7 +9386,7 @@ void md_submit_discard_bio(struct mddev mddev, struct md_rdev rdev,\n \t\treturn;\n \n \tbio_chain(discard_bio, bio);\n-\tbio_clone_blkg_association(discard_bio, bio);\n+\tbio_clone_blkcg_association(discard_bio, bio);\n \tmddev_trace_remap(mddev, discard_bio, bio-\u003ebi_iter.bi_sector);\n \tsubmit_bio_noacct(discard_bio);\n }\ndiff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c\nindex 4176046627beb..54e4adb0ccb7d 100644\n--- a/drivers/nvdimm/nd_virtio.c\n+++ b/drivers/nvdimm/nd_virtio.c\n@@ -121,7 +121,7 @@ int async_pmem_flush(struct nd_region nd_region, struct bio bio)\n \n \t\tif (!child)\n \t\t\treturn -ENOMEM;\n-\t\tbio_clone_blkg_association(child, bio);\n+\t\tbio_clone_blkcg_association(child, bio);\n \t\tchild-\u003ebi_iter.bi_sector = -1;\n \t\tbio_chain(child, bio);\n \t\tsubmit_bio(child);\ndiff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c\nindex 6dabe73ad790d..6512dbd9516fc 100644\n--- a/fs/gfs2/lops.c\n+++ b/fs/gfs2/lops.c\n@@ -484,7 +484,7 @@ static struct bio gfs2_chain_bio(struct bio prev, unsigned int nr_iovecs,\n \tstruct bio new;\n \n \tnew = bio_alloc(prev-\u003ebi_bdev, nr_iovecs, opf, GFP_NOIO);\n-\tbio_clone_blkg_association(new, prev);\n+\tbio_clone_blkcg_association(new, prev);\n \tnew-\u003ebi_iter.bi_sector = sector;\n \tbio_chain(new, prev);\n \tsubmit_bio(prev);\n@@ -1114,4 +1114,3 @@ const struct gfs2_log_operations gfs2_log_ops[] = {\n \t\u0026gfs2_revoke_lops,\n \tNULL,\n };\n-\ndiff --git a/include/linux/bio.h b/include/linux/bio.h\nindex bb3235497e673..11637f85c38d4 100644\n--- a/include/linux/bio.h\n+++ b/include/linux/bio.h\n@@ -542,18 +542,20 @@ static inline void bio_release_pages(struct bio bio, bool mark_dirty)\n \tdisk_devt((bio)-\u003ebi_bdev-\u003ebd_disk)\n \n #ifdef CONFIG_BLK_CGROUP\n-void bio_associate_blkg(struct bio bio);\n-void bio_associate_blkg_from_css(struct bio bio,\n-\t\t\t\t struct cgroup_subsys_state css);\n-void bio_clone_blkg_association(struct bio dst, struct bio src);\n+void bio_associate_blkcg(struct bio bio);\n+void bio_associate_blkcg_from_css(struct bio bio,\n+\t\t\t\t struct cgroup_subsys_state css);\n+void bio_clone_blkcg_association(struct bio dst, struct bio src);\n+void bio_put_blkg_ref(struct bio bio);\n void blkcg_punt_bio_submit(struct bio bio);\n #else\t/ CONFIG_BLK_CGROUP /\n-static inline void bio_associate_blkg(struct bio bio) { }\n-static inline void bio_associate_blkg_from_css(struct bio bio,\n-\t\t\t\t\t struct cgroup_subsys_state css)\n+static inline void bio_associate_blkcg(struct bio bio) { }\n+static inline void bio_associate_blkcg_from_css(struct bio bio,\n+\t\t\t\t\t\tstruct cgroup_subsys_state css)\n { }\n-static inline void bio_clone_blkg_association(struct bio dst,\n-\t\t\t\t\t struct bio src) { }\n+static inline void bio_clone_blkcg_association(struct bio dst,\n+\t\t\t\t\t struct bio src) { }\n+static inline void bio_put_blkg_ref(struct bio bio) { }\n static inline void blkcg_punt_bio_submit(struct bio bio)\n {\n \tsubmit_bio(bio);\n@@ -563,10 +565,12 @@ static inline void blkcg_punt_bio_submit(struct bio bio)\n static inline void bio_set_dev(struct bio bio, struct block_device bdev)\n {\n \tbio_clear_flag(bio, BIO_REMAPPED);\n-\tif (bio-\u003ebi_bdev != bdev)\n+\tif (bio-\u003ebi_bdev != bdev) {\n+\t\tbio_put_blkg_ref(bio);\n \t\tbio_clear_flag(bio, BIO_BPS_THROTTLED);\n+\t}\n \tbio-\u003ebi_bdev = bdev;\n-\tbio_associate_blkg(bio);\n+\tbio_associate_blkcg(bio);\n }\n \n /\ndiff --git a/include/linux/blk_types.h b/include/linux/blk_types.h\nindex d49d97a050d0e..53ba307b91cb3 100644\n--- a/include/linux/blk_types.h\n+++ b/include/linux/blk_types.h\n@@ -246,12 +246,10 @@ struct bio {\n \tvoid\t\t\tbi_private;\n #ifdef CONFIG_BLK_CGROUP\n \t/\n-\t Represents the association of the css and request_queue for the bio.\n-\t * If a bio goes direct to device, it will not have a blkg as it will\n-\t * not have a request_queue associated with it. The reference is put\n-\t * on release of the bio.\n+\t * Represents the blkcg css association for the bio. The reference is\n+\t * put on release of the bio.\n \t /\n-\tstruct blkcg_gq\t\tbi_blkg;\n+\tstruct blkcg\t\tbi_blkcg;\n \t/ Time that this bio was issued. /\n \tu64\t\t\tissue_time_ns;\n #ifdef CONFIG_BLK_CGROUP_IOCOST\n@@ -309,6 +307,7 @@ enum {\n \tBIO_TRACE_COMPLETION,\t/ bio_endio() should trace the final completion\n \t\t\t\t * of this bio. /\n \tBIO_CGROUP_ACCT,\t/ has been accounted to a cgroup /\n+\tBIO_BLKG_REF,\t\t/ bio pins the associated blkg /\n \tBIO_QOS_THROTTLED,\t/ bio went through rq_qos throttle path /\n \t/\n \t * This bio has completed bps throttling at the single tg granularity,\ndiff --git a/include/linux/blkdev.h b/include/linux/blkdev.h\nindex 20cb8ed7d987a..f3b0e39d3f6b4 100644\n--- a/include/linux/blkdev.h\n+++ b/include/linux/blkdev.h\n@@ -27,6 +27,7 @@\n #include \u003clinux/xarray.h\u003e\n #include \u003clinux/file.h\u003e\n #include \u003clinux/lockdep.h\u003e\n+#include \u003clinux/rhashtable-types.h\u003e\n \n struct module;\n struct request_queue;\n@@ -581,6 +582,7 @@ struct request_queue {\n #ifdef CONFIG_BLK_CGROUP\n \tDECLARE_BITMAP\t\t(blkcg_pols, BLKCG_MAX_POLS);\n \tstruct blkcg_gq\t\troot_blkg;\n+\tstruct rhashtable\tblkg_hash;\n \tstruct list_head\tblkg_list;\n \tstruct mutex\t\tblkcg_mutex;\n #endif\ndiff --git a/include/linux/writeback.h b/include/linux/writeback.h\nindex 62552a2ce5b9e..4f869fe9cc90f 100644\n--- a/include/linux/writeback.h\n+++ b/include/linux/writeback.h\n@@ -262,7 +262,7 @@ static inline void wbc_init_bio(struct writeback_control wbc, struct bio bio)\n \t regular writeback instead of writing things out itself.\n \t /\n \tif (wbc-\u003ewb)\n-\t\tbio_associate_blkg_from_css(bio, wbc-\u003ewb-\u003eblkcg_css);\n+\t\tbio_associate_blkcg_from_css(bio, wbc-\u003ewb-\u003eblkcg_css);\n }\n \n void inode_switch_wbs_work_fn(struct work_struct work);\ndiff --git a/mm/page_io.c b/mm/page_io.c\nindex b23f494fcc83d..a4a1b8a8a0e38 100644\n--- a/mm/page_io.c\n+++ b/mm/page_io.c\n@@ -316,7 +316,7 @@ static inline void count_swpout_vm_event(struct folio folio)\n }\n \n #if defined(CONFIG_MEMCG) \u0026\u0026 defined(CONFIG_BLK_CGROUP)\n-static void bio_associate_blkg_from_page(struct bio bio, struct folio folio)\n+static void bio_associate_blkcg_from_page(struct bio bio, struct folio folio)\n {\n \tstruct cgroup_subsys_state css;\n \tstruct mem_cgroup memcg;\n@@ -331,12 +331,15 @@ static void bio_associate_blkg_from_page(struct bio bio, struct folio folio)\n \t\tcss = NULL;\n \trcu_read_unlock();\n \n-\tbio_associate_blkg_from_css(bio, css);\n+\tbio_associate_blkcg_from_css(bio, css);\n \tif (css)\n \t\tcss_put(css);\n }\n #else\n-#define bio_associate_blkg_from_page(bio, folio)\t\tdo { } while (0)\n+static inline void bio_associate_blkcg_from_page(struct bio bio,\n+\t\t\t\t\t\t struct folio folio)\n+{\n+}\n #endif / CONFIG_MEMCG \u0026\u0026 CONFIG_BLK_CGROUP /\n \n struct swap_iocb {\n@@ -436,7 +439,7 @@ static void swap_writepage_bdev_sync(struct folio folio,\n \tbio.bi_iter.bi_sector = swap_folio_sector(folio);\n \tbio_add_folio_nofail(\u0026bio, folio, folio_size(folio), 0);\n \n-\tbio_associate_blkg_from_page(\u0026bio, folio);\n+\tbio_associate_blkcg_from_page(\u0026bio, folio);\n \tcount_swpout_vm_event(folio);\n \n \tfolio_start_writeback(folio);\n@@ -456,7 +459,7 @@ static void swap_writepage_bdev_async(struct folio *folio,\n \tbio-\u003ebi_end_io = end_swap_bio_write;\n \tbio_add_folio_nofail(bio, folio, folio_size(folio), 0);\n \n-\tbio_associate_blkg_from_page(bio, folio);\n+\tbio_associate_blkcg_from_page(bio, folio);\n \tcount_swpout_vm_event(folio);\n \tfolio_start_writeback(folio);\n \tfolio_unlock(folio);\n" }
2/1	2026/08/18 08:04	agent	patch-evaluator	4m Model: best-expensive Args: null Results: { "EnableConfigs": [ "BLK_CGROUP", "IOSCHED_BFQ", "BFQ_GROUP_IOSCHED", "BLK_CGROUP_IOCOST", "BLK_CGROUP_IOLATENCY", "BLK_CGROUP_IOPRIO", "NVME_FC", "BLK_CGROUP_FC_APPID", "BTRFS_FS" ], "FocusSymbols": [ "blkg_conf_prep", "blkcg_punt_bio_submit", "blkg_init_queue", "blkg_exit_queue", "blkcg_init_disk", "blkcg_exit_disk" ], "Reasoning": "The patch significantly refactors the block cgroup (blkcg) subsystem by changing how `bio`s are associated with block cgroups. Instead of directly storing a queue-specific `blkcg_gq` (blkg) in `bio-\u003ebi_blkg`, it now stores the generic `blkcg` in `bio-\u003ebi_blkcg` and dynamically looks up or pins the queue-specific `blkg` using a new `BIO_BLKG_REF` flag. It also replaces the `radix_tree` used for `blkg` lookups with an `rhashtable`. These are core functional changes to the block layer and cgroup IO controllers, affecting IO scheduling, throttling, and accounting.", "WorthFuzzing": true } Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit abc0b1a65e87d979eec053ca54836b90fdec9729 Author: syz-cluster <triage@syzkaller.com> Date: Tue Aug 18 08:04:15 2026 +0000 syz-cluster: applied patch under review diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 14b8c571c0d14..bbd79931d5ab5 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -3235,7 +3235,7 @@ the configuration, the bio may be executed at a lower priority and if the writeback session is holding shared resources, e.g. a journal entry, may lead to priority inversion. There is no one easy solution for the problem. Filesystems can try to work around specific problem -cases by skipping wbc_init_bio() and using bio_associate_blkg() +cases by skipping wbc_init_bio() and using bio_associate_blkcg() directly. diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index ef9d63e8542ed..4a3975f9ff744 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -363,11 +363,13 @@ void bfqg_and_blkg_put(struct bfq_group bfqg) void bfqg_stats_update_legacy_io(struct request_queue q, struct request rq) { - struct bfq_group bfqg = blkg_to_bfqg(rq->bio->bi_blkg); + struct blkcg_gq blkg = bio_blkg_lookup(rq->bio); + struct bfq_group bfqg; - if (!bfqg) + if (!blkg) return; + bfqg = blkg_to_bfqg(blkg); blkg_rwstat_add(&bfqg->stats.bytes, rq->cmd_flags, blk_rq_bytes(rq)); blkg_rwstat_add(&bfqg->stats.ios, rq->cmd_flags, 1); } @@ -606,7 +608,7 @@ static void bfq_link_bfqg(struct bfq_data bfqd, struct bfq_group bfqg) struct bfq_group bfq_bio_bfqg(struct bfq_data bfqd, struct bio bio) { - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg_gq blkg = bio_blkg_lookup(bio); struct bfq_group bfqg; while (blkg) { @@ -614,15 +616,17 @@ struct bfq_group bfq_bio_bfqg(struct bfq_data bfqd, struct bio bio) blkg = blkg->parent; continue; } + bfqg = blkg_to_bfqg(blkg); if (bfqg->pd.online) { - bio_associate_blkg_from_css(bio, &blkg->blkcg->css); + bio_associate_blkcg_from_css(bio, &blkg->blkcg->css); return bfqg; } blkg = blkg->parent; } - bio_associate_blkg_from_css(bio, - &bfqg_to_blkg(bfqd->root_group)->blkcg->css); + + blkg = bfqg_to_blkg(bfqd->root_group); + bio_associate_blkcg_from_css(bio, &blkg->blkcg->css); return bfqd->root_group; } diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 0f75301b3115f..d4f657bc7a4e5 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -128,6 +128,7 @@ #include "elevator.h" #include "blk.h" +#include "blk-cgroup.h" #include "blk-mq.h" #include "blk-mq-sched.h" #include "bfq-iosched.h" @@ -2452,15 +2453,16 @@ static bool bfq_bio_merge(struct request_queue q, struct bio bio, struct request free = NULL; bool ret; +#ifdef CONFIG_BFQ_GROUP_IOSCHED + /* blkg creation takes q->queue_lock, so do it before bfqd->lock. / + if (bic) + bio_blkg(bio); +#endif + spin_lock_irq(&bfqd->lock); if (bic) { - / - * Make sure cgroup info is uptodate for current process before - * considering the merge. - / bfq_bic_update_cgroup(bic, bio); - bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf), bfq_actuator_index(bfqd, bio)); } else { @@ -6245,6 +6247,13 @@ static void bfq_insert_request(struct blk_mq_hw_ctx hctx, struct request rq, LIST_HEAD(free); #ifdef CONFIG_BFQ_GROUP_IOSCHED + / + * Pin the blkg used to look up bfqg. If this is the first IO for + * the blkcg on this queue, create the bfqg before holding bfqd->lock. + / + if (rq->bio && !bio_flagged(rq->bio, BIO_BLKG_REF)) + bio_blkg(rq->bio); + if (!cgroup_subsys_on_dfl(io_cgrp_subsys) && rq->bio) bfqg_stats_update_legacy_io(q, rq); #endif diff --git a/block/bio.c b/block/bio.c index 898b2f5ef8c81..6c84042adf5dd 100644 --- a/block/bio.c +++ b/block/bio.c @@ -179,12 +179,7 @@ static inline gfp_t try_alloc_gfp(gfp_t gfp) void bio_uninit(struct bio bio) { -#ifdef CONFIG_BLK_CGROUP - if (bio->bi_blkg) { - blkg_put(bio->bi_blkg); - bio->bi_blkg = NULL; - } -#endif + bio_clear_blkcg(bio); if (bio_integrity(bio)) bio_integrity_free(bio); @@ -233,10 +228,10 @@ void bio_init(struct bio bio, struct block_device bdev, struct bio_vec table, bio->bi_end_io = NULL; bio->bi_private = NULL; #ifdef CONFIG_BLK_CGROUP - bio->bi_blkg = NULL; + bio->bi_blkcg = NULL; bio->issue_time_ns = 0; if (bdev) - bio_associate_blkg(bio); + bio_associate_blkcg(bio); #ifdef CONFIG_BLK_CGROUP_IOCOST bio->bi_iocost_cost = 0; #endif @@ -281,7 +276,7 @@ void bio_reset(struct bio bio, struct block_device bdev, blk_opf_t opf) bio->bi_io_vec = bv; bio->bi_bdev = bdev; if (bio->bi_bdev) - bio_associate_blkg(bio); + bio_associate_blkcg(bio); bio->bi_opf = opf; } EXPORT_SYMBOL(bio_reset); @@ -866,7 +861,7 @@ static int __bio_clone(struct bio bio, struct bio bio_src, gfp_t gfp) if (bio->bi_bdev == bio_src->bi_bdev && bio_flagged(bio_src, BIO_REMAPPED)) bio_set_flag(bio, BIO_REMAPPED); - bio_clone_blkg_association(bio, bio_src); + bio_clone_blkcg_association(bio, bio_src); } if (bio_crypt_clone(bio, bio_src, gfp) < 0) @@ -1910,17 +1905,12 @@ void bio_endio(struct bio bio) goto again; } -#ifdef CONFIG_BLK_CGROUP /* * Release cgroup info. We shouldn't have to do this here, but quite * a few callers of bio_init fail to call bio_uninit, so we cover up * for that here at least for now. / - if (bio->bi_blkg) { - blkg_put(bio->bi_blkg); - bio->bi_blkg = NULL; - } -#endif + bio_clear_blkcg(bio); if (bio_flagged(bio, BIO_COMPLETE_IN_TASK) && bio_in_atomic()) __bio_complete_in_task(bio); diff --git a/block/blk-cgroup-fc-appid.c b/block/blk-cgroup-fc-appid.c index 3ec21333f393b..7589c62099898 100644 --- a/block/blk-cgroup-fc-appid.c +++ b/block/blk-cgroup-fc-appid.c @@ -50,8 +50,14 @@ EXPORT_SYMBOL_GPL(blkcg_set_fc_appid); / char blkcg_get_fc_appid(struct bio bio) { - if (!bio->bi_blkg \|\| bio->bi_blkg->blkcg->fc_app_id[0] == '\0') + struct blkcg blkcg = bio_blkcg(bio); + + if (!blkcg) + return NULL; + + if (blkcg->fc_app_id[0] == '\0') return NULL; - return bio->bi_blkg->blkcg->fc_app_id; + + return blkcg->fc_app_id; } EXPORT_SYMBOL_GPL(blkcg_get_fc_appid); diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 2b5c29434e426..2c758c26bce60 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -66,6 +66,13 @@ static DEFINE_RAW_SPINLOCK(blkg_stat_lock); #define BLKG_DESTROY_BATCH_SIZE 64 +const struct rhashtable_params blkg_hash_params = { + .key_len = sizeof_field(struct blkcg_gq, blkcg_id), + .key_offset = offsetof(struct blkcg_gq, blkcg_id), + .head_offset = offsetof(struct blkcg_gq, q_hash_node), + .automatic_shrinking = true, +}; + / * Lockless lists for tracking IO stats update * @@ -173,10 +180,6 @@ static void __blkg_release(struct rcu_head rcu) { struct blkcg_gq blkg = container_of(rcu, struct blkcg_gq, rcu_head); -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO - WARN_ON(!bio_list_empty(&blkg->async_bios)); -#endif - blkg_free(blkg); } @@ -194,6 +197,16 @@ static void blkg_release(struct percpu_ref ref) struct blkcg blkcg = blkg->blkcg; int cpu; + /* + * A blkg that was never inserted into q->blkg_list has no hash + * entry. This happens when allocation or creation fails before + * rhashtable_insert_fast() succeeds. + / + if (!list_empty(&blkg->q_node)) + WARN_ON_ONCE(rhashtable_remove_fast(&blkg->q->blkg_hash, + &blkg->q_hash_node, + blkg_hash_params)); + / * Flush all the non-empty percpu lockless lists before releasing * us, given these stat belongs to us. @@ -209,19 +222,18 @@ static void blkg_release(struct percpu_ref ref) #ifdef CONFIG_BLK_CGROUP_PUNT_BIO static struct workqueue_struct blkcg_punt_bio_wq; -static void blkg_async_bio_workfn(struct work_struct work) +static void blkcg_async_bio_workfn(struct work_struct work) { - struct blkcg_gq blkg = container_of(work, struct blkcg_gq, - async_bio_work); + struct blkcg blkcg = container_of(work, struct blkcg, async_bio_work); struct bio_list bios = BIO_EMPTY_LIST; struct bio bio; struct blk_plug plug; bool need_plug = false; - / as long as there are pending bios, @blkg can't go away / - spin_lock(&blkg->async_bio_lock); - bio_list_merge_init(&bios, &blkg->async_bios); - spin_unlock(&blkg->async_bio_lock); + / as long as there are pending bios, @blkcg can't go away / + spin_lock(&blkcg->async_bio_lock); + bio_list_merge_init(&bios, &blkcg->async_bios); + spin_unlock(&blkcg->async_bio_lock); / start plug only when bio_list contains at least 2 bios / if (bios.head && bios.head->bi_next) { @@ -242,15 +254,15 @@ static void blkg_async_bio_workfn(struct work_struct work) / void blkcg_punt_bio_submit(struct bio bio) { - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg blkcg = bio_blkcg(bio); - if (blkg->parent) { - spin_lock(&blkg->async_bio_lock); - bio_list_add(&blkg->async_bios, bio); - spin_unlock(&blkg->async_bio_lock); - queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work); + if (blkcg && cgroup_parent(blkcg->css.cgroup)) { + spin_lock(&blkcg->async_bio_lock); + bio_list_add(&blkcg->async_bios, bio); + spin_unlock(&blkcg->async_bio_lock); + queue_work(blkcg_punt_bio_wq, &blkcg->async_bio_work); } else { - /* never bounce for the root cgroup / + / Never bounce if there is no non-root blkcg to queue on. / submit_bio(bio); } } @@ -278,9 +290,13 @@ subsys_initcall(blkcg_punt_bio_init); / struct cgroup_subsys_state bio_blkcg_css(struct bio bio) { - if (!bio \|\| !bio->bi_blkg) + struct blkcg blkcg; + + if (!bio) return NULL; - return &bio->bi_blkg->blkcg->css; + + blkcg = bio_blkcg(bio); + return blkcg ? &blkcg->css : NULL; } EXPORT_SYMBOL_GPL(bio_blkcg_css); @@ -327,12 +343,8 @@ static struct blkcg_gq blkg_alloc(struct blkcg blkcg, struct gendisk disk, blkg->q = disk->queue; INIT_LIST_HEAD(&blkg->q_node); blkg->blkcg = blkcg; + blkg->blkcg_id = blkcg->css.id; blkg->iostat.blkg = blkg; -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO - spin_lock_init(&blkg->async_bio_lock); - bio_list_init(&blkg->async_bios); - INIT_WORK(&blkg->async_bio_work, blkg_async_bio_workfn); -#endif u64_stats_init(&blkg->iostat.sync); for_each_possible_cpu(cpu) { @@ -423,7 +435,8 @@ static struct blkcg_gq blkg_create(struct blkcg blkcg, struct gendisk disk, / insert / spin_lock(&blkcg->lock); - ret = radix_tree_insert(&blkcg->blkg_tree, disk->queue->id, blkg); + ret = rhashtable_insert_fast(&disk->queue->blkg_hash, + &blkg->q_hash_node, blkg_hash_params); if (likely(!ret)) { hlist_add_head_rcu(&blkg->blkcg_node, &blkcg->blkg_list); list_add(&blkg->q_node, &disk->queue->blkg_list); @@ -454,6 +467,17 @@ static struct blkcg_gq blkg_create(struct blkcg blkcg, struct gendisk disk, return ERR_PTR(ret); } +/* + * The root blkg holds a live reference while the disk is active, so walking + * the parent chain always finds a blkg which can be pinned. + / +static struct blkcg_gq blkg_lookup_tryget(struct blkcg_gq blkg) +{ + while (!blkg_tryget(blkg)) + blkg = blkg->parent; + return blkg; +} + /* * blkg_lookup_create - lookup blkg, try to create one if not there * @blkcg: blkcg of interest @@ -461,11 +485,13 @@ static struct blkcg_gq blkg_create(struct blkcg blkcg, struct gendisk disk, * Lookup blkg for the @blkcg - @disk pair. If it doesn't exist, try to * create one. blkg creation is performed recursively from blkcg_root such - * that all non-root blkg's have access to the parent blkg. This function - * should be called under RCU read lock and takes @disk->queue->queue_lock. + * that all non-root blkg's have access to the parent blkg. * - * Returns the blkg or the closest blkg if blkg_create() fails as it walks - * down from root. + * Must be called with @disk->queue->queue_lock held. + * + * Returns the closest blkg with an extra reference acquired. If + * blkg_create() fails while walking down from root, the returned blkg may + * belong to an ancestor of @blkcg. This function never returns %NULL. / static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg, struct gendisk disk) @@ -473,12 +499,12 @@ static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg, struct request_queue q = disk->queue; struct blkcg_gq blkg; + lockdep_assert_held(&q->queue_lock); + rcu_read_lock(); blkg = blkg_lookup(blkcg, q); if (blkg) { - if (blkcg != &blkcg_root && - blkg != rcu_dereference(blkcg->blkg_hint)) - rcu_assign_pointer(blkcg->blkg_hint, blkg); + blkg = blkg_lookup_tryget(blkg); rcu_read_unlock(); return blkg; } @@ -514,7 +540,7 @@ static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg, break; } - return blkg; + return blkg_lookup_tryget(blkg); } static void blkg_destroy(struct blkcg_gq blkg) @@ -546,17 +572,8 @@ static void blkg_destroy(struct blkcg_gq blkg) blkg->online = false; - radix_tree_delete(&blkcg->blkg_tree, blkg->q->id); hlist_del_init_rcu(&blkg->blkcg_node); - /* - * Both setting lookup hint to and clearing it from @blkg are done - * under queue_lock. If it's not pointing to @blkg now, it never - * will. Hint assignment itself can race safely. - / - if (rcu_access_pointer(blkcg->blkg_hint) == blkg) - rcu_assign_pointer(blkcg->blkg_hint, NULL); - / * Put the reference taken at the time of creation so that when all * queues are gone, group can be destroyed. @@ -882,18 +899,12 @@ int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol, goto fail_exit; } - if (radix_tree_preload(GFP_KERNEL)) { - blkg_free(new_blkg); - ret = -ENOMEM; - goto fail_exit; - } - spin_lock_irq(&q->queue_lock); if (!blkcg_policy_enabled(q, pol)) { blkg_free(new_blkg); ret = -EOPNOTSUPP; - goto fail_preloaded; + goto fail_unlock; } blkg = blkg_lookup(pos, q); @@ -903,12 +914,10 @@ int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol, blkg = blkg_create(pos, disk, new_blkg); if (IS_ERR(blkg)) { ret = PTR_ERR(blkg); - goto fail_preloaded; + goto fail_unlock; } } - radix_tree_preload_end(); - if (pos == blkcg) goto success; } @@ -917,8 +926,6 @@ int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol, ctx->blkg = blkg; return 0; -fail_preloaded: - radix_tree_preload_end(); fail_unlock: spin_unlock_irq(&q->queue_lock); fail_exit: @@ -1373,6 +1380,9 @@ static void blkcg_css_free(struct cgroup_subsys_state css) mutex_unlock(&blkcg_pol_mutex); +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO + WARN_ON(!bio_list_empty(&blkcg->async_bios)); +#endif free_percpu(blkcg->lhead); kfree(blkcg); } @@ -1420,8 +1430,12 @@ blkcg_css_alloc(struct cgroup_subsys_state parent_css) spin_lock_init(&blkcg->lock); refcount_set(&blkcg->online_pin, 1); - INIT_RADIX_TREE(&blkcg->blkg_tree, GFP_NOWAIT); INIT_HLIST_HEAD(&blkcg->blkg_list); +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO + spin_lock_init(&blkcg->async_bio_lock); + bio_list_init(&blkcg->async_bios); + INIT_WORK(&blkcg->async_bio_work, blkcg_async_bio_workfn); +#endif #ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD(&blkcg->cgwb_list); #endif @@ -1457,17 +1471,22 @@ static int blkcg_css_online(struct cgroup_subsys_state css) return 0; } -void blkg_init_queue(struct request_queue q) +int blkg_init_queue(struct request_queue q) { INIT_LIST_HEAD(&q->blkg_list); mutex_init(&q->blkcg_mutex); + return rhashtable_init(&q->blkg_hash, &blkg_hash_params); +} + +void blkg_exit_queue(struct request_queue q) +{ + rhashtable_destroy(&q->blkg_hash); } int blkcg_init_disk(struct gendisk disk) { struct request_queue q = disk->queue; struct blkcg_gq new_blkg, blkg; - bool preloaded; /* * If the queue is shared across disk rebind (e.g., SCSI), the @@ -1485,8 +1504,6 @@ int blkcg_init_disk(struct gendisk disk) if (!new_blkg) return -ENOMEM; - preloaded = !radix_tree_preload(GFP_KERNEL); - / Make sure the root blkg exists. / / spin_lock_irq can serve as RCU read-side critical section. / spin_lock_irq(&q->queue_lock); @@ -1496,15 +1513,10 @@ int blkcg_init_disk(struct gendisk disk) q->root_blkg = blkg; spin_unlock_irq(&q->queue_lock); - if (preloaded) - radix_tree_preload_end(); - return 0; err_unlock: spin_unlock_irq(&q->queue_lock); - if (preloaded) - radix_tree_preload_end(); return PTR_ERR(blkg); } @@ -2064,129 +2076,165 @@ void blkcg_add_delay(struct blkcg_gq blkg, u64 now, u64 delta) atomic64_add(delta, &blkg->delay_nsec); } -static inline struct blkcg_gq blkg_lookup_tryget(struct blkcg_gq blkg) +/ + * Return the blkg pinned by @bio through BIO_BLKG_REF. The returned blkg is + * already owned by @bio and no extra reference is acquired. + / +static struct blkcg_gq bio_pinned_blkg(struct bio bio) { -retry: - if (blkg_tryget(blkg)) - return blkg; + struct request_queue q = bdev_get_queue(bio->bi_bdev); + struct blkcg_gq blkg; - blkg = blkg->parent; - if (blkg) - goto retry; + rcu_read_lock(); + blkg = blkg_lookup_any(bio_blkcg(bio), q); + rcu_read_unlock(); - return NULL; + WARN_ON_ONCE(!blkg); + return blkg; +} + +static void bio_set_blkg_ref(struct bio bio, struct blkcg_gq blkg) +{ + if (bio_blkcg(bio) != blkg->blkcg) { + css_get(&blkg->blkcg->css); + bio_clear_blkcg(bio); + bio->bi_blkcg = blkg->blkcg; + } + bio_set_flag(bio, BIO_BLKG_REF); } + /* - * blkg_tryget_closest - try and get a blkg ref on the closet blkg + * bio_blkg_lookup - look up a blkg associated with a bio * @bio: target bio - * @css: target css * - * As the failure mode here is to walk up the blkg tree, this ensure that the - * blkg->parent pointers are always valid. This returns the blkg that it ended - * up taking a reference on or %NULL if no reference was taken. + * Return the blkg already pinned by @bio, or %NULL if @bio doesn't own a blkg + * reference. Call bio_blkg() instead when a missing blkg should be created. / -static inline struct blkcg_gq blkg_tryget_closest(struct bio bio, - struct cgroup_subsys_state css) +struct blkcg_gq bio_blkg_lookup(struct bio bio) { - struct request_queue q = bio->bi_bdev->bd_queue; - struct blkcg blkcg = css_to_blkcg(css); + if (!bio_flagged(bio, BIO_BLKG_REF)) + return NULL; + return bio_pinned_blkg(bio); +} +EXPORT_SYMBOL_GPL(bio_blkg_lookup); + +/** + * bio_put_blkg_ref - drop the blkg reference pinned by a bio + * @bio: target bio + * + * Drop the bio-owned blkg reference acquired by bio_blkg(), if any. + / +void bio_put_blkg_ref(struct bio bio) +{ + if (bio_flagged(bio, BIO_BLKG_REF)) { + struct blkcg_gq blkg = bio_pinned_blkg(bio); + + if (blkg) + blkg_put(blkg); + bio_clear_flag(bio, BIO_BLKG_REF); + } +} +EXPORT_SYMBOL_GPL(bio_put_blkg_ref); + +/* + * bio_blkg - look up the blkg associated with a bio + * @bio: target bio + * + * Look up the queue-local blkg for @bio's current device and blkcg. If this + * is the first policy use of @bio, create the missing blkg hierarchy if + * necessary, pin the closest available blkg, and update @bio's blkcg + * association if allocation falls back to an ancestor. + / +struct blkcg_gq bio_blkg(struct bio bio) +{ + struct blkcg blkcg = bio_blkcg(bio); + struct gendisk disk; + struct request_queue q; struct blkcg_gq blkg; - rcu_read_lock(); - blkg = blkg_lookup(blkcg, q); - if (likely(blkg)) - blkg = blkg_lookup_tryget(blkg); - rcu_read_unlock(); + if (!blkcg \|\| !bio->bi_bdev) + return NULL; - if (blkg) - return blkg; + if (bio_flagged(bio, BIO_BLKG_REF)) + return bio_pinned_blkg(bio); + + disk = bio->bi_bdev->bd_disk; + q = disk->queue; - / - * Fast path failed, we're probably issuing IO in this cgroup the first - * time, hold lock to create new blkg. - / spin_lock_irq(&q->queue_lock); - blkg = blkg_lookup_create(blkcg, bio->bi_bdev->bd_disk); - if (blkg) - blkg = blkg_lookup_tryget(blkg); + blkg = blkg_lookup_create(blkcg, disk); spin_unlock_irq(&q->queue_lock); + bio_set_blkg_ref(bio, blkg); return blkg; } +EXPORT_SYMBOL_GPL(bio_blkg); /* - * bio_associate_blkg_from_css - associate a bio with a specified css + * bio_associate_blkcg_from_css - associate a bio with a specified css * @bio: target bio * @css: target css * - * Associate @bio with the blkg found by combining the css's blkg and the - * request_queue of the @bio. An association failure is handled by walking up - * the blkg tree. Therefore, the blkg associated can be anything between @blkg - * and q->root_blkg. This situation only happens when a cgroup is dying and - * then the remaining bios will spill to the closest alive blkg. + * Associate @bio with the blkcg found from @css. The queue-local blkg is + * created and pinned by bio_blkg() when blkcg policies need it. If @css + * can't be referenced online, associate @bio with the root blkcg instead. * - * A reference will be taken on the blkg and will be released when @bio is + * A reference will be taken on the blkcg and will be released when @bio is * freed. / -void bio_associate_blkg_from_css(struct bio bio, - struct cgroup_subsys_state css) +void bio_associate_blkcg_from_css(struct bio bio, + struct cgroup_subsys_state css) { - if (bio->bi_blkg) - blkg_put(bio->bi_blkg); + struct blkcg blkcg; - if (css && css->parent) { - bio->bi_blkg = blkg_tryget_closest(bio, css); - } else { - blkg_get(bdev_get_queue(bio->bi_bdev)->root_blkg); - bio->bi_blkg = bdev_get_queue(bio->bi_bdev)->root_blkg; + if (!css \|\| !css_tryget_online(css)) { + css = &blkcg_root.css; + css_get(css); } + + blkcg = css_to_blkcg(css); + if (bio_blkcg(bio) == blkcg) { + css_put(css); + return; + } + + bio_clear_blkcg(bio); + bio->bi_blkcg = blkcg; } -EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css); +EXPORT_SYMBOL_GPL(bio_associate_blkcg_from_css); /** - * bio_associate_blkg - associate a bio with a blkg + * bio_associate_blkcg - associate a bio with a blkcg * @bio: target bio * - * Associate @bio with the blkg found from the bio's css and request_queue. - * If one is not found, bio_lookup_blkg() creates the blkg. If a blkg is - * already associated, the css is reused and association redone as the - * request_queue may have changed. + * Associate @bio with the blkcg found from the bio's css. If a blkcg is + * already associated, keep it as blkcg association is not queue-local. / -void bio_associate_blkg(struct bio bio) +void bio_associate_blkcg(struct bio bio) { - struct cgroup_subsys_state css; - if (blk_op_is_passthrough(bio->bi_opf)) return; - if (bio->bi_blkg) { - css = bio_blkcg_css(bio); - bio_associate_blkg_from_css(bio, css); - } else { - rcu_read_lock(); - css = blkcg_css(); - if (!css_tryget_online(css)) - css = NULL; - rcu_read_unlock(); + if (bio_blkcg(bio)) + return; - bio_associate_blkg_from_css(bio, css); - if (css) - css_put(css); - } + rcu_read_lock(); + bio_associate_blkcg_from_css(bio, blkcg_css()); + rcu_read_unlock(); } -EXPORT_SYMBOL_GPL(bio_associate_blkg); +EXPORT_SYMBOL_GPL(bio_associate_blkcg); /** - * bio_clone_blkg_association - clone blkg association from src to dst bio + * bio_clone_blkcg_association - clone blkcg association from src to dst bio * @dst: destination bio * @src: source bio / -void bio_clone_blkg_association(struct bio dst, struct bio src) +void bio_clone_blkcg_association(struct bio dst, struct bio src) { - if (src->bi_blkg) - bio_associate_blkg_from_css(dst, bio_blkcg_css(src)); + if (bio_blkcg(src)) + bio_associate_blkcg_from_css(dst, bio_blkcg_css(src)); } -EXPORT_SYMBOL_GPL(bio_clone_blkg_association); +EXPORT_SYMBOL_GPL(bio_clone_blkcg_association); static int blk_cgroup_io_type(struct bio bio) { @@ -2199,20 +2247,27 @@ static int blk_cgroup_io_type(struct bio bio) void blk_cgroup_bio_start(struct bio bio) { - struct blkcg blkcg = bio->bi_blkg->blkcg; + struct blkcg blkcg = bio_blkcg(bio); + struct blkcg_gq blkg; int rwd = blk_cgroup_io_type(bio), cpu; struct blkg_iostat_set bis; unsigned long flags; if (!cgroup_subsys_on_dfl(io_cgrp_subsys)) return; + if (!blkcg) + return; /* Root-level stats are sourced from system-wide IO stats / if (!cgroup_parent(blkcg->css.cgroup)) return; + blkg = bio_blkg_lookup(bio); + if (!blkg) + return; + cpu = get_cpu(); - bis = per_cpu_ptr(bio->bi_blkg->iostat_cpu, cpu); + bis = per_cpu_ptr(blkg->iostat_cpu, cpu); flags = u64_stats_update_begin_irqsave(&bis->sync); / diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index e67c698391294..eb76cc38b41ba 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -19,6 +19,8 @@ #include <linux/kthread.h> #include <linux/blk-mq.h> #include <linux/llist.h> +#include <linux/rhashtable.h> +#include <linux/rcupdate.h> #include "blk.h" struct blkcg_gq; @@ -56,9 +58,11 @@ struct blkg_iostat_set { struct blkcg_gq { /* Pointer to the associated request_queue / struct request_queue q; + struct rhash_head q_hash_node; struct list_head q_node; struct hlist_node blkcg_node; struct blkcg blkcg; + int blkcg_id; / all non-root blkcg_gq's are guaranteed to have access to parent / struct blkcg_gq parent; @@ -73,14 +77,7 @@ struct blkcg_gq { struct blkg_iostat_set iostat; struct blkg_policy_data pd[BLKCG_MAX_POLS]; -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO - spinlock_t async_bio_lock; - struct bio_list async_bios; -#endif - union { - struct work_struct async_bio_work; - struct work_struct free_work; - }; + struct work_struct free_work; atomic_t use_delay; atomic64_t delay_nsec; @@ -98,8 +95,6 @@ struct blkcg { / If there is block congestion on this cgroup. / atomic_t congestion_count; - struct radix_tree_root blkg_tree; - struct blkcg_gq __rcu blkg_hint; struct hlist_head blkg_list; struct blkcg_policy_data cpd[BLKCG_MAX_POLS]; @@ -111,6 +106,11 @@ struct blkcg { / struct llist_head __percpu lhead; +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO + spinlock_t async_bio_lock; / protects async_bios / + struct bio_list async_bios; + struct work_struct async_bio_work; +#endif #ifdef CONFIG_BLK_CGROUP_FC_APPID char fc_app_id[FC_APPID_LEN]; #endif @@ -124,6 +124,11 @@ static inline struct blkcg css_to_blkcg(struct cgroup_subsys_state css) return css ? container_of(css, struct blkcg, css) : NULL; } +static inline struct blkcg bio_blkcg(struct bio bio) +{ + return bio->bi_blkcg; +} + / * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a * request_queue (q). This is used by blkcg policies which need to track @@ -192,8 +197,10 @@ struct blkcg_policy { extern struct blkcg blkcg_root; extern bool blkcg_debug_stats; +extern const struct rhashtable_params blkg_hash_params; -void blkg_init_queue(struct request_queue q); +int blkg_init_queue(struct request_queue q); +void blkg_exit_queue(struct request_queue q); int blkcg_init_disk(struct gendisk disk); void blkcg_exit_disk(struct gendisk disk); @@ -249,11 +256,33 @@ static inline bool bio_issue_as_root_blkg(struct bio bio) } /** - * blkg_lookup - lookup blkg for the specified blkcg - q pair + * blkg_lookup_any - lookup any blkg for the specified blkcg - q pair + * @blkcg: blkcg of interest + * @q: request_queue of interest + * + * Lookup a blkg for the @blkcg - @q pair, whether it is online or dying. + * + * Must be called in a RCU critical section. + * + * This does not acquire a reference. The caller must already hold one, or + * have the blkg pinned by I/O. + / +static inline struct blkcg_gq blkg_lookup_any(struct blkcg blkcg, + struct request_queue q) +{ + RCU_LOCKDEP_WARN(!rcu_read_lock_held(), + "blkg_lookup_any() requires an RCU read lock"); + + return rhashtable_lookup(&q->blkg_hash, &blkcg->css.id, + blkg_hash_params); +} + +/** + * blkg_lookup - lookup an online blkg for the specified blkcg - q pair * @blkcg: blkcg of interest * @q: request_queue of interest * - * Lookup blkg for the @blkcg - @q pair. + * Lookup an online blkg for the @blkcg - @q pair. * * Must be called in a RCU critical section. / @@ -265,17 +294,15 @@ static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg, if (blkcg == &blkcg_root) return q->root_blkg; - blkg = rcu_dereference_check(blkcg->blkg_hint, - lockdep_is_held(&q->queue_lock)); - if (blkg && blkg->q == q) - return blkg; - - blkg = radix_tree_lookup(&blkcg->blkg_tree, q->id); - if (blkg && blkg->q != q) + blkg = blkg_lookup_any(blkcg, q); + if (blkg && !READ_ONCE(blkg->online)) blkg = NULL; return blkg; } +struct blkcg_gq bio_blkg_lookup(struct bio bio); +struct blkcg_gq bio_blkg(struct bio bio); + /* * blkg_to_pd - get policy private data * @blkg: blkg of interest @@ -343,6 +370,18 @@ static inline void blkg_put(struct blkcg_gq blkg) percpu_ref_put(&blkg->refcnt); } +static inline void bio_clear_blkcg(struct bio bio) +{ + struct blkcg blkcg = bio_blkcg(bio); + + bio_put_blkg_ref(bio); + + if (blkcg) { + css_put(&blkcg->css); + bio->bi_blkcg = NULL; + } +} + /* * blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants * @d_blkg: loop cursor pointing to the current descendant @@ -472,7 +511,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq blkg) / static inline bool blk_cgroup_mergeable(struct request rq, struct bio bio) { - return rq->bio->bi_blkg == bio->bi_blkg && + return bio_blkcg(rq->bio) == bio_blkcg(bio) && bio_issue_as_root_blkg(rq->bio) == bio_issue_as_root_blkg(bio); } @@ -498,8 +537,13 @@ struct blkcg_policy { struct blkcg { }; +static inline struct blkcg bio_blkcg(struct bio bio) { return NULL; } +static inline struct blkcg_gq bio_blkg_lookup(struct bio bio) { return NULL; } +static inline struct blkcg_gq bio_blkg(struct bio bio) { return NULL; } +static inline struct blkcg_gq blkg_lookup_any(struct blkcg blkcg, void key) { return NULL; } static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg, void key) { return NULL; } -static inline void blkg_init_queue(struct request_queue q) { } +static inline int blkg_init_queue(struct request_queue q) { return 0; } +static inline void blkg_exit_queue(struct request_queue q) { } static inline int blkcg_init_disk(struct gendisk disk) { return 0; } static inline void blkcg_exit_disk(struct gendisk disk) { } static inline int blkcg_policy_register(struct blkcg_policy pol) { return 0; } @@ -514,6 +558,7 @@ static inline struct blkg_policy_data blkg_to_pd(struct blkcg_gq blkg, static inline struct blkcg_gq pd_to_blkg(struct blkg_policy_data pd) { return NULL; } static inline void blkg_get(struct blkcg_gq blkg) { } static inline void blkg_put(struct blkcg_gq blkg) { } +static inline void bio_clear_blkcg(struct bio bio) { } static inline void blk_cgroup_bio_start(struct bio bio) { } static inline bool blk_cgroup_mergeable(struct request rq, struct bio bio) { return true; } diff --git a/block/blk-core.c b/block/blk-core.c index 196bccf27f58d..e396a316b8bf6 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -303,6 +303,7 @@ static void blk_free_queue(struct request_queue q) if (queue_is_mq(q)) blk_mq_release(q); + blkg_exit_queue(q); ida_free(&blk_queue_ida, q->id); lockdep_unregister_key(&q->io_lock_cls_key); lockdep_unregister_key(&q->q_lock_cls_key); @@ -481,7 +482,9 @@ struct request_queue blk_alloc_queue(struct queue_limits lim, int node_id) init_waitqueue_head(&q->mq_freeze_wq); mutex_init(&q->mq_freeze_lock); - blkg_init_queue(q); + error = blkg_init_queue(q); + if (error) + goto fail_stats; / * Init percpu_ref in atomic mode so that it's faster to shutdown. @@ -491,7 +494,7 @@ struct request_queue blk_alloc_queue(struct queue_limits lim, int node_id) blk_queue_usage_counter_release, PERCPU_REF_INIT_ATOMIC, GFP_KERNEL); if (error) - goto fail_stats; + goto fail_blkg; lockdep_register_key(&q->io_lock_cls_key); lockdep_register_key(&q->q_lock_cls_key); lockdep_init_map(&q->io_lockdep_map, "&q->q_usage_counter(io)", @@ -510,6 +513,8 @@ struct request_queue blk_alloc_queue(struct queue_limits lim, int node_id) return q; +fail_blkg: + blkg_exit_queue(q); fail_stats: blk_free_queue_stats(q->stats); fail_id: diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c index 2a5c52ab74b4c..5ef4baab444bb 100644 --- a/block/blk-crypto-fallback.c +++ b/block/blk-crypto-fallback.c @@ -187,7 +187,7 @@ static struct bio blk_crypto_alloc_enc_bio(struct bio bio_src, bio->bi_write_hint = bio_src->bi_write_hint; bio->bi_write_stream = bio_src->bi_write_stream; bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; - bio_clone_blkg_association(bio, bio_src); + bio_clone_blkcg_association(bio, bio_src); /* * Move page array up in the allocated memory for the bio vecs as far as diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 2745bffcd5eef..57f2b4d4af20a 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -2686,7 +2686,7 @@ iocg_handle_over_budget(struct rq_qos rqos, struct ioc_gq iocg, static void ioc_rqos_throttle(struct rq_qos rqos, struct bio bio) { - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg_gq blkg = bio_blkg(bio); struct ioc ioc = rqos_to_ioc(rqos); struct ioc_gq iocg = blkg_to_iocg(blkg); struct ioc_now now; @@ -2775,7 +2775,7 @@ static void ioc_rqos_throttle(struct rq_qos rqos, struct bio bio) static void ioc_rqos_merge(struct rq_qos rqos, struct request rq, struct bio bio) { - struct ioc_gq iocg = blkg_to_iocg(bio->bi_blkg); + struct ioc_gq iocg = blkg_to_iocg(bio_blkg(bio)); struct ioc ioc = rqos_to_ioc(rqos); sector_t bio_end = bio_end_sector(bio); struct ioc_now now; @@ -2833,9 +2833,13 @@ static void ioc_rqos_merge(struct rq_qos rqos, struct request rq, static void ioc_rqos_done_bio(struct rq_qos rqos, struct bio bio) { - struct ioc_gq iocg = blkg_to_iocg(bio->bi_blkg); + struct ioc_gq iocg; + + if (!bio->bi_iocost_cost) + return; - if (iocg && bio->bi_iocost_cost) + iocg = blkg_to_iocg(bio_blkg_lookup(bio)); + if (iocg) atomic64_add(bio->bi_iocost_cost, &iocg->done_vtime); } diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 2caa79a008ad1..7220edafd72b6 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -463,7 +463,7 @@ static void check_scale_change(struct iolatency_grp iolat) static void blkcg_iolatency_throttle(struct rq_qos rqos, struct bio bio) { struct blk_iolatency blkiolat = BLKIOLATENCY(rqos); - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg_gq blkg = bio_blkg(bio); bool issue_as_root = bio_issue_as_root_blkg(bio); if (!blkiolat->enabled) @@ -590,11 +590,14 @@ static void blkcg_iolatency_done_bio(struct rq_qos rqos, struct bio bio) bool issue_as_root = bio_issue_as_root_blkg(bio); int inflight = 0; - blkg = bio->bi_blkg; - if (!blkg \|\| !bio_flagged(bio, BIO_QOS_THROTTLED)) + if (!bio_flagged(bio, BIO_QOS_THROTTLED)) return; - iolat = blkg_to_lat(bio->bi_blkg); + blkg = bio_blkg_lookup(bio); + if (!blkg) + return; + + iolat = blkg_to_lat(blkg); if (!iolat) return; diff --git a/block/blk-ioprio.c b/block/blk-ioprio.c index 8fa8bca350629..5e3f8e49550da 100644 --- a/block/blk-ioprio.c +++ b/block/blk-ioprio.c @@ -132,7 +132,7 @@ static struct blkcg_policy ioprio_policy = { void blkcg_set_ioprio(struct bio bio) { - struct ioprio_blkcg blkcg = blkcg_to_ioprio_blkcg(bio->bi_blkg->blkcg); + struct ioprio_blkcg blkcg = blkcg_to_ioprio_blkcg(bio_blkcg(bio)); u16 prio; if (!blkcg \|\| blkcg->prio_policy == POLICY_NO_CHANGE) diff --git a/block/blk-throttle.c b/block/blk-throttle.c index ffc3b70065d4b..3828c38579002 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -1792,7 +1792,7 @@ static bool tg_within_limit(struct throtl_grp tg, struct bio bio, bool rw) bool __blk_throtl_bio(struct bio bio) { struct request_queue q = bdev_get_queue(bio->bi_bdev); - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg_gq blkg = bio_blkg(bio); struct throtl_qnode qn = NULL; struct throtl_grp tg = blkg_to_tg(blkg); struct throtl_service_queue sq; diff --git a/block/blk-throttle.h b/block/blk-throttle.h index 1b5775771e078..45f30179fd9c0 100644 --- a/block/blk-throttle.h +++ b/block/blk-throttle.h @@ -171,7 +171,7 @@ static inline bool blk_should_throtl(struct bio bio) if (!blk_throtl_activated(bio->bi_bdev->bd_queue)) return false; - tg = blkg_to_tg(bio->bi_blkg); + tg = blkg_to_tg(bio_blkg(bio)); if (!cgroup_subsys_on_dfl(io_cgrp_subsys)) { if (!bio_flagged(bio, BIO_CGROUP_ACCT)) { bio_set_flag(bio, BIO_CGROUP_ACCT); diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index 3fa3b13a410f4..c0f945b8d9413 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -848,7 +848,7 @@ static CLOSURE_CALLBACK(cached_dev_read_done) s->iop.bio->bi_iter.bi_sector = s->cache_miss->bi_iter.bi_sector; s->iop.bio->bi_iter.bi_size = s->insert_bio_sectors << 9; - bio_clone_blkg_association(s->iop.bio, s->cache_miss); + bio_clone_blkcg_association(s->iop.bio, s->cache_miss); bch_bio_map(s->iop.bio, NULL); bio_copy_data(s->cache_miss, s->iop.bio); diff --git a/drivers/md/dm.c b/drivers/md/dm.c index d413bfaf35278..cd68eec77f5a1 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1373,7 +1373,7 @@ void dm_submit_bio_remap(struct bio clone, struct bio tgt_clone) if (!tgt_clone) tgt_clone = clone; - bio_clone_blkg_association(tgt_clone, io->orig_bio); + bio_clone_blkcg_association(tgt_clone, io->orig_bio); / * Account io->origin_bio to DM dev on behalf of target diff --git a/drivers/md/md.c b/drivers/md/md.c index 680b34a63cb3d..fd78d6cc8da17 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -9386,7 +9386,7 @@ void md_submit_discard_bio(struct mddev mddev, struct md_rdev rdev, return; bio_chain(discard_bio, bio); - bio_clone_blkg_association(discard_bio, bio); + bio_clone_blkcg_association(discard_bio, bio); mddev_trace_remap(mddev, discard_bio, bio->bi_iter.bi_sector); submit_bio_noacct(discard_bio); } diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c index 4176046627beb..54e4adb0ccb7d 100644 --- a/drivers/nvdimm/nd_virtio.c +++ b/drivers/nvdimm/nd_virtio.c @@ -121,7 +121,7 @@ int async_pmem_flush(struct nd_region nd_region, struct bio bio) if (!child) return -ENOMEM; - bio_clone_blkg_association(child, bio); + bio_clone_blkcg_association(child, bio); child->bi_iter.bi_sector = -1; bio_chain(child, bio); submit_bio(child); diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c index 6dabe73ad790d..6512dbd9516fc 100644 --- a/fs/gfs2/lops.c +++ b/fs/gfs2/lops.c @@ -484,7 +484,7 @@ static struct bio gfs2_chain_bio(struct bio prev, unsigned int nr_iovecs, struct bio new; new = bio_alloc(prev->bi_bdev, nr_iovecs, opf, GFP_NOIO); - bio_clone_blkg_association(new, prev); + bio_clone_blkcg_association(new, prev); new->bi_iter.bi_sector = sector; bio_chain(new, prev); submit_bio(prev); @@ -1114,4 +1114,3 @@ const struct gfs2_log_operations gfs2_log_ops[] = { &gfs2_revoke_lops, NULL, }; - diff --git a/include/linux/bio.h b/include/linux/bio.h index bb3235497e673..11637f85c38d4 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -542,18 +542,20 @@ static inline void bio_release_pages(struct bio bio, bool mark_dirty) disk_devt((bio)->bi_bdev->bd_disk) #ifdef CONFIG_BLK_CGROUP -void bio_associate_blkg(struct bio bio); -void bio_associate_blkg_from_css(struct bio bio, - struct cgroup_subsys_state css); -void bio_clone_blkg_association(struct bio dst, struct bio src); +void bio_associate_blkcg(struct bio bio); +void bio_associate_blkcg_from_css(struct bio bio, + struct cgroup_subsys_state css); +void bio_clone_blkcg_association(struct bio dst, struct bio src); +void bio_put_blkg_ref(struct bio bio); void blkcg_punt_bio_submit(struct bio bio); #else / CONFIG_BLK_CGROUP / -static inline void bio_associate_blkg(struct bio bio) { } -static inline void bio_associate_blkg_from_css(struct bio bio, - struct cgroup_subsys_state css) +static inline void bio_associate_blkcg(struct bio bio) { } +static inline void bio_associate_blkcg_from_css(struct bio bio, + struct cgroup_subsys_state css) { } -static inline void bio_clone_blkg_association(struct bio dst, - struct bio src) { } +static inline void bio_clone_blkcg_association(struct bio dst, + struct bio src) { } +static inline void bio_put_blkg_ref(struct bio bio) { } static inline void blkcg_punt_bio_submit(struct bio bio) { submit_bio(bio); @@ -563,10 +565,12 @@ static inline void blkcg_punt_bio_submit(struct bio bio) static inline void bio_set_dev(struct bio bio, struct block_device bdev) { bio_clear_flag(bio, BIO_REMAPPED); - if (bio->bi_bdev != bdev) + if (bio->bi_bdev != bdev) { + bio_put_blkg_ref(bio); bio_clear_flag(bio, BIO_BPS_THROTTLED); + } bio->bi_bdev = bdev; - bio_associate_blkg(bio); + bio_associate_blkcg(bio); } /* diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index d49d97a050d0e..53ba307b91cb3 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -246,12 +246,10 @@ struct bio { void bi_private; #ifdef CONFIG_BLK_CGROUP / - * Represents the association of the css and request_queue for the bio. - * If a bio goes direct to device, it will not have a blkg as it will - * not have a request_queue associated with it. The reference is put - * on release of the bio. + * Represents the blkcg css association for the bio. The reference is + * put on release of the bio. / - struct blkcg_gq bi_blkg; + struct blkcg bi_blkcg; / Time that this bio was issued. / u64 issue_time_ns; #ifdef CONFIG_BLK_CGROUP_IOCOST @@ -309,6 +307,7 @@ enum { BIO_TRACE_COMPLETION, / bio_endio() should trace the final completion * of this bio. / BIO_CGROUP_ACCT, / has been accounted to a cgroup / + BIO_BLKG_REF, / bio pins the associated blkg / BIO_QOS_THROTTLED, / bio went through rq_qos throttle path / / * This bio has completed bps throttling at the single tg granularity, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 20cb8ed7d987a..f3b0e39d3f6b4 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -27,6 +27,7 @@ #include <linux/xarray.h> #include <linux/file.h> #include <linux/lockdep.h> +#include <linux/rhashtable-types.h> struct module; struct request_queue; @@ -581,6 +582,7 @@ struct request_queue { #ifdef CONFIG_BLK_CGROUP DECLARE_BITMAP (blkcg_pols, BLKCG_MAX_POLS); struct blkcg_gq root_blkg; + struct rhashtable blkg_hash; struct list_head blkg_list; struct mutex blkcg_mutex; #endif diff --git a/include/linux/writeback.h b/include/linux/writeback.h index 62552a2ce5b9e..4f869fe9cc90f 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -262,7 +262,7 @@ static inline void wbc_init_bio(struct writeback_control wbc, struct bio bio) regular writeback instead of writing things out itself. / if (wbc->wb) - bio_associate_blkg_from_css(bio, wbc->wb->blkcg_css); + bio_associate_blkcg_from_css(bio, wbc->wb->blkcg_css); } void inode_switch_wbs_work_fn(struct work_struct work); diff --git a/mm/page_io.c b/mm/page_io.c index b23f494fcc83d..a4a1b8a8a0e38 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -316,7 +316,7 @@ static inline void count_swpout_vm_event(struct folio folio) } #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) -static void bio_associate_blkg_from_page(struct bio bio, struct folio folio) +static void bio_associate_blkcg_from_page(struct bio bio, struct folio folio) { struct cgroup_subsys_state css; struct mem_cgroup memcg; @@ -331,12 +331,15 @@ static void bio_associate_blkg_from_page(struct bio bio, struct folio folio) css = NULL; rcu_read_unlock(); - bio_associate_blkg_from_css(bio, css); + bio_associate_blkcg_from_css(bio, css); if (css) css_put(css); } #else -#define bio_associate_blkg_from_page(bio, folio) do { } while (0) +static inline void bio_associate_blkcg_from_page(struct bio bio, + struct folio folio) +{ +} #endif / CONFIG_MEMCG && CONFIG_BLK_CGROUP / struct swap_iocb { @@ -436,7 +439,7 @@ static void swap_writepage_bdev_sync(struct folio folio, bio.bi_iter.bi_sector = swap_folio_sector(folio); bio_add_folio_nofail(&bio, folio, folio_size(folio), 0); - bio_associate_blkg_from_page(&bio, folio); + bio_associate_blkcg_from_page(&bio, folio); count_swpout_vm_event(folio); folio_start_writeback(folio); @@ -456,7 +459,7 @@ static void swap_writepage_bdev_async(struct folio *folio, bio->bi_end_io = end_swap_bio_write; bio_add_folio_nofail(bio, folio, folio_size(folio), 0); - bio_associate_blkg_from_page(bio, folio); + bio_associate_blkcg_from_page(bio, folio); count_swpout_vm_event(folio); folio_start_writeback(folio); folio_unlock(folio);
3/2	2026/08/18 08:04	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 21145 output: 31 thoughts: 519 Thoughts: Analyzing a Block Cgroup Patch: Reachability and Configuration Okay, so I'm looking at this patch that fundamentally alters how `bio`s are linked to block cgroups – a significant change to the `blkcg` and `bio` association logic. Instead of the current `bio->bi_blkg` using queue-specific `blkcg_gq`, it's switching to `bio->bi_blkcg` holding the general `blkcg`. The `blkcg_gq` (blkg) is now either looked up dynamically or pinned via the new `BIO_BLKG_REF` flag. This is a big deal; this impacts the core block layer. My initial thoughts are, let's nail down reachability and configurations. The code's spread across `block/blk-cgroup.c`, `block/bio.c`, `block/bfq-cgroup.c`, and likely others. Given it touches core block layer components, there's no question about reachability here. Now, configuration is next. I see these configurations involved: `CONFIG_BLK_CGROUP`, `CONFIG_BFQ_GROUP_IOSCHED`, `CONFIG_BLK_CGROUP_PUNT_BIO`, `CONFIG_BLK_CGROUP_IOCOST`, and `CONFIG_BLK_CGROUP_FC_APPID`. Next step, I need to zero in on the functions that have been modified or introduced. I see: `bio_blkg_lookup`, `bio_blkg`, `bio_associate_blkcg_from_css`, `bio_associate_blkcg`, `bio_clone_blkcg_association`, and `bio_put_blkg_ref`. I'll start by checking `block/blk-cgroup.c` to see if those functions are defined there. I confirm, they are all defined there. Alright, let's do a quick grep to make sure they're exported. Yep, they are using `EXPORT_SYMBOL_GPL`, which is as expected. Finally, I need to check the Kconfig for `BLK_CGROUP`. Looks like it depends on both `CGROUPS` and `BLOCK`, which is standard. `BFQ_GROUP_IOSCHED` in turn depends on `IOSCHED_BFQ` and, as expected, `BLK_CGROUP`. What about `BLK_CGROUP_PUNT_BIO`? That needs closer examination to see if it's an internal config or if it has any additional dependencies.
4/2	2026/08/18 08:04	tool	grepper	0m Args: { "Expression": "config BLK_CGROUP", "PathPrefix": "block/Kconfig" } Results: { "Output": "block/Kconfig=39=config BLK_RQ_ALLOC_TIME\n--\nblock/Kconfig-41-\nblock/Kconfig:42:config BLK_CGROUP_RWSTAT\nblock/Kconfig-43-\tbool\nblock/Kconfig-44-\nblock/Kconfig:45:config BLK_CGROUP_PUNT_BIO\nblock/Kconfig-46-\tbool\n--\nblock/Kconfig=127=config BLK_WBT_MQ\n--\nblock/Kconfig-133-\nblock/Kconfig:134:config BLK_CGROUP_IOLATENCY\nblock/Kconfig-135-\tbool \"Enable support for latency based cgroup IO protection\"\n--\nblock/Kconfig-144-\nblock/Kconfig:145:config BLK_CGROUP_FC_APPID\nblock/Kconfig-146-\tbool \"Enable support to track FC I/O Traffic across cgroup applications\"\n--\nblock/Kconfig-153-\nblock/Kconfig:154:config BLK_CGROUP_IOCOST\nblock/Kconfig-155-\tbool \"Enable support for cost model based cgroup IO controller\"\n--\nblock/Kconfig-163-\nblock/Kconfig:164:config BLK_CGROUP_IOPRIO\nblock/Kconfig-165-\tbool \"Cgroup I/O controller for assigning an I/O priority class\"\n" }
5/2	2026/08/18 08:04	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 22609 output: 27 thoughts: 25 Thoughts: Here's my thought process, as if I were considering this myself: Investigating the BLK_CGROUP Configuration Option Okay, so I need to track down where the `BLK_CGROUP` configuration option is defined. My first instinct, given its relation to block devices and control groups, is to look in the kernel configuration files. The two most likely candidates are `init/Kconfig` and `block/Kconfig`. `init/Kconfig` often handles fundamental initialization options, and `block/Kconfig` obviously deals with block device-related configurations. I'll start by checking those two files – that should be a good starting point to locate the definition or dependency of `BLK_CGROUP`. It's a pretty standard approach. I expect to find it declared with its help text and default value. From there, I can trace any dependencies if needed. Let's dig in.
6/2	2026/08/18 08:04	tool	grepper	0m Args: { "Expression": "config BLK_CGROUP", "PathPrefix": "" } Results: { "Output": "block/Kconfig=39=config BLK_RQ_ALLOC_TIME\n--\nblock/Kconfig-41-\nblock/Kconfig:42:config BLK_CGROUP_RWSTAT\nblock/Kconfig-43-\tbool\nblock/Kconfig-44-\nblock/Kconfig:45:config BLK_CGROUP_PUNT_BIO\nblock/Kconfig-46-\tbool\n--\nblock/Kconfig=127=config BLK_WBT_MQ\n--\nblock/Kconfig-133-\nblock/Kconfig:134:config BLK_CGROUP_IOLATENCY\nblock/Kconfig-135-\tbool \"Enable support for latency based cgroup IO protection\"\n--\nblock/Kconfig-144-\nblock/Kconfig:145:config BLK_CGROUP_FC_APPID\nblock/Kconfig-146-\tbool \"Enable support to track FC I/O Traffic across cgroup applications\"\n--\nblock/Kconfig-153-\nblock/Kconfig:154:config BLK_CGROUP_IOCOST\nblock/Kconfig-155-\tbool \"Enable support for cost model based cgroup IO controller\"\n--\nblock/Kconfig-163-\nblock/Kconfig:164:config BLK_CGROUP_IOPRIO\nblock/Kconfig-165-\tbool \"Cgroup I/O controller for assigning an I/O priority class\"\n--\ninit/Kconfig=1099=config MEMCG_V1\n--\ninit/Kconfig-1115-\ninit/Kconfig:1116:config BLK_CGROUP\ninit/Kconfig-1117-\tbool \"IO controller\"\n" }
7/2	2026/08/18 08:04	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 23288 output: 33 thoughts: 18 Thoughts: Here's a summary of my thought process, as requested: Checking BLK_CGROUP Configuration Okay, so I need to quickly ascertain the status of `BLK_CGROUP` within the kernel configuration. My immediate instinct is to check `init/Kconfig`. That's the most logical place to start digging for definitions related to block layer control groups, especially since it's a core configuration file. Let's see if it's defined there, and if so, what the dependencies and default value are. This should give me a good initial overview of how `BLK_CGROUP` is handled within the build system and if it is enabled or disabled by default. Understanding the configuration is critical to moving forward.
8/2	2026/08/18 08:04	tool	read-file	0m Args: { "File": "init/Kconfig", "FirstLine": 1110, "LineCount": 20 } Results: { "Contents": "1110:\t\t Please note that feature set of the legacy memory controller is likely\n1111:\t\t going to shrink due to deprecation process. New deployments with v1\n1112:\t\t controller are highly discouraged.\n1113:\t\n1114:\t\t Say N if unsure.\n1115:\t\n1116:\tconfig BLK_CGROUP\n1117:\t\tbool \"IO controller\"\n1118:\t\tdepends on BLOCK\n1119:\t\tdefault n\n1120:\t\thelp\n1121:\t\tGeneric block IO controller cgroup interface. This is the common\n1122:\t\tcgroup interface which should be used by various IO controlling\n1123:\t\tpolicies.\n1124:\t\n1125:\t\tCurrently, CFQ IO scheduler uses it to recognize task groups and\n1126:\t\tcontrol disk bandwidth allocation (proportional time slice allocation)\n1127:\t\tto such task groups. It is also used by bio throttling logic in\n1128:\t\tblock layer to implement upper limit in IO rates on a device.\n1129:\t\n" }
9/2	2026/08/18 08:04	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 23781 output: 29 thoughts: 42 Thoughts: Okay, here's my summary, taking on the persona you've described: Understanding the Block IO Scheduler Dependencies Alright, let's break this down. From what I'm seeing here, `BLK_CGROUP` is a core dependency for the block layer, that's not surprising. But more interestingly, `BFQ_GROUP_IOSCHED` requires `IOSCHED_BFQ` and, critically, `BLK_CGROUP`. So, to get the BFQ group I/O scheduler up and running, I need both the base BFQ scheduler and the control group infrastructure. This confirms my understanding. I need to make sure I have the necessary configurations enabled in my kernel to use this correctly. These dependencies are crucial to managing and prioritizing block I/O at a granular, group-specific level. Okay, that's straightforward.
10/2	2026/08/18 08:04	tool	grepper	0m Args: { "Expression": "config BFQ_GROUP_IOSCHED", "PathPrefix": "" } Results: { "Output": "block/Kconfig.iosched=19=config IOSCHED_BFQ\n--\nblock/Kconfig.iosched-29-\nblock/Kconfig.iosched:30:config BFQ_GROUP_IOSCHED\nblock/Kconfig.iosched-31- bool \"BFQ hierarchical scheduling support\"\n" }
11/2	2026/08/18 08:04	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 24125 output: 34 thoughts: 24 Thoughts: Here's a summary of my immediate thought process: Kconfig Dive for BFQ Group Scheduling Okay, time to get to the heart of this. I need to figure out if `BFQ_GROUP_IOSCHED` is enabled and how it's configured. That means I need to start digging into the kernel configuration. The logical place to start is the `block/Kconfig.iosched` file. It's the standard location for I/O scheduler-related configurations, and `BFQ` is an I/O scheduler, so that's where I'd expect to find this particular option. Let's get to it. Time to open that file and see what's what.
12/2	2026/08/18 08:04	tool	read-file	0m Args: { "File": "block/Kconfig.iosched", "FirstLine": 25, "LineCount": 20 } Results: { "Contents": " 25:\t\tregardless of the device parameters and with any workload. It\n 26:\t\talso guarantees a low latency to interactive and soft\n 27:\t\treal-time applications. Details in\n 28:\t\tDocumentation/block/bfq-iosched.rst\n 29:\t\n 30:\tconfig BFQ_GROUP_IOSCHED\n 31:\t bool \"BFQ hierarchical scheduling support\"\n 32:\t depends on IOSCHED_BFQ \u0026\u0026 BLK_CGROUP\n 33:\t default y\n 34:\t select BLK_CGROUP_RWSTAT\n 35:\t\thelp\n 36:\t\n 37:\t Enable hierarchical scheduling in BFQ, using the blkio\n 38:\t (cgroups-v1) or io (cgroups-v2) controller.\n 39:\t\n 40:\tconfig BFQ_CGROUP_DEBUG\n 41:\t\tbool \"BFQ IO controller debugging\"\n 42:\t\tdepends on BFQ_GROUP_IOSCHED\n 43:\t\thelp\n 44:\t\tEnable some debugging help. Currently it exports additional stat\n" }
13/2	2026/08/18 08:04	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 24625 output: 26 thoughts: 522 Thoughts: Fuzzing Target Identification: Block Cgroup Changes Alright, let's break this down. This patch touches a whole slew of files within the `block/` directory: `blk-cgroup.c`, `bio.c`, `bfq-cgroup.c`, `blk-iocost.c`, `blk-iolatency.c`, `blk-ioprio.c`, and `blk-throttle.c`. All block cgroup related, naturally. The changes seem functional, which makes them prime fuzzing targets. Specifically, I need to zero in on these functions: `bio_blkg_lookup`, `bio_blkg`, `bio_associate_blkcg_from_css`, `bio_associate_blkcg`, `bio_clone_blkcg_association`, and `bio_put_blkg_ref`. Now, the key is to determine which of these are likely to be in the hot path, and therefore more impactful to fuzz. `bio_associate_blkcg` is called within both `bio_init` and `bio_reset` if a `bdev` is set. `bio_init` is definitely hot-path. Which means `bio_associate_blkcg`, which calls `bio_associate_blkcg_from_css`, is hot path. I think these should be the first ones I focus on. `bio_blkg` is also interesting because it's called within `bfq_bio_merge`, `bfq_insert_request`, `ioc_rqos_throttle`, `blkcg_iolatency_throttle`, and `__blk_throtl_bio`, which are all hot paths since they are called with every bio. While `bio_blkg` is a new function, it is being called in hot-path, so it's a good place to concentrate on. I also need to consider: `blkg_lookup_create`, `blkg_conf_prep`, `blkcg_css_alloc`, `blkg_init_queue`, and `blkcg_init_disk`. However, a quick scan suggests these are not as frequently accessed: `blkg_conf_prep` is involved in configuring blkcg policies (not hot-path), `blkg_init_queue` is used during queue allocation (also not hot-path), and `blkcg_init_disk` during disk initialization (again, not hot-path). `bio_associate_blkcg_from_css` also gets called in `bio_clone_blkcg_association`. Therefore, I'll prioritize fuzzing `bio_blkg` and `bio_associate_blkcg`, `bio_associate_blkcg_from_css` and `bio_clone_blkcg_association` and dig deeper into those. Let's see how they work.
14/2	2026/08/18 08:05	tool	grepper	0m Args: { "Expression": "bio_blkg", "PathPrefix": "block/" } Results: { "Output": "block/bfq-cgroup.c=364=void bfqg_stats_update_legacy_io(struct request_queue q, struct request rq)\nblock/bfq-cgroup.c-365-{\nblock/bfq-cgroup.c:366:\tstruct blkcg_gq blkg = bio_blkg_lookup(rq-\u003ebio);\nblock/bfq-cgroup.c-367-\tstruct bfq_group bfqg;\n--\nblock/bfq-cgroup.c=609=struct bfq_group bfq_bio_bfqg(struct bfq_data bfqd, struct bio bio)\nblock/bfq-cgroup.c-610-{\nblock/bfq-cgroup.c:611:\tstruct blkcg_gq blkg = bio_blkg_lookup(bio);\nblock/bfq-cgroup.c-612-\tstruct bfq_group bfqg;\n--\nblock/bfq-iosched.c=2448=static bool bfq_bio_merge(struct request_queue q, struct bio bio,\n--\nblock/bfq-iosched.c-2458-\tif (bic)\nblock/bfq-iosched.c:2459:\t\tbio_blkg(bio);\nblock/bfq-iosched.c-2460-#endif\n--\nblock/bfq-iosched.c=6239=static void bfq_insert_request(struct blk_mq_hw_ctx hctx, struct request rq,\n--\nblock/bfq-iosched.c-6254-\tif (rq-\u003ebio \u0026\u0026 !bio_flagged(rq-\u003ebio, BIO_BLKG_REF))\nblock/bfq-iosched.c:6255:\t\tbio_blkg(rq-\u003ebio);\nblock/bfq-iosched.c-6256-\n--\nblock/blk-cgroup.c=2096=static void bio_set_blkg_ref(struct bio bio, struct blkcg_gq blkg)\n--\nblock/blk-cgroup.c-2106-/\nblock/blk-cgroup.c:2107: bio_blkg_lookup - look up a blkg associated with a bio\nblock/blk-cgroup.c-2108- * @bio: target bio\n--\nblock/blk-cgroup.c-2110- * Return the blkg already pinned by @bio, or %NULL if @bio doesn't own a blkg\nblock/blk-cgroup.c:2111: * reference. Call bio_blkg() instead when a missing blkg should be created.\nblock/blk-cgroup.c-2112- /\nblock/blk-cgroup.c:2113:struct blkcg_gq bio_blkg_lookup(struct bio bio)\nblock/blk-cgroup.c-2114-{\n--\nblock/blk-cgroup.c-2118-}\nblock/blk-cgroup.c:2119:EXPORT_SYMBOL_GPL(bio_blkg_lookup);\nblock/blk-cgroup.c-2120-\n--\nblock/blk-cgroup.c-2124- \nblock/blk-cgroup.c:2125: * Drop the bio-owned blkg reference acquired by bio_blkg(), if any.\nblock/blk-cgroup.c-2126- /\n--\nblock/blk-cgroup.c=2137=EXPORT_SYMBOL_GPL(bio_put_blkg_ref);\n--\nblock/blk-cgroup.c-2139-/\nblock/blk-cgroup.c:2140: bio_blkg - look up the blkg associated with a bio\nblock/blk-cgroup.c-2141- * @bio: target bio\n--\nblock/blk-cgroup.c-2147- /\nblock/blk-cgroup.c:2148:struct blkcg_gq bio_blkg(struct bio bio)\nblock/blk-cgroup.c-2149-{\n--\nblock/blk-cgroup.c-2170-}\nblock/blk-cgroup.c:2171:EXPORT_SYMBOL_GPL(bio_blkg);\nblock/blk-cgroup.c-2172-\n--\nblock/blk-cgroup.c-2178- Associate @bio with the blkcg found from @css. The queue-local blkg is\nblock/blk-cgroup.c:2179: * created and pinned by bio_blkg() when blkcg policies need it. If @css\nblock/blk-cgroup.c-2180- * can't be referenced online, associate @bio with the root blkcg instead.\n--\nblock/blk-cgroup.c=2248=void blk_cgroup_bio_start(struct bio bio)\n--\nblock/blk-cgroup.c-2264-\nblock/blk-cgroup.c:2265:\tblkg = bio_blkg_lookup(bio);\nblock/blk-cgroup.c-2266-\tif (!blkg)\n--\nblock/blk-cgroup.h=289=static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg,\n--\nblock/blk-cgroup.h-302-\nblock/blk-cgroup.h:303:struct blkcg_gq bio_blkg_lookup(struct bio bio);\nblock/blk-cgroup.h:304:struct blkcg_gq bio_blkg(struct bio bio);\nblock/blk-cgroup.h-305-\n--\nblock/blk-cgroup.h=540=static inline struct blkcg bio_blkcg(struct bio bio) { return NULL; }\nblock/blk-cgroup.h:541:static inline struct blkcg_gq bio_blkg_lookup(struct bio bio) { return NULL; }\nblock/blk-cgroup.h:542:static inline struct blkcg_gq bio_blkg(struct bio bio) { return NULL; }\nblock/blk-cgroup.h-543-static inline struct blkcg_gq blkg_lookup_any(struct blkcg blkcg, void key) { return NULL; }\n--\nblock/blk-iocost.c=2687=static void ioc_rqos_throttle(struct rq_qos rqos, struct bio bio)\nblock/blk-iocost.c-2688-{\nblock/blk-iocost.c:2689:\tstruct blkcg_gq blkg = bio_blkg(bio);\nblock/blk-iocost.c-2690-\tstruct ioc ioc = rqos_to_ioc(rqos);\n--\nblock/blk-iocost.c=2775=static void ioc_rqos_merge(struct rq_qos rqos, struct request rq,\n--\nblock/blk-iocost.c-2777-{\nblock/blk-iocost.c:2778:\tstruct ioc_gq iocg = blkg_to_iocg(bio_blkg(bio));\nblock/blk-iocost.c-2779-\tstruct ioc ioc = rqos_to_ioc(rqos);\n--\nblock/blk-iocost.c=2834=static void ioc_rqos_done_bio(struct rq_qos rqos, struct bio bio)\n--\nblock/blk-iocost.c-2840-\nblock/blk-iocost.c:2841:\tiocg = blkg_to_iocg(bio_blkg_lookup(bio));\nblock/blk-iocost.c-2842-\tif (iocg)\n--\nblock/blk-iolatency.c=463=static void blkcg_iolatency_throttle(struct rq_qos rqos, struct bio bio)\n--\nblock/blk-iolatency.c-465-\tstruct blk_iolatency blkiolat = BLKIOLATENCY(rqos);\nblock/blk-iolatency.c:466:\tstruct blkcg_gq blkg = bio_blkg(bio);\nblock/blk-iolatency.c-467-\tbool issue_as_root = bio_issue_as_root_blkg(bio);\n--\nblock/blk-iolatency.c=583=static void blkcg_iolatency_done_bio(struct rq_qos rqos, struct bio bio)\n--\nblock/blk-iolatency.c-595-\nblock/blk-iolatency.c:596:\tblkg = bio_blkg_lookup(bio);\nblock/blk-iolatency.c-597-\tif (!blkg)\n--\nblock/blk-throttle.c=1792=bool __blk_throtl_bio(struct bio bio)\n--\nblock/blk-throttle.c-1794-\tstruct request_queue q = bdev_get_queue(bio-\u003ebi_bdev);\nblock/blk-throttle.c:1795:\tstruct blkcg_gq blkg = bio_blkg(bio);\nblock/blk-throttle.c-1796-\tstruct throtl_qnode qn = NULL;\n--\nblock/blk-throttle.h=166=static inline bool blk_should_throtl(struct bio *bio)\n--\nblock/blk-throttle.h-173-\nblock/blk-throttle.h:174:\ttg = blkg_to_tg(bio_blkg(bio));\nblock/blk-throttle.h-175-\tif (!cgroup_subsys_on_dfl(io_cgrp_subsys)) {\n" }
15/2	2026/08/18 08:05	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 28262 output: 28 thoughts: 449 Thoughts: Analyzing Block I/O Cgroup Function Usage Okay, so I'm looking at these exported functions related to block I/O cgroups. I see `bio_blkg`, `bio_blkg_lookup`, `bio_associate_blkcg`, `bio_associate_blkcg_from_css`, `bio_clone_blkcg_association`, and `bio_put_blkg_ref` are all exposed. That's the baseline. Now, I want to see which of these are not in a hot path, since those are the ones that would be more interesting to potentially optimize or refactor. I've identified a few that seem less performance-critical: `blkg_init_queue` during queue allocation, `blkcg_init_disk` during disk initialization, `blkg_conf_prep` when configuring policies, `blkcg_css_alloc`, `blkcg_css_online`, `blkcg_css_free` relating to cgroup creation and destruction, `blkg_destroy`, `blkg_release` which handles refcounts, and `blkcg_punt_bio_submit` for asynchronous bio submission. This list seems like a decent starting point. But wait, I need to double-check my assumptions. `bio_associate_blkcg_from_css` is on the list, but it's called by `bio_associate_blkcg`, and that is called by `bio_init` which is hot-path! Same with `bio_clone_blkcg_association`, which is called by `__bio_clone`. So, those are actually not good candidates. Hmmm, now I need to check `blkcg_punt_bio_submit`. I think that's invoked from `bio_endio` or something similar, and that would make that a hot-path. Gotta confirm that before I dig deeper.
16/2	2026/08/18 08:05	tool	grepper	0m Args: { "Expression": "blkcg_punt_bio_submit", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=225=static void blkcg_async_bio_workfn(struct work_struct work)\n--\nblock/blk-cgroup.c-254- /\nblock/blk-cgroup.c:255:void blkcg_punt_bio_submit(struct bio bio)\nblock/blk-cgroup.c-256-{\n--\nblock/blk-cgroup.c-268-}\nblock/blk-cgroup.c:269:EXPORT_SYMBOL_GPL(blkcg_punt_bio_submit);\nblock/blk-cgroup.c-270-\n--\nfs/btrfs/bio.c=495=static void btrfs_submit_dev_bio(struct btrfs_device dev, struct bio bio)\n--\nfs/btrfs/bio.c-535-\tif (bio-\u003ebi_opf \u0026 REQ_BTRFS_CGROUP_PUNT)\nfs/btrfs/bio.c:536:\t\tblkcg_punt_bio_submit(bio);\nfs/btrfs/bio.c-537-\telse\n--\nfs/btrfs/bio.h=122=void btrfs_bio_end_io(struct btrfs_bio bbio, blk_status_t status);\nfs/btrfs/bio.h-123-\nfs/btrfs/bio.h:124:/* Submit using blkcg_punt_bio_submit. /\nfs/btrfs/bio.h-125-#define REQ_BTRFS_CGROUP_PUNT\t\t\tREQ_FS_PRIVATE\n--\ninclude/linux/bio.h=549=void bio_put_blkg_ref(struct bio bio);\ninclude/linux/bio.h:550:void blkcg_punt_bio_submit(struct bio bio);\ninclude/linux/bio.h-551-#else\t/ CONFIG_BLK_CGROUP /\n--\ninclude/linux/bio.h=558=static inline void bio_put_blkg_ref(struct bio bio) { }\ninclude/linux/bio.h:559:static inline void blkcg_punt_bio_submit(struct bio *bio)\ninclude/linux/bio.h-560-{\n" }
17/2	2026/08/18 08:05	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 29725 output: 26 thoughts: 50 Thoughts: Here's my summary, as though I'm thinking through this: Analyzing Btrfs IO Submission Okay, so I'm looking at `blkcg_punt_bio_submit` and immediately, my brain flags it as something Btrfs-specific. It's not a general-purpose, high-frequency core function, but it is used for IO submission, which is critical. Given its role in IO submission, I need to understand its relationship to block I/O controller configuration, and where the resource management fits in. The next thing I need to investigate is `blkg_conf_prep`. I need to figure out how I need to configure the IO submission in this function.
18/2	2026/08/18 08:05	tool	grepper	0m Args: { "Expression": "blkg_conf_prep", "PathPrefix": "" } Results: { "Output": "block/bfq-cgroup.c=1063=static ssize_t bfq_io_set_device_weight(struct kernfs_open_file of,\n--\nblock/bfq-cgroup.c-1078-\nblock/bfq-cgroup.c:1079:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_bfq, \u0026ctx);\nblock/bfq-cgroup.c-1080-\tif (ret)\n--\nblock/blk-cgroup.c=770=EXPORT_SYMBOL_GPL(__blkg_prfill_u64);\n--\nblock/blk-cgroup.c-778- Once initialized, @ctx can be used with blkg_conf_open_bdev() and\nblock/blk-cgroup.c:779: * blkg_conf_prep().\nblock/blk-cgroup.c-780- /\n--\nblock/blk-cgroup.c=834=EXPORT_SYMBOL_GPL(blkg_conf_open_bdev);\n--\nblock/blk-cgroup.c-836-/\nblock/blk-cgroup.c:837: blkg_conf_prep - parse and prepare for per-blkg config update\nblock/blk-cgroup.c-838- * @blkcg: target block cgroup\n--\nblock/blk-cgroup.c-850- /\nblock/blk-cgroup.c:851:int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol,\nblock/blk-cgroup.c-852-\t\t struct blkg_conf_ctx ctx)\n--\nblock/blk-cgroup.c-944-}\nblock/blk-cgroup.c:945:EXPORT_SYMBOL_GPL(blkg_conf_prep);\nblock/blk-cgroup.c-946-\nblock/blk-cgroup.c-947-/*\nblock/blk-cgroup.c:948: blkg_conf_unprep - counterpart of blkg_conf_prep()\nblock/blk-cgroup.c-949- * @ctx: blkg_conf_ctx initialized with blkg_conf_init()\n--\nblock/blk-cgroup.h=230=int blkg_conf_open_bdev(struct blkg_conf_ctx ctx)\nblock/blk-cgroup.h-231-\t__cond_acquires(0, \u0026ctx-\u003ebdev-\u003ebd_queue-\u003erq_qos_mutex);\nblock/blk-cgroup.h:232:int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol,\nblock/blk-cgroup.h-233-\t\t struct blkg_conf_ctx ctx)\n--\nblock/blk-iocost.c=3149=static ssize_t ioc_weight_write(struct kernfs_open_file of, char buf,\n--\nblock/blk-iocost.c-3191-\nblock/blk-iocost.c:3192:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_iocost, \u0026ctx);\nblock/blk-iocost.c-3193-\tif (ret)\n--\nblock/blk-iolatency.c=830=static ssize_t iolatency_set_limit(struct kernfs_open_file of, char buf,\n--\nblock/blk-iolatency.c-857-\nblock/blk-iolatency.c:858:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_iolatency, \u0026ctx);\nblock/blk-iolatency.c-859-\tif (ret)\n--\nblock/blk-throttle.c=1352=static ssize_t tg_set_conf(struct kernfs_open_file of,\n--\nblock/blk-throttle.c-1372-\nblock/blk-throttle.c:1373:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_throtl, \u0026ctx);\nblock/blk-throttle.c-1374-\tif (ret)\n--\nblock/blk-throttle.c=1540=static ssize_t tg_set_limit(struct kernfs_open_file of,\n--\nblock/blk-throttle.c-1560-\nblock/blk-throttle.c:1561:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_throtl, \u0026ctx);\nblock/blk-throttle.c-1562-\tif (ret)\n" }
19/2	2026/08/18 08:05	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 31119 output: 26 thoughts: 194 Thoughts: Analyzing Block I/O Controller (blkcg) Initialization and Lifecycle Okay, so I'm thinking about the critical points for integrating new blkcg-related functionality. Let's break down where the key initialization, allocation, and destruction routines are called. `blkg_conf_prep` appears to be the go-to function for preparing things when configuring block I/O controller policies through sysfs or cgroupfs; this seems like a solid entry point for potential instrumentation or modifications. Similarly, `blkg_init_queue`, called from the core `blk_alloc_queue` function, will be important to observe how queue structures are initialized. `blkcg_init_disk`, initiated by `add_disk`, should be an equally important point of focus. Now, I'm zeroing in on the blkcg lifecycle itself. `blkcg_css_alloc` handles the creation of a new blkcg, so this is where resource initialization happens. Then, `blkcg_css_online` is triggered when a blkcg comes online. Finally, `blkcg_css_free` is the critical point for freeing a blkcg. We have to consider how these calls interact and make sure all resources are properly managed. Let's also not forget about `blkg_destroy`, which deals with the destruction of a blkg (block group?), and `blkg_release`, which handles refcount dropping to 0 for a blkg. These are all equally important to track. For now, I'm going to focus my attention specifically on `blkcg_css_alloc`, `blkcg_css_online`, and `blkcg_css_free`. These are going to be the most critical for understanding the lifecycle. I need to understand what's happening during these allocation/online/free phases.
20/2	2026/08/18 08:05	tool	grepper	0m Args: { "Expression": "blkcg_css_alloc", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=1390=static struct cgroup_subsys_state \nblock/blk-cgroup.c:1391:blkcg_css_alloc(struct cgroup_subsys_state parent_css)\nblock/blk-cgroup.c-1392-{\n--\nblock/blk-cgroup.c=1536=struct cgroup_subsys io_cgrp_subsys = {\nblock/blk-cgroup.c:1537:\t.css_alloc = blkcg_css_alloc,\nblock/blk-cgroup.c-1538-\t.css_online = blkcg_css_online,\n" }
21/2	2026/08/18 08:05	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 31903 output: 28 thoughts: 232 Thoughts: Analyzing Cgroup Subsystem Callbacks Okay, so I'm looking at these cgroup subsystem callbacks. I'm pretty sure they are definitely reachable but not in the hot-path. My first instinct is to check out `bio_blkg` and `bio_blkg_lookup`. Hmm, but `bio_blkg` is called from `bfq_bio_merge`, `bfq_insert_request`, `ioc_rqos_throttle`, `blkcg_iolatency_throttle`, and `__blk_throtl_bio`, which are all hot-path functions. Wait, though, this `bio_blkg` is new, replacing the direct `bio->bi_blkg` access, which is interesting. Maybe there's a non-hot-path function we should focus on? Let me see. I've got `blkg_conf_prep`, `blkcg_css_alloc`, `blkcg_css_free`, `blkg_init_queue`, `blkcg_init_disk`, `bio_associate_blkcg_from_css`, `bio_clone_blkcg_association`, and `bio_put_blkg_ref`. Right, I'll go with `bio_put_blkg_ref` for now.
22/2	2026/08/18 08:05	tool	grepper	0m Args: { "Expression": "bio_put_blkg_ref", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=2119=EXPORT_SYMBOL_GPL(bio_blkg_lookup);\n--\nblock/blk-cgroup.c-2121-/*\nblock/blk-cgroup.c:2122: bio_put_blkg_ref - drop the blkg reference pinned by a bio\nblock/blk-cgroup.c-2123- * @bio: target bio\n--\nblock/blk-cgroup.c-2126- /\nblock/blk-cgroup.c:2127:void bio_put_blkg_ref(struct bio bio)\nblock/blk-cgroup.c-2128-{\n--\nblock/blk-cgroup.c-2136-}\nblock/blk-cgroup.c:2137:EXPORT_SYMBOL_GPL(bio_put_blkg_ref);\nblock/blk-cgroup.c-2138-\n--\nblock/blk-cgroup.h=373=static inline void bio_clear_blkcg(struct bio bio)\n--\nblock/blk-cgroup.h-376-\nblock/blk-cgroup.h:377:\tbio_put_blkg_ref(bio);\nblock/blk-cgroup.h-378-\n--\ninclude/linux/bio.h=548=void bio_clone_blkcg_association(struct bio dst, struct bio src);\ninclude/linux/bio.h:549:void bio_put_blkg_ref(struct bio bio);\ninclude/linux/bio.h-550-void blkcg_punt_bio_submit(struct bio bio);\n--\ninclude/linux/bio.h=556=static inline void bio_clone_blkcg_association(struct bio dst,\ninclude/linux/bio.h-557-\t\t\t\t\t struct bio src) { }\ninclude/linux/bio.h:558:static inline void bio_put_blkg_ref(struct bio bio) { }\ninclude/linux/bio.h-559-static inline void blkcg_punt_bio_submit(struct bio bio)\n--\ninclude/linux/bio.h=565=static inline void bio_set_dev(struct bio bio, struct block_device *bdev)\n--\ninclude/linux/bio.h-568-\tif (bio-\u003ebi_bdev != bdev) {\ninclude/linux/bio.h:569:\t\tbio_put_blkg_ref(bio);\ninclude/linux/bio.h-570-\t\tbio_clear_flag(bio, BIO_BPS_THROTTLED);\n" }
23/2	2026/08/18 08:05	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 33098 output: 30 thoughts: 76 Thoughts: Here's my summary, as I see it: Thinking Through Block I/O Context Management Okay, so I'm looking at how block I/O context is being managed. `bio_put_blkg_ref` is called from two key places: `bio_clear_blkcg` and `bio_set_dev`. Both of these are important, but `bio_clear_blkcg` gets called from `bio_uninit` and `bio_endio`! Hmm, now that definitely raises a flag for performance sensitivity. Those are very hot paths. I'm starting to think about where the performance is. The real question I should be asking is: what about `bio_associate_blkcg_from_css`? Where does that fit into the picture? What's the relative frequency of that call versus the ones I've just highlighted? Is there something in the association and deassociation of the `blkcg` that is creating a performance bottleneck? I'll need to dig into the call graphs and analyze the code around `bio_associate_blkcg_from_css` carefully. That's the likely weak spot.
24/2	2026/08/18 08:05	tool	grepper	0m Args: { "Expression": "bio_associate_blkcg_from_css", "PathPrefix": "" } Results: { "Output": "block/bfq-cgroup.c=609=struct bfq_group bfq_bio_bfqg(struct bfq_data bfqd, struct bio bio)\n--\nblock/bfq-cgroup.c-621-\t\tif (bfqg-\u003epd.online) {\nblock/bfq-cgroup.c:622:\t\t\tbio_associate_blkcg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\nblock/bfq-cgroup.c-623-\t\t\treturn bfqg;\n--\nblock/bfq-cgroup.c-628-\tblkg = bfqg_to_blkg(bfqd-\u003eroot_group);\nblock/bfq-cgroup.c:629:\tbio_associate_blkcg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\nblock/bfq-cgroup.c-630-\treturn bfqd-\u003eroot_group;\n--\nblock/blk-cgroup.c=2171=EXPORT_SYMBOL_GPL(bio_blkg);\n--\nblock/blk-cgroup.c-2173-/\nblock/blk-cgroup.c:2174: bio_associate_blkcg_from_css - associate a bio with a specified css\nblock/blk-cgroup.c-2175- * @bio: target bio\n--\nblock/blk-cgroup.c-2184- /\nblock/blk-cgroup.c:2185:void bio_associate_blkcg_from_css(struct bio bio,\nblock/blk-cgroup.c-2186-\t\t\t\t struct cgroup_subsys_state css)\n--\nblock/blk-cgroup.c-2203-}\nblock/blk-cgroup.c:2204:EXPORT_SYMBOL_GPL(bio_associate_blkcg_from_css);\nblock/blk-cgroup.c-2205-\n--\nblock/blk-cgroup.c=2213=void bio_associate_blkcg(struct bio bio)\n--\nblock/blk-cgroup.c-2221-\trcu_read_lock();\nblock/blk-cgroup.c:2222:\tbio_associate_blkcg_from_css(bio, blkcg_css());\nblock/blk-cgroup.c-2223-\trcu_read_unlock();\n--\nblock/blk-cgroup.c=2232=void bio_clone_blkcg_association(struct bio dst, struct bio src)\n--\nblock/blk-cgroup.c-2234-\tif (bio_blkcg(src))\nblock/blk-cgroup.c:2235:\t\tbio_associate_blkcg_from_css(dst, bio_blkcg_css(src));\nblock/blk-cgroup.c-2236-}\n--\ninclude/linux/bio.h=545=void bio_associate_blkcg(struct bio bio);\ninclude/linux/bio.h:546:void bio_associate_blkcg_from_css(struct bio bio,\ninclude/linux/bio.h-547-\t\t\t\t struct cgroup_subsys_state css);\n--\ninclude/linux/bio.h=552=static inline void bio_associate_blkcg(struct bio bio) { }\ninclude/linux/bio.h:553:static inline void bio_associate_blkcg_from_css(struct bio bio,\ninclude/linux/bio.h-554-\t\t\t\t\t\tstruct cgroup_subsys_state css)\n--\ninclude/linux/writeback.h=256=static inline void wbc_init_bio(struct writeback_control wbc, struct bio bio)\n--\ninclude/linux/writeback.h-264-\tif (wbc-\u003ewb)\ninclude/linux/writeback.h:265:\t\tbio_associate_blkcg_from_css(bio, wbc-\u003ewb-\u003eblkcg_css);\ninclude/linux/writeback.h-266-}\n--\nmm/page_io.c=319=static void bio_associate_blkcg_from_page(struct bio bio, struct folio folio)\n--\nmm/page_io.c-333-\nmm/page_io.c:334:\tbio_associate_blkcg_from_css(bio, css);\nmm/page_io.c-335-\tif (css)\n" }
25/2	2026/08/18 08:05	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 34564 output: 26 thoughts: 265 Thoughts: Analyzing Block I/O Cgroup Hotspots Okay, so I'm digging into the block I/O cgroup code and need to identify potential performance bottlenecks. `bio_associate_blkcg_from_css` is clearly in the hot path, given its calls from `bio_associate_blkcg`, `bio_clone_blkcg_association`, `wbc_init_bio`, `bio_associate_blkcg_from_page`, and `bfq_bio_bfqg`. That's a lot of important places. I should definitely keep an eye on that. Now, what about the other functions? Let's see... `blkg_conf_prep` is called from sysfs and cgroupfs writes. That could be interesting, perhaps something we could optimize via `FocusSymbols`. `blkg_init_queue` is part of `blk_alloc_queue`, which could also be a performance critical piece, but might not be as hot as the I/O path. `blkcg_init_disk` is triggered by `add_disk`, and `blkcg_css_alloc`, `blkcg_css_online`, and `blkcg_css_free` are involved in the lifecycle of block I/O cgroups, so that's something to keep in mind, and the `blkg_destroy` and `blkg_release` functions will be important when cleaning up. For now, `blkg_conf_prep` looks like a prime candidate for further investigation. Sysfs/cgroupfs interaction is often a point of contention. I should also take a look at `blkg_lookup_create`.
26/2	2026/08/18 08:05	tool	grepper	0m Args: { "Expression": "blkg_lookup_create", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=474=static struct blkcg_gq blkg_lookup_tryget(struct blkcg_gq blkg)\n--\nblock/blk-cgroup.c-481-/*\nblock/blk-cgroup.c:482: blkg_lookup_create - lookup blkg, try to create one if not there\nblock/blk-cgroup.c-483- * @blkcg: blkcg of interest\n--\nblock/blk-cgroup.c-495- /\nblock/blk-cgroup.c:496:static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg,\nblock/blk-cgroup.c-497-\t\tstruct gendisk disk)\n--\nblock/blk-cgroup.c=2148=struct blkcg_gq bio_blkg(struct bio bio)\n--\nblock/blk-cgroup.c-2164-\tspin_lock_irq(\u0026q-\u003equeue_lock);\nblock/blk-cgroup.c:2165:\tblkg = blkg_lookup_create(blkcg, disk);\nblock/blk-cgroup.c-2166-\tspin_unlock_irq(\u0026q-\u003equeue_lock);\n" }
27/2	2026/08/18 08:05	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 35538 output: 26 thoughts: 263 Thoughts: Analyzing the Block Group Lookup Okay, so `blkg_lookup_create` is called from `bio_blkg`. And I know `bio_blkg` is in the hot-path, which immediately flags it as something to pay attention to. Wait a minute, I have to remember, `bio_blkg` handles the creation of a blkg for a bio, so `bio_blkg` is called for every bio that needs a blkg. I have to make sure I am thinking this through correctly because it's a very fast execution path. So, `blkg_lookup_create` is only called if `bio_flagged(bio, BIO_BLKG_REF)` is false. Ah, right! That means it's only called the first time a bio needs a blkg. The subsequent calls for that bio shouldn't be calling this function. Okay, now I have it. `bio_blkg` is called when `bio_flagged(bio, BIO_BLKG_REF)` is false, meaning it is called for the first IO of a bio that needs a blkg. But, is `bio_blkg` hot-path? I have to confirm. Let me see... yes, it's called in several key places: `bfq_bio_merge`, `bfq_insert_request`, `ioc_rqos_throttle`, `blkcg_iolatency_throttle`, and `__blk_throtl_bio`, if the bio doesn't already have `BIO_BLKG_REF` set. That reinforces that is definitely in the hot-path! Wait, `bio_blkg` sets `BIO_BLKG_REF` and pins the blkg, so that's why it is only called once per bio. That is a crucial optimization. Okay, it's still hot-path, but at least the number of lookups is reduced. Now, I think I need to look at `blkg_init_queue` and `blkcg_init_disk` to understand the initialization side better.
28/2	2026/08/18 08:05	tool	grepper	0m Args: { "Expression": "blkg_init_queue", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=1460=static int blkcg_css_online(struct cgroup_subsys_state css)\n--\nblock/blk-cgroup.c-1473-\nblock/blk-cgroup.c:1474:int blkg_init_queue(struct request_queue q)\nblock/blk-cgroup.c-1475-{\n--\nblock/blk-cgroup.h=200=extern const struct rhashtable_params blkg_hash_params;\nblock/blk-cgroup.h-201-\nblock/blk-cgroup.h:202:int blkg_init_queue(struct request_queue q);\nblock/blk-cgroup.h-203-void blkg_exit_queue(struct request_queue q);\n--\nblock/blk-cgroup.h=544=static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg, void key) { return NULL; }\nblock/blk-cgroup.h:545:static inline int blkg_init_queue(struct request_queue q) { return 0; }\nblock/blk-cgroup.h-546-static inline void blkg_exit_queue(struct request_queue q) { }\n--\nblock/blk-core.c=437=struct request_queue blk_alloc_queue(struct queue_limits *lim, int node_id)\n--\nblock/blk-core.c-484-\nblock/blk-core.c:485:\terror = blkg_init_queue(q);\nblock/blk-core.c-486-\tif (error)\n" }
29/2	2026/08/18 08:05	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 36708 output: 26 thoughts: 38 Thoughts: Here's my understanding, as if these were my internal thoughts: Analyzing Block Device Initialization Okay, so I'm looking at the block device initialization flow. It seems pretty clear that the `blkg_init_queue` function is called directly from `blk_alloc_queue`. That makes sense; we're initializing the queue structure, likely related to I/O accounting at the group level, right as the queue is being allocated. I'd need to dig deeper into the code of `blk_alloc_queue` to understand the conditions triggering the call. Now, the connection between `blkcg_init_disk` and `add_disk` is the more interesting piece. The comments state that `blkcg_init_disk` gets called from `add_disk`. This strongly suggests that we're dealing with initialization related to block cgroups whenever a new disk is added to the system. This seems correct. It likely sets up the cgroup integration points within the disk structure itself, so that subsequent I/O on that disk can be properly attributed to the right control groups. It’s also entirely possible that disk addition is what triggers the initialization of the disk-specific control group data, which would make sense. I'll need to confirm the conditions for the call.
30/2	2026/08/18 08:06	tool	grepper	0m Args: { "Expression": "blkcg_init_disk", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=1481=void blkg_exit_queue(struct request_queue q)\n--\nblock/blk-cgroup.c-1485-\nblock/blk-cgroup.c:1486:int blkcg_init_disk(struct gendisk disk)\nblock/blk-cgroup.c-1487-{\n--\nblock/blk-cgroup.h=203=void blkg_exit_queue(struct request_queue q);\nblock/blk-cgroup.h:204:int blkcg_init_disk(struct gendisk disk);\nblock/blk-cgroup.h-205-void blkcg_exit_disk(struct gendisk disk);\n--\nblock/blk-cgroup.h=546=static inline void blkg_exit_queue(struct request_queue q) { }\nblock/blk-cgroup.h:547:static inline int blkcg_init_disk(struct gendisk disk) { return 0; }\nblock/blk-cgroup.h-548-static inline void blkcg_exit_disk(struct gendisk disk) { }\n--\nblock/genhd.c=1447=struct gendisk __alloc_disk_node(struct request_queue q, int node_id,\n--\nblock/genhd.c-1475-\nblock/genhd.c:1476:\tif (blkcg_init_disk(disk))\nblock/genhd.c-1477-\t\tgoto out_erase_part0;\n" }
31/2	2026/08/18 08:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 37442 output: 24 thoughts: 260 Thoughts: Analyzing Block IO CGroups Initialization and Destruction Okay, so I'm looking at these block I/O cgroup initialization and destruction paths. `blkcg_init_disk` gets called from `__alloc_disk_node`, which is definitely not in a performance-critical area, so that's good. Now, let's see, `blkg_conf_prep` is the next one to investigate, which is called from sysfs and cgroupfs writes. Then we have `blkcg_css_alloc`, that's happening whenever a new blkcg is created, `blkcg_css_online` onlining and `blkcg_css_free` when freeing. Finally, `blkg_destroy` is called upon destroy and `blkg_release` when the refcount drops to zero. Based on that analysis, the non-hot-path functions that warrant further examination are: `blkg_conf_prep`, `blkcg_css_alloc`, `blkcg_css_online`, `blkcg_css_free`, `blkg_init_queue`, `blkcg_init_disk`, `blkg_destroy`, and `blkg_release`. Hang on a second… weren't `blkg_destroy` and `blkg_release` static? I should probably double-check that. Static means, potentially, they are internal to a particular module, and we might need to think about that implication. I'll make sure to verify that.
32/2	2026/08/18 08:06	tool	grepper	0m Args: { "Expression": "blkg_destroy", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=124=static void blkg_free_workfn(struct work_struct work)\n--\nblock/blk-cgroup.c-133-\t in order to make sure pd_free_fn() is called in order, the deletion\nblock/blk-cgroup.c:134:\t * of the list blkg-\u003eq_node is delayed to here from blkg_destroy(), and\nblock/blk-cgroup.c-135-\t * blkcg_mutex is used to synchronize blkg_free_workfn() and\n--\nblock/blk-cgroup.c=496=static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg,\n--\nblock/blk-cgroup.c-545-\nblock/blk-cgroup.c:546:static void blkg_destroy(struct blkcg_gq blkg)\nblock/blk-cgroup.c-547-{\n--\nblock/blk-cgroup.c-556-\t blkg_free_workfn(), hence this function can be called from\nblock/blk-cgroup.c:557:\t * blkcg_destroy_blkgs() first and again from blkg_destroy_all() before\nblock/blk-cgroup.c-558-\t * blkg_free_workfn().\n--\nblock/blk-cgroup.c-583-\nblock/blk-cgroup.c:584:static void blkg_destroy_all(struct gendisk disk)\nblock/blk-cgroup.c-585-{\n--\nblock/blk-cgroup.c-600-\t\tspin_lock(\u0026blkcg-\u003elock);\nblock/blk-cgroup.c:601:\t\tblkg_destroy(blkg);\nblock/blk-cgroup.c-602-\t\tspin_unlock(\u0026blkcg-\u003elock);\n--\nblock/blk-cgroup.c=1281=static void blkcg_destroy_blkgs(struct blkcg blkcg)\n--\nblock/blk-cgroup.c-1292-\nblock/blk-cgroup.c:1293:\t\tblkg_destroy(blkg);\nblock/blk-cgroup.c-1294-\n--\nblock/blk-cgroup.c=1523=void blkcg_exit_disk(struct gendisk disk)\nblock/blk-cgroup.c-1524-{\nblock/blk-cgroup.c:1525:\tblkg_destroy_all(disk);\nblock/blk-cgroup.c-1526-\tblk_throtl_exit(disk);\n--\nblock/blk-throttle.c=1886=void blk_throtl_exit(struct gendisk disk)\n--\nblock/blk-throttle.c-1890-\t/\nblock/blk-throttle.c:1891:\t blkg_destroy_all() already deactivate throtl policy, just check and\nblock/blk-throttle.c-1892-\t * free throtl data.\n" }
33/2	2026/08/18 08:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 38814 output: 26 thoughts: 64 Thoughts: Okay, here's my take on those functions. Static Functions in the Block I/O Control Group Implementation Alright, let's see. So, `blkg_destroy` is definitely static, yep. And `blkg_release`, that's static as well, no question. Okay, moving on... `blkcg_css_alloc`, `blkcg_css_online`, and `blkcg_css_free` are all static too. Hmm, wait a minute... I have to double-check that `blkcg_css_alloc` is static. I could have sworn it was static. Just want to make absolutely sure before I commit that to memory. This is foundational to the concurrency design, so I need to get this right.
34/2	2026/08/18 08:06	tool	grepper	0m Args: { "Expression": "blkcg_css_alloc", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=1390=static struct cgroup_subsys_state \nblock/blk-cgroup.c:1391:blkcg_css_alloc(struct cgroup_subsys_state parent_css)\nblock/blk-cgroup.c-1392-{\n--\nblock/blk-cgroup.c=1536=struct cgroup_subsys io_cgrp_subsys = {\nblock/blk-cgroup.c:1537:\t.css_alloc = blkcg_css_alloc,\nblock/blk-cgroup.c-1538-\t.css_online = blkcg_css_online,\n" }
35/2	2026/08/18 08:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 39242 output: 34 thoughts: 340 Thoughts: Analyzing Block I/O Control Group Function Calls Okay, so I'm looking at these functions, and it's clear the static ones aren't the focus here. The non-static ones are what matter. Right now, I'm thinking about `blkg_conf_prep`, `blkg_init_queue`, `blkcg_init_disk`, `bio_associate_blkcg_from_css`, `bio_associate_blkcg`, `bio_clone_blkcg_association`, `bio_put_blkg_ref`, `bio_blkg`, and `bio_blkg_lookup`. Hang on, `bio_blkg` and `bio_blkg_lookup` are exported, which means they're used outside this module and therefore probably critical. The question now is whether they're in the hot-path. Since `bio_blkg` and `bio_blkg_lookup` are called for every bio that needs a blkg, yeah, they're definitely hot-path functions. Now, `bio_associate_blkcg_from_css` is called for every bio that needs a blkcg, so, yes, hot-path. `bio_associate_blkcg` is called for every bio... that's also hot-path. And `bio_clone_blkcg_association`, since it's called for every cloned bio? Yep, that's hot-path too. And `bio_put_blkg_ref` is called for every bio that is freed or remapped. So, again, it's hot-path. That leaves the non-hot-path functions: `blkg_conf_prep`, `blkg_init_queue`, and `blkcg_init_disk`. Now, let me just double check. Is `blkg_conf_prep` exported? It's important to be thorough.
36/2	2026/08/18 08:06	tool	grepper	0m Args: { "Expression": "EXPORT_SYMBOL_GPL\\(blkg_conf_prep\\)", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=851=int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol,\n--\nblock/blk-cgroup.c-944-}\nblock/blk-cgroup.c:945:EXPORT_SYMBOL_GPL(blkg_conf_prep);\nblock/blk-cgroup.c-946-\n" }
37/2	2026/08/18 08:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 40140 output: 34 thoughts: 18 Thoughts: Okay, here's my summary, in the first person, as requested, focusing on my internal thought process: Exported Functions: Quick Check Right, so the code indicates that something is definitely exported. My immediate thought is, "Yes, it is exported." That much is clear. Now, to the more interesting point, a critical function for managing a block queue, `blkg_init_queue`, is that also exported? I need to know for sure. It's a key piece in many designs. This requires a quick check to confirm access.
38/2	2026/08/18 08:06	tool	grepper	0m Error: no matches Args: { "Expression": "EXPORT_SYMBOL_GPL\\(blkg_init_queue\\)", "PathPrefix": "" } Results: { "error": "no matches" }
39/2	2026/08/18 08:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 40325 output: 34 thoughts: 27 Thoughts: Okay, here's my summary, presented as a stream of thought: Checking the Export Status and Function Availability Wait a second, the item in question isn't explicitly exported, but that doesn't necessarily mean it's entirely static either. Okay, so that means I need to dig deeper. Let's see... the core question I really need to answer now is: is `blkcg_init_disk` actually exposed? Is it even exported for use elsewhere? That’s the key. Knowing that will tell me a lot about how I should be handling this.
40/2	2026/08/18 08:06	tool	grepper	0m Error: no matches Args: { "Expression": "EXPORT_SYMBOL_GPL\\(blkcg_init_disk\\)", "PathPrefix": "" } Results: { "error": "no matches" }
41/2	2026/08/18 08:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 40524 output: 36 thoughts: 29 Thoughts: Okay, here's my interpretation of those thoughts, tailored for an expert audience: Analyzing the Dynamic Nature of the System No, this isn't a simple, fixed situation. My mind immediately jumps to the dynamic aspects of the problem. I need to understand what's changing here. Is it an issue with data that can change, or the way the data changes? Let's dig into a particular function to see what we can find, and to build context around this problem. I'll start with `blkcg_punt_bio_submit`. I seem to recall that one being exported. It's a key entry point, and that usually means it's pretty well-understood and documented, making it a good place to start my analysis.
42/2	2026/08/18 08:06	tool	grepper	0m Args: { "Expression": "EXPORT_SYMBOL_GPL\\(blkcg_punt_bio_submit\\)", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=255=void blkcg_punt_bio_submit(struct bio *bio)\n--\nblock/blk-cgroup.c-268-}\nblock/blk-cgroup.c:269:EXPORT_SYMBOL_GPL(blkcg_punt_bio_submit);\nblock/blk-cgroup.c-270-\n" }
43/2	2026/08/18 08:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 40855 output: 28 thoughts: 64 Thoughts: Here's my take on those thoughts: Analyzing a Specific Block CGROUP Call Okay, so I'm looking at this call, and my initial thought is, "Is this a performance-critical path? Is it truly a 'hot-path' operation?" The context is a bit unusual; it's being called from btrfs specifically for certain BIOS interactions (`REQ_BTRFS_CGROUP_PUNT`). That's not your typical, widely-used, performance-sensitive code branch. That's good to keep in mind, and that the function itself, is not generic. To be sure, I'll need to dig deeper. I need to understand what this code is actually doing. So, next up I should really look at `blkcg_set_fc_appid` and `blkcg_get_fc_appid`. Those will show me what parameters are set and used. That will allow me to check the data flow, and determine if my initial assumption is correct, or if there is something else going on here.
44/2	2026/08/18 08:06	tool	grepper	0m Args: { "Expression": "blkcg_get_fc_appid", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup-fc-appid.c=43=EXPORT_SYMBOL_GPL(blkcg_set_fc_appid);\n--\nblock/blk-cgroup-fc-appid.c-45-/*\nblock/blk-cgroup-fc-appid.c:46: blkcg_get_fc_appid - get the fc app identifier associated with a bio\nblock/blk-cgroup-fc-appid.c-47- * @bio: target bio\n--\nblock/blk-cgroup-fc-appid.c-50- /\nblock/blk-cgroup-fc-appid.c:51:char blkcg_get_fc_appid(struct bio bio)\nblock/blk-cgroup-fc-appid.c-52-{\n--\nblock/blk-cgroup-fc-appid.c-62-}\nblock/blk-cgroup-fc-appid.c:63:EXPORT_SYMBOL_GPL(blkcg_get_fc_appid);\n--\ndrivers/nvme/host/fc.c=1884=char nvme_fc_io_getuuid(struct nvmefc_fcp_req req)\n--\ndrivers/nvme/host/fc.c-1890-\t\treturn NULL;\ndrivers/nvme/host/fc.c:1891:\treturn blkcg_get_fc_appid(rq-\u003ebio);\ndrivers/nvme/host/fc.c-1892-}\n--\ndrivers/scsi/lpfc/lpfc_scsi.c=5213=static char lpfc_is_command_vm_io(struct scsi_cmnd cmd)\n--\ndrivers/scsi/lpfc/lpfc_scsi.c-5218-\t\treturn NULL;\ndrivers/scsi/lpfc/lpfc_scsi.c:5219:\treturn blkcg_get_fc_appid(bio);\ndrivers/scsi/lpfc/lpfc_scsi.c-5220-}\n--\ninclude/linux/blk-cgroup.h=69=int blkcg_set_fc_appid(char app_id, u64 cgrp_id, size_t app_id_len);\ninclude/linux/blk-cgroup.h:70:char blkcg_get_fc_appid(struct bio bio);\ninclude/linux/blk-cgroup.h-71-\n" }
45/2	2026/08/18 08:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 41766 output: 26 thoughts: 216 Thoughts: Analyzing the `blkcg_get_fc_appid` Modification Okay, so I'm looking at this patch and the first thing I notice is the change to `blkcg_get_fc_appid`. It's a straightforward tweak, but the implications are important. The original code had a rather verbose check that combined NULL checks of `bio->bi_blkg` and a null-string check of `fc_app_id`. This has been streamlined by fetching `blkcg` first with `bio_blkcg(bio)` and then proceeding to check its `fc_app_id` directly. The change is conceptually simple: make a local variable and simplify the if statement. I see that this function, based on the context provided, is called from the nvme/fc and lpfc drivers. Now, while this isn't a generic hot-path, it is a hot-path for those specific Fibre Channel NVMe drivers, so even minor optimizations here can matter. Since this is in the storage layers it is possible this could be on the hot-path and therefore it is good to have more efficient code. Let's check the code for `bio_blkcg_css`. I want to make sure the blkcg fetch is still correct and performs as expected.
46/2	2026/08/18 08:06	tool	grepper	0m Args: { "Expression": "bio_blkcg_css", "PathPrefix": "" } Results: { "Output": "block/blk-cgroup.c=280=subsys_initcall(blkcg_punt_bio_init);\n--\nblock/blk-cgroup.c-283-/*\nblock/blk-cgroup.c:284: bio_blkcg_css - return the blkcg CSS associated with a bio\nblock/blk-cgroup.c-285- * @bio: target bio\n--\nblock/blk-cgroup.c-290- /\nblock/blk-cgroup.c:291:struct cgroup_subsys_state bio_blkcg_css(struct bio bio)\nblock/blk-cgroup.c-292-{\n--\nblock/blk-cgroup.c-300-}\nblock/blk-cgroup.c:301:EXPORT_SYMBOL_GPL(bio_blkcg_css);\nblock/blk-cgroup.c-302-\n--\nblock/blk-cgroup.c=2232=void bio_clone_blkcg_association(struct bio dst, struct bio src)\n--\nblock/blk-cgroup.c-2234-\tif (bio_blkcg(src))\nblock/blk-cgroup.c:2235:\t\tbio_associate_blkcg_from_css(dst, bio_blkcg_css(src));\nblock/blk-cgroup.c-2236-}\n--\nblock/blk-cgroup.h=237=void blkg_conf_close_bdev(struct blkg_conf_ctx ctx)\n--\nblock/blk-cgroup.h-247- * it were attached to the root blkg, and then backcharge to the actual owning\nblock/blk-cgroup.h:248: * blkg. The idea is we do bio_blkcg_css() to look up the actual context for\nblock/blk-cgroup.h-249- * the bio and attach the appropriate blkg to the bio. Then we call this helper\n--\ndrivers/block/loop.c=1873=static blk_status_t loop_queue_rq(struct blk_mq_hw_ctx hctx,\n--\ndrivers/block/loop.c-1900-\tif (rq-\u003ebio) {\ndrivers/block/loop.c:1901:\t\tcmd-\u003eblkcg_css = bio_blkcg_css(rq-\u003ebio);\ndrivers/block/loop.c-1902-#ifdef CONFIG_MEMCG\n--\ninclude/linux/blk-cgroup.h=54=struct list_head blkcg_get_cgwb_list(struct cgroup_subsys_state css);\ninclude/linux/blk-cgroup.h:55:struct cgroup_subsys_state bio_blkcg_css(struct bio bio);\ninclude/linux/blk-cgroup.h-56-\n--\ninclude/linux/blk-cgroup.h=62=static inline bool blk_cgroup_congested(void) { return false; }\ninclude/linux/blk-cgroup.h:63:static inline struct cgroup_subsys_state bio_blkcg_css(struct bio bio)\ninclude/linux/blk-cgroup.h-64-{\n--\nkernel/trace/blktrace.c=974=static u64 blk_trace_bio_get_cgid(struct request_queue q, struct bio *bio)\n--\nkernel/trace/blktrace.c-983-\nkernel/trace/blktrace.c:984:\tblkcg_css = bio_blkcg_css(bio);\nkernel/trace/blktrace.c-985-\tif (!blkcg_css)\n" }
47/2	2026/08/18 08:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 43196 output: 32 thoughts: 1317 Thoughts: Focusing on Block I/O Cgroup Functions Okay, let's see, my goal here is to identify good candidates for further investigation within the block I/O cgroup system. I'm looking for functions that are not in the hot path, but are still significant and preferably non-static, non-inlined API entrypoints. The initial list includes several interesting functions: `blkg_conf_prep`, `blkcg_punt_bio_submit`, `bio_associate_blkcg_from_css`, `bio_clone_blkcg_association`, `bio_blkg`, and `bio_blkg_lookup`. Let me think. `bio_blkg` and `bio_blkg_lookup` are definitely hot-path functions, so those are out, based on the requirements. Now, the instructions I've been given are pretty specific about avoiding generic hot-path stuff to prevent test bias. Okay, let's look at the remaining candidates, digging in further. `bio_associate_blkcg_from_css` is called from `bio_associate_blkcg` which itself is called from `bio_init`, which is hot-path. So, no. `bio_clone_blkcg_association` is called from `__bio_clone`, another hot path. So, also no. I should probably avoid `bio_put_blkg_ref` as well if it's hot path, and let's assume it is. Now, considering the remaining functions, I'm thinking about `blkcg_punt_bio_submit`. This one seems promising because it's called from `btrfs endio` which is a hot-path for btrfs. However, `blkcg_punt_bio_submit` queues a work item, so it's likely not hot-path on its own, given that. I need to make sure I'm getting the right level of abstraction here. So, yes, that one is a strong possibility. I'll need to check the code. Alright, what about `blkg_conf_prep`? It's called from sysfs writes, so it's not in the hot-path and is exported. That's a good sign. Let's keep that on the list. Now, what about `blkg_init_queue`? It's called from `blk_alloc_queue`, which I think is not hot-path. And `blkcg_init_disk` gets called from `add_disk`, which is not hot-path either. Neither of those is static, so they're in the running. That's good. But are those exported? No, they're not. So I will have to be mindful of that limitation. But, it is okay since it's an internal function. Let me just quickly check `blkcg_css_alloc`, `blkcg_css_online`, and `blkcg_css_free` – those are all static, so no. Not a candidate. Okay, is anything else interesting? I should probably check any modifications, though. Hmm, `blkcg_set_ioprio` seems to just be using `bio_blkcg(bio)`, which is something I should keep in mind. And, `blkcg_get_fc_appid` is modified, but that's hot-path for fc. So, no. Back to `blkcg_punt_bio_submit`. Examining the code, it looks like it's called from `btrfs_submit_dev_bio` if `REQ_BTRFS_CGROUP_PUNT` is set. It queues a work item, so that's good in terms of avoiding the hot-path concern. That's solid. Okay, let's re-evaluate. I think these are my best candidates so far: `blkcg_punt_bio_submit` and `blkg_conf_prep`. These are the non-hot-path, exported functions. The others I found are a bit too hot-path for this exercise.
48/2	2026/08/18 08:07	tool	grepper	0m Args: { "Expression": "config BLK_CGROUP_PUNT_BIO", "PathPrefix": "" } Results: { "Output": "block/Kconfig=42=config BLK_CGROUP_RWSTAT\n--\nblock/Kconfig-44-\nblock/Kconfig:45:config BLK_CGROUP_PUNT_BIO\nblock/Kconfig-46-\tbool\n" }
49/2	2026/08/18 08:07	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 45498 output: 32 thoughts: 18 Thoughts: Here's a summary of my thinking, presented as if it were my thought process: Decoding the Configuration Selection Okay, so I'm looking at this... and it's definitely a hidden configuration option. The key giveaway is that it's selected – meaning it's dependent on something else being true. The trigger, the parent setting, the thing that makes this specific configuration active, is BTRFS_FS. So, the presence of BTRFS_FS in the overall config file is what's telling the system to include this, whatever it is. I need to now figure out exactly what this hidden config entails and why BTRFS_FS is the key. Perhaps it's some sort of optimization or feature specific to that filesystem? The fact it's hidden implies it's either an internal detail, a rarely-used option, or something that shouldn't typically be tweaked directly by users.
50/2	2026/08/18 08:07	tool	grepper	0m Args: { "Expression": "select BLK_CGROUP_PUNT_BIO", "PathPrefix": "" } Results: { "Output": "fs/btrfs/Kconfig=3=config BTRFS_FS\nfs/btrfs/Kconfig-4-\ttristate \"Btrfs filesystem support\"\nfs/btrfs/Kconfig:5:\tselect BLK_CGROUP_PUNT_BIO\nfs/btrfs/Kconfig-6-\tselect CRC32\n" }
51/2	2026/08/18 08:07	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 45840 output: 30 thoughts: 258 Thoughts: My Configuration Quandary Okay, so the initial thought was to explicitly enable `BLK_CGROUP_PUNT_BIO` since it seems relevant. But hang on, it appears that `BLK_CGROUP_PUNT_BIO` is actually selected by `BTRFS_FS`. So, if I enable `BTRFS_FS`, `BLK_CGROUP_PUNT_BIO` should automatically be included, right? But then I thought, maybe I should just list `BLK_CGROUP_PUNT_BIO` directly? Hmm, no, the instructions are very clear: list it in the output array without the `CONFIG_` prefix. However, if `BLK_CGROUP_PUNT_BIO` is a hidden config, there's no way to enable it directly in the configuration menu. Listing both `BTRFS_FS` and `BLK_CGROUP_PUNT_BIO` seemed like the safer bet. Then I realized something crucial: `BLK_CGROUP_PUNT_BIO` is a boolean without a prompt. This means it can't be enabled directly in a menuconfig session. But I remember that syzkaller can enable it by simply including it in the configuration file, right? Actually, hold on. I remember that syzkaller uses `make olddefconfig`. With `make olddefconfig`, hidden configs cannot be directly enabled unless their dependencies or selects are already met. Therefore, the most reliable and correct approach is to simply include `BTRFS_FS`. It's the dependency, and that should be enough. Now, let me check the status of `BLK_CGROUP_IOCOST`...
52/2	2026/08/18 08:07	tool	grepper	0m Args: { "Expression": "config BLK_CGROUP_IOCOST", "PathPrefix": "" } Results: { "Output": "block/Kconfig=145=config BLK_CGROUP_FC_APPID\n--\nblock/Kconfig-153-\nblock/Kconfig:154:config BLK_CGROUP_IOCOST\nblock/Kconfig-155-\tbool \"Enable support for cost model based cgroup IO controller\"\n" }
53/2	2026/08/18 08:07	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 46587 output: 27 thoughts: 210 Thoughts: My Configuration Strategy Okay, so I have a set of configurations I need to enable based on prompts. I see that `BLK_CGROUP`, `BFQ_GROUP_IOSCHED`, `BLK_CGROUP_IOCOST`, `BLK_CGROUP_IOLATENCY`, `BLK_CGROUP_IOPRIO`, and `BLK_CGROUP_FC_APPID` all have prompts, so they're straightforward enables. And, of course, `BTRFS_FS` is on the list, too. However, let me just quickly confirm something about `BFQ_GROUP_IOSCHED` before I finalize this. Ah, yes, it depends on `IOSCHED_BFQ`. That's important! So I'll definitely need to include `IOSCHED_BFQ` in my final configuration as well to make sure `BFQ_GROUP_IOSCHED` functions correctly.
54/2	2026/08/18 08:07	tool	grepper	0m Args: { "Expression": "config IOSCHED_BFQ", "PathPrefix": "" } Results: { "Output": "block/Kconfig.iosched=10=config MQ_IOSCHED_KYBER\n--\nblock/Kconfig.iosched-18-\nblock/Kconfig.iosched:19:config IOSCHED_BFQ\nblock/Kconfig.iosched-20-\ttristate \"BFQ I/O scheduler\"\n" }
55/2	2026/08/18 08:07	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 47117 output: 32 thoughts: 58 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience: Analyzing IO Schedulers and FC Support Alright, so `IOSCHED_BFQ` definitely has a prompt, got it. Now, let's pivot to `BLK_CGROUP_FC_APPID`. My initial thought is to quickly verify its dependencies. Does it rely on `NVME_FC` or `SCSI_LPFC`, like I'd expect for related functionalities? Nope, it's just a straight boolean. So, it’s a pretty simple flag, probably enabling or disabling a core feature. Good to know.
56/2	2026/08/18 08:07	tool	read-file	0m Args: { "File": "block/Kconfig", "FirstLine": 140, "LineCount": 15 } Results: { "Contents": " 140:\t\tthe configured latency target, throttling anybody with a higher latency\n 141:\t\ttarget than the victimized group.\n 142:\t\n 143:\t\tNote, this is an experimental interface and could be changed someday.\n 144:\t\n 145:\tconfig BLK_CGROUP_FC_APPID\n 146:\t\tbool \"Enable support to track FC I/O Traffic across cgroup applications\"\n 147:\t\tdepends on BLK_CGROUP \u0026\u0026 NVME_FC\n 148:\t\thelp\n 149:\t\t Enabling this option enables the support to track FC I/O traffic across\n 150:\t\t cgroup applications. It enables the Fabric and the storage targets to\n 151:\t\t identify, monitor, and handle FC traffic based on VM tags by inserting\n 152:\t\t application specific identification into the FC frame.\n 153:\t\n 154:\tconfig BLK_CGROUP_IOCOST\n" }
57/2	2026/08/18 08:07	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 47599 output: 25 thoughts: 27 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience, presented in the first person: Navigating the NVMe-FC Dependency Alright, looking at this, it seems like the inclusion of `NVME_FC` hinges on the state of `BLK_CGROUP`. Given my understanding of the system's architecture, this is the crux of the matter. We need to be absolutely certain of the configuration surrounding `BLK_CGROUP` before we make any decisions about `NVME_FC`. Ultimately though, I'm thinking we should definitely be including `NVME_FC`. Given the functionality it enables, it's something that we should always make available in our configuration.
58/2	2026/08/18 08:08	tool	grepper	0m Args: { "Expression": "config NVME_FC", "PathPrefix": "" } Results: { "Output": "drivers/nvme/host/Kconfig=52=config NVME_RDMA\n--\ndrivers/nvme/host/Kconfig-66-\ndrivers/nvme/host/Kconfig:67:config NVME_FC\ndrivers/nvme/host/Kconfig-68-\ttristate \"NVM Express over Fabrics FC host driver\"\n" }
59/2	2026/08/18 08:08	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 47897 output: 32 thoughts: 27 Thoughts: Here's my take on the thought process: NVMe over Fibre Channel Investigation Okay, so I've been diving into this `NVMe_FC` thing, and it does seem like there's a specific prompt associated with it. That's good to know. Now, before I get too deep into this particular aspect, I think it's worth taking a quick look at `BLK_CGROUP_IOCOST`. It's connected, and understanding the interaction with I/O costs and cgroups is often critical in performance analysis and optimization. Let's just make sure I have the full picture before moving on.
60/2	2026/08/18 08:08	tool	read-file	0m Args: { "File": "block/Kconfig", "FirstLine": 150, "LineCount": 15 } Results: { "Contents": " 150:\t\t cgroup applications. It enables the Fabric and the storage targets to\n 151:\t\t identify, monitor, and handle FC traffic based on VM tags by inserting\n 152:\t\t application specific identification into the FC frame.\n 153:\t\n 154:\tconfig BLK_CGROUP_IOCOST\n 155:\t\tbool \"Enable support for cost model based cgroup IO controller\"\n 156:\t\tdepends on BLK_CGROUP\n 157:\t\tselect BLK_RQ_ALLOC_TIME\n 158:\t\thelp\n 159:\t\tEnabling this option enables the .weight interface for cost\n 160:\t\tmodel based proportional IO control. The IO controller\n 161:\t\tdistributes IO capacity between different groups based on\n 162:\t\ttheir share of the overall weight distribution.\n 163:\t\n 164:\tconfig BLK_CGROUP_IOPRIO\n" }
61/2	2026/08/18 08:08	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 48337 output: 32 thoughts: 27 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience: Analyzing Block I/O Control Right, so the behavior here hinges on `BLK_CGROUP`. That's the baseline. Whether this specific aspect of block I/O control even comes into play is the critical first question. We need to establish the foundational infrastructure before diving deeper. Now, presuming that `BLK_CGROUP` is indeed enabled and we are, therefore, operating within the cgroup ecosystem, the logical next step is to examine `BLK_CGROUP_IOLATENCY`. That's where the interesting stuff likely resides. Is I/O latency accounting enabled at the cgroup level? That setting dictates whether we can even see and control latency at this level. Understanding its configuration will determine our potential actions and insights going forward.
62/2	2026/08/18 08:08	tool	read-file	0m Args: { "File": "block/Kconfig", "FirstLine": 130, "LineCount": 15 } Results: { "Contents": " 130:\t\tdepends on BLK_WBT\n 131:\t\thelp\n 132:\t\tEnable writeback throttling by default for request-based block devices.\n 133:\t\n 134:\tconfig BLK_CGROUP_IOLATENCY\n 135:\t\tbool \"Enable support for latency based cgroup IO protection\"\n 136:\t\tdepends on BLK_CGROUP\n 137:\t\thelp\n 138:\t\tEnabling this option enables the .latency interface for IO throttling.\n 139:\t\tThe IO controller will attempt to maintain average IO latencies below\n 140:\t\tthe configured latency target, throttling anybody with a higher latency\n 141:\t\ttarget than the victimized group.\n 142:\t\n 143:\t\tNote, this is an experimental interface and could be changed someday.\n 144:\t\n" }
63/2	2026/08/18 08:08	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 48807 output: 32 thoughts: 26 Thoughts: Here's my summary, as I see it: Navigating the Block I/O Landscape Okay, so we're talking about block I/O, and the crucial piece here is `BLK_CGROUP`. Everything hinges on that. Before I go further down the rabbit hole, I need to verify that we're dealing with cgroups. I think `BLK_CGROUP_IOPRIO` is the key. Let's start there and see where it takes us. We'll verify this configuration and then proceed with further analysis.
64/2	2026/08/18 08:08	tool	read-file	0m Args: { "File": "block/Kconfig", "FirstLine": 160, "LineCount": 15 } Results: { "Contents": " 160:\t\tmodel based proportional IO control. The IO controller\n 161:\t\tdistributes IO capacity between different groups based on\n 162:\t\ttheir share of the overall weight distribution.\n 163:\t\n 164:\tconfig BLK_CGROUP_IOPRIO\n 165:\t\tbool \"Cgroup I/O controller for assigning an I/O priority class\"\n 166:\t\tdepends on BLK_CGROUP\n 167:\t\thelp\n 168:\t\tEnable the .prio interface for assigning an I/O priority class to\n 169:\t\trequests. The I/O priority class affects the order in which an I/O\n 170:\t\tscheduler and block devices process requests. Only some I/O schedulers\n 171:\t\tand some block devices support I/O priorities.\n 172:\t\n 173:\tconfig BLK_DEBUG_FS\n 174:\t\tbool \"Block layer debugging information in debugfs\"\n" }
65/2	2026/08/18 08:08	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 49239 output: 32 thoughts: 2512 Thoughts: Analysis of Block Cgroup Refactoring Patch for Fuzzing Okay, so I need to assess this patch for fuzzing potential. It seems like a significant refactoring of the block cgroup subsystem, which immediately piques my interest. It looks like the core of the change revolves around how `bio` structures are associated with block cgroups. Instead of a direct pointer to a `blkcg_gq` (blkg) in the `bio`, the patch now uses a more generic `blkcg` and dynamically looks up or pins the queue-specific `blkg` when needed, or when `BIO_BLKG_REF` flag is set. That sounds like a fundamental shift in how the system handles IO scheduling and resource allocation at the block layer. The `radix_tree` has been replaced with an `rhashtable` for `blkg` lookups, which also suggests performance implications. These are definitely functional changes worth investigating, since these kinds of core changes can easily introduce regressions or new corner cases. First, I need to identify the relevant configurations. Based on the code, it looks like these are the important ones: `BLK_CGROUP`, `IOSCHED_BFQ`, `BFQ_GROUP_IOSCHED`, `BLK_CGROUP_IOCOST`, `BLK_CGROUP_IOLATENCY`, `BLK_CGROUP_IOPRIO`, `NVME_FC`, `BLK_CGROUP_FC_APPID`, and `BTRFS_FS`. I'll need to remember those for building a fuzzer configuration. And because `BLK_CGROUP_PUNT_BIO` is selected by `BTRFS_FS`, I can just list `BTRFS_FS`. Next, I need to pinpoint the key functions (FocusSymbols) that are most impacted by this change. I've already identified several candidates, going through the code. These are: `blkg_conf_prep`, `blkcg_punt_bio_submit`, `blkg_init_queue`, `blkg_exit_queue`, `blkcg_init_disk`, and `blkcg_exit_disk`. I need to confirm that these functions are actually reachable and how they are invoked. `blkg_conf_prep` is a good candidate since it's called from sysfs write operations, which are readily accessible. `blkcg_punt_bio_submit` looks interesting too. It's called from the `btrfs_submit_dev_bio` function if `REQ_BTRFS_CGROUP_PUNT` is set, so BTRFS IO is a good target. I can potentially craft IO scenarios with BTRFS to trigger it. The device creation/destruction functions `blkg_init_queue`, `blkg_exit_queue`, `blkcg_init_disk`, `blkcg_exit_disk` also look promising, since syzkaller can create/destroy loop devices, null_blk, etc. to trigger these. I double-checked to confirm the dependencies for some of the related configurations: `BTRFS_FS`, `NVME_FC`, `IOSCHED_BFQ`. Good to know. Therefore, my strategy is to focus on these functions and configurations when constructing fuzz test cases. The refactoring itself looks pretty significant. I'm seeing a deep change that's fundamental to block cgroup IO control, the block layer, and btrfs.
66/2	2026/08/18 08:08	tool	read-file	0m Args: { "File": "block/Kconfig", "FirstLine": 130, "LineCount": 10 } Results: { "Contents": " 130:\t\tdepends on BLK_WBT\n 131:\t\thelp\n 132:\t\tEnable writeback throttling by default for request-based block devices.\n 133:\t\n 134:\tconfig BLK_CGROUP_IOLATENCY\n 135:\t\tbool \"Enable support for latency based cgroup IO protection\"\n 136:\t\tdepends on BLK_CGROUP\n 137:\t\thelp\n 138:\t\tEnabling this option enables the .latency interface for IO throttling.\n 139:\t\tThe IO controller will attempt to maintain average IO latencies below\n" }
67/2	2026/08/18 08:08	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 52683 output: 32 thoughts: 28 Thoughts: Okay, here's my take on those thoughts, framed as an expert's internal monologue: Investigating Block Device I/O Control Alright, this situation hinges on `BLK_CGROUP`. That's the core, isn't it? It dictates how much of the relevant functionality is even available in the first place. Without it, we're likely dealing with a severely limited scenario. I need to keep that in mind as the fundamental building block. Now, before I dive too deep, I need to revisit the dependency relationships for `BLK_CGROUP_IOCOST`. This is crucial. It's easy to get lost in a tangled web of kernel configuration options, and I want to be absolutely sure that I understand which options are enabled. I'll re-examine the direct dependencies, any indirect ones, and the conditional dependencies, to make certain that `BLK_CGROUP_IOCOST` is being pulled in as expected under the relevant conditions. There's a risk of introducing the wrong behaviour.
68/2	2026/08/18 08:08	tool	read-file	0m Args: { "File": "block/Kconfig", "FirstLine": 150, "LineCount": 10 } Results: { "Contents": " 150:\t\t cgroup applications. It enables the Fabric and the storage targets to\n 151:\t\t identify, monitor, and handle FC traffic based on VM tags by inserting\n 152:\t\t application specific identification into the FC frame.\n 153:\t\n 154:\tconfig BLK_CGROUP_IOCOST\n 155:\t\tbool \"Enable support for cost model based cgroup IO controller\"\n 156:\t\tdepends on BLK_CGROUP\n 157:\t\tselect BLK_RQ_ALLOC_TIME\n 158:\t\thelp\n 159:\t\tEnabling this option enables the .weight interface for cost\n" }
69/2	2026/08/18 08:08	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 53137 output: 32 thoughts: 29 Thoughts: Here's my take on those thoughts: Analyzing Block I/O Cgroup Configuration Okay, so the architecture I'm working with hinges on `BLK_CGROUP`. That's the baseline. I need to understand how block I/O cgroups are actually implemented in this system before I go any further. It's the foundation upon which everything else is built. Now, let me double-check the dependencies for `BLK_CGROUP_IOPRIO`. I thought I had the relationship mapped, but I need to make sure I haven't missed anything. I want to be absolutely certain that I haven't overlooked any crucial interactions or potential conflicts. I need to make sure that its configuration is solid, to avoid issues downstream. Gotta get this right. This is important. I’ll make sure it plays nicely with other relevant features.
70/2	2026/08/18 08:08	tool	read-file	0m Args: { "File": "block/Kconfig", "FirstLine": 160, "LineCount": 10 } Results: { "Contents": " 160:\t\tmodel based proportional IO control. The IO controller\n 161:\t\tdistributes IO capacity between different groups based on\n 162:\t\ttheir share of the overall weight distribution.\n 163:\t\n 164:\tconfig BLK_CGROUP_IOPRIO\n 165:\t\tbool \"Cgroup I/O controller for assigning an I/O priority class\"\n 166:\t\tdepends on BLK_CGROUP\n 167:\t\thelp\n 168:\t\tEnable the .prio interface for assigning an I/O priority class to\n 169:\t\trequests. The I/O priority class affects the order in which an I/O\n" }
71/2	2026/08/18 08:08	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 53563 output: 327 thoughts: 121 Thoughts: Okay, here's my interpretation of this configuration landscape, presented as a summary of my thought process: Analyzing Block I/O Configuration Dependencies Alright, let's break this down. My initial observation hinges directly on the `BLK_CGROUP` setting. That's the linchpin, the foundation of this whole block I/O configuration space. The presence (or absence) of that macro will significantly influence everything else. Therefore, the relevant configurations here are: `BLK_CGROUP` itself, of course. Then we have `IOSCHED_BFQ`, `BFQ_GROUP_IOSCHED`, `BLK_CGROUP_IOCOST`, `BLK_CGROUP_IOLATENCY`, `BLK_CGROUP_IOPRIO`, `NVME_FC`, `BLK_CGROUP_FC_APPID` and `BTRFS_FS`. Each of these are likely conditional on that initial `BLK_CGROUP` being defined, or at least they are interconnected. The specific behavior of each depends on which features are enabled. At this point, I'll switch over to the `set-results` tool to analyze the specific combinations of these configurations and see their impact on the system. Time to execute.
72/2	2026/08/18 08:09	tool	set-results	0m Args: { "EnableConfigs": [ "BLK_CGROUP", "IOSCHED_BFQ", "BFQ_GROUP_IOSCHED", "BLK_CGROUP_IOCOST", "BLK_CGROUP_IOLATENCY", "BLK_CGROUP_IOPRIO", "NVME_FC", "BLK_CGROUP_FC_APPID", "BTRFS_FS" ], "FocusSymbols": [ "blkg_conf_prep", "blkcg_punt_bio_submit", "blkg_init_queue", "blkg_exit_queue", "blkcg_init_disk", "blkcg_exit_disk" ], "Reasoning": "The patch significantly refactors the block cgroup (blkcg) subsystem by changing how `bio`s are associated with block cgroups. Instead of directly storing a queue-specific `blkcg_gq` (blkg) in `bio-\u003ebi_blkg`, it now stores the generic `blkcg` in `bio-\u003ebi_blkcg` and dynamically looks up or pins the queue-specific `blkg` using a new `BIO_BLKG_REF` flag. It also replaces the `radix_tree` used for `blkg` lookups with an `rhashtable`. These are core functional changes to the block layer and cgroup IO controllers, affecting IO scheduling, throttling, and accounting.", "WorthFuzzing": true } Results: { "EnableConfigs": [ "BLK_CGROUP", "IOSCHED_BFQ", "BFQ_GROUP_IOSCHED", "BLK_CGROUP_IOCOST", "BLK_CGROUP_IOLATENCY", "BLK_CGROUP_IOPRIO", "NVME_FC", "BLK_CGROUP_FC_APPID", "BTRFS_FS" ], "FocusSymbols": [ "blkg_conf_prep", "blkcg_punt_bio_submit", "blkg_init_queue", "blkg_exit_queue", "blkcg_init_disk", "blkcg_exit_disk" ], "Reasoning": "The patch significantly refactors the block cgroup (blkcg) subsystem by changing how `bio`s are associated with block cgroups. Instead of directly storing a queue-specific `blkcg_gq` (blkg) in `bio-\u003ebi_blkg`, it now stores the generic `blkcg` in `bio-\u003ebi_blkcg` and dynamically looks up or pins the queue-specific `blkg` using a new `BIO_BLKG_REF` flag. It also replaces the `radix_tree` used for `blkg` lookups with an `rhashtable`. These are core functional changes to the block layer and cgroup IO controllers, affecting IO scheduling, throttling, and accounting.", "WorthFuzzing": true }
73/1	2026/08/18 08:09	action	If	0m Args: { "WorthFuzzing": true } Results: null
74/2	2026/08/18 08:09	agent	kmsan-evaluator	0m Model: best-expensive Args: null Results: { "KMSANReasoning": "The patch refactors the block cgroup (blkcg) association for `bio` structures, replacing the direct `bi_blkg` pointer with `bi_blkcg` and dynamically looking up or pinning the queue-local `blkcg_gq` (blkg) using a hash table instead of a radix tree.\n\nThese changes involve modifying internal kernel structures (`bio`, `request_queue`, `blkcg_gq`) and their initialization/cleanup routines (`bio_init`, `bio_uninit`, `bio_clear_blkcg`). No new structures are copied to user space, nor are there any new buffers or complex data structures allocated without proper initialization. The modified fields in `bio` are explicitly initialized to `NULL` or appropriate values in `bio_init()`.\n\nAny bugs introduced by these changes would likely manifest as NULL pointer dereferences, use-after-free (e.g., incorrect reference counting of `blkcg` or `blkg`), or memory leaks, all of which are effectively caught by standard KASAN and other default debugging tools. There is no risk of uninitialized memory reads or kernel-to-user info-leaks that would necessitate a dedicated KMSAN fuzzing session.", "NeedsKMSAN": false } Instruction: You are an expert Linux kernel security engineer specializing in kernel memory error detectors (KASAN and KMSAN). Your job is to review the provided patch series and determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer) fuzzing session in addition to standard KASAN fuzzing. CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN: - Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable a comprehensive suite of debugging tools and sanitizers, including KASAN (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks. - KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap, or page allocations) and kernel-to-user memory info-leaks. Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN, LOCKDEP, OR OTHER STANDARD BUG DETECTORS. A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks that are detected ONLY by KMSAN. Look holistically at the patch series and surrounding code. Even if no direct uninitialized field accesses or new buffer allocations are added in the diff itself, a patch may alter control flow, bounds checking, or data length calculations in ways that change how the rest of the code operates on existing buffers (e.g. allowing uninitialized stack/heap memory to be read, copied to user space, or used in control flow). Do not hesitate to use your code access tools to inspect the surrounding code, called functions, and callers. Set NeedsKMSAN=true ONLY IF the patch introduces or modifies: 1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb attributes, ioctl output arguments, socket options, or BPF buffers) where fields or structure padding might not be fully initialized/zeroed. 2. Conditional logic or branching that depends on potentially uninitialized variables or struct fields. 3. Allocation or initialization of complex data structures where uninitialized fields could be read later in reachable code paths. 4. Bounds checks, lengths, or logic in a way that allows surrounding code to access uninitialized bytes of existing buffers. Set NeedsKMSAN=false IF: - The code changes primarily risk out-of-bounds access, array overflows, NULL pointer dereferences, locking deadlocks, or use-after-free bugs (these are already caught by KASAN, LOCKDEP, or standard bug detectors). - All stack/heap structures touched or introduced by the patch are fully zeroed or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied. - The patch does not introduce any risk of uninitialized memory usage or info-leaks. Use your code access tools to inspect the surrounding code if necessary, then provide detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch. Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit abc0b1a65e87d979eec053ca54836b90fdec9729 Author: syz-cluster <triage@syzkaller.com> Date: Tue Aug 18 08:04:15 2026 +0000 syz-cluster: applied patch under review diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 14b8c571c0d14..bbd79931d5ab5 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -3235,7 +3235,7 @@ the configuration, the bio may be executed at a lower priority and if the writeback session is holding shared resources, e.g. a journal entry, may lead to priority inversion. There is no one easy solution for the problem. Filesystems can try to work around specific problem -cases by skipping wbc_init_bio() and using bio_associate_blkg() +cases by skipping wbc_init_bio() and using bio_associate_blkcg() directly. diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index ef9d63e8542ed..4a3975f9ff744 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -363,11 +363,13 @@ void bfqg_and_blkg_put(struct bfq_group bfqg) void bfqg_stats_update_legacy_io(struct request_queue q, struct request rq) { - struct bfq_group bfqg = blkg_to_bfqg(rq->bio->bi_blkg); + struct blkcg_gq blkg = bio_blkg_lookup(rq->bio); + struct bfq_group bfqg; - if (!bfqg) + if (!blkg) return; + bfqg = blkg_to_bfqg(blkg); blkg_rwstat_add(&bfqg->stats.bytes, rq->cmd_flags, blk_rq_bytes(rq)); blkg_rwstat_add(&bfqg->stats.ios, rq->cmd_flags, 1); } @@ -606,7 +608,7 @@ static void bfq_link_bfqg(struct bfq_data bfqd, struct bfq_group bfqg) struct bfq_group bfq_bio_bfqg(struct bfq_data bfqd, struct bio bio) { - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg_gq blkg = bio_blkg_lookup(bio); struct bfq_group bfqg; while (blkg) { @@ -614,15 +616,17 @@ struct bfq_group bfq_bio_bfqg(struct bfq_data bfqd, struct bio bio) blkg = blkg->parent; continue; } + bfqg = blkg_to_bfqg(blkg); if (bfqg->pd.online) { - bio_associate_blkg_from_css(bio, &blkg->blkcg->css); + bio_associate_blkcg_from_css(bio, &blkg->blkcg->css); return bfqg; } blkg = blkg->parent; } - bio_associate_blkg_from_css(bio, - &bfqg_to_blkg(bfqd->root_group)->blkcg->css); + + blkg = bfqg_to_blkg(bfqd->root_group); + bio_associate_blkcg_from_css(bio, &blkg->blkcg->css); return bfqd->root_group; } diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 0f75301b3115f..d4f657bc7a4e5 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -128,6 +128,7 @@ #include "elevator.h" #include "blk.h" +#include "blk-cgroup.h" #include "blk-mq.h" #include "blk-mq-sched.h" #include "bfq-iosched.h" @@ -2452,15 +2453,16 @@ static bool bfq_bio_merge(struct request_queue q, struct bio bio, struct request free = NULL; bool ret; +#ifdef CONFIG_BFQ_GROUP_IOSCHED + /* blkg creation takes q->queue_lock, so do it before bfqd->lock. / + if (bic) + bio_blkg(bio); +#endif + spin_lock_irq(&bfqd->lock); if (bic) { - / - * Make sure cgroup info is uptodate for current process before - * considering the merge. - / bfq_bic_update_cgroup(bic, bio); - bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf), bfq_actuator_index(bfqd, bio)); } else { @@ -6245,6 +6247,13 @@ static void bfq_insert_request(struct blk_mq_hw_ctx hctx, struct request rq, LIST_HEAD(free); #ifdef CONFIG_BFQ_GROUP_IOSCHED + / + * Pin the blkg used to look up bfqg. If this is the first IO for + * the blkcg on this queue, create the bfqg before holding bfqd->lock. + / + if (rq->bio && !bio_flagged(rq->bio, BIO_BLKG_REF)) + bio_blkg(rq->bio); + if (!cgroup_subsys_on_dfl(io_cgrp_subsys) && rq->bio) bfqg_stats_update_legacy_io(q, rq); #endif diff --git a/block/bio.c b/block/bio.c index 898b2f5ef8c81..6c84042adf5dd 100644 --- a/block/bio.c +++ b/block/bio.c @@ -179,12 +179,7 @@ static inline gfp_t try_alloc_gfp(gfp_t gfp) void bio_uninit(struct bio bio) { -#ifdef CONFIG_BLK_CGROUP - if (bio->bi_blkg) { - blkg_put(bio->bi_blkg); - bio->bi_blkg = NULL; - } -#endif + bio_clear_blkcg(bio); if (bio_integrity(bio)) bio_integrity_free(bio); @@ -233,10 +228,10 @@ void bio_init(struct bio bio, struct block_device bdev, struct bio_vec table, bio->bi_end_io = NULL; bio->bi_private = NULL; #ifdef CONFIG_BLK_CGROUP - bio->bi_blkg = NULL; + bio->bi_blkcg = NULL; bio->issue_time_ns = 0; if (bdev) - bio_associate_blkg(bio); + bio_associate_blkcg(bio); #ifdef CONFIG_BLK_CGROUP_IOCOST bio->bi_iocost_cost = 0; #endif @@ -281,7 +276,7 @@ void bio_reset(struct bio bio, struct block_device bdev, blk_opf_t opf) bio->bi_io_vec = bv; bio->bi_bdev = bdev; if (bio->bi_bdev) - bio_associate_blkg(bio); + bio_associate_blkcg(bio); bio->bi_opf = opf; } EXPORT_SYMBOL(bio_reset); @@ -866,7 +861,7 @@ static int __bio_clone(struct bio bio, struct bio bio_src, gfp_t gfp) if (bio->bi_bdev == bio_src->bi_bdev && bio_flagged(bio_src, BIO_REMAPPED)) bio_set_flag(bio, BIO_REMAPPED); - bio_clone_blkg_association(bio, bio_src); + bio_clone_blkcg_association(bio, bio_src); } if (bio_crypt_clone(bio, bio_src, gfp) < 0) @@ -1910,17 +1905,12 @@ void bio_endio(struct bio bio) goto again; } -#ifdef CONFIG_BLK_CGROUP /* * Release cgroup info. We shouldn't have to do this here, but quite * a few callers of bio_init fail to call bio_uninit, so we cover up * for that here at least for now. / - if (bio->bi_blkg) { - blkg_put(bio->bi_blkg); - bio->bi_blkg = NULL; - } -#endif + bio_clear_blkcg(bio); if (bio_flagged(bio, BIO_COMPLETE_IN_TASK) && bio_in_atomic()) __bio_complete_in_task(bio); diff --git a/block/blk-cgroup-fc-appid.c b/block/blk-cgroup-fc-appid.c index 3ec21333f393b..7589c62099898 100644 --- a/block/blk-cgroup-fc-appid.c +++ b/block/blk-cgroup-fc-appid.c @@ -50,8 +50,14 @@ EXPORT_SYMBOL_GPL(blkcg_set_fc_appid); / char blkcg_get_fc_appid(struct bio bio) { - if (!bio->bi_blkg \|\| bio->bi_blkg->blkcg->fc_app_id[0] == '\0') + struct blkcg blkcg = bio_blkcg(bio); + + if (!blkcg) + return NULL; + + if (blkcg->fc_app_id[0] == '\0') return NULL; - return bio->bi_blkg->blkcg->fc_app_id; + + return blkcg->fc_app_id; } EXPORT_SYMBOL_GPL(blkcg_get_fc_appid); diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 2b5c29434e426..2c758c26bce60 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -66,6 +66,13 @@ static DEFINE_RAW_SPINLOCK(blkg_stat_lock); #define BLKG_DESTROY_BATCH_SIZE 64 +const struct rhashtable_params blkg_hash_params = { + .key_len = sizeof_field(struct blkcg_gq, blkcg_id), + .key_offset = offsetof(struct blkcg_gq, blkcg_id), + .head_offset = offsetof(struct blkcg_gq, q_hash_node), + .automatic_shrinking = true, +}; + / * Lockless lists for tracking IO stats update * @@ -173,10 +180,6 @@ static void __blkg_release(struct rcu_head rcu) { struct blkcg_gq blkg = container_of(rcu, struct blkcg_gq, rcu_head); -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO - WARN_ON(!bio_list_empty(&blkg->async_bios)); -#endif - blkg_free(blkg); } @@ -194,6 +197,16 @@ static void blkg_release(struct percpu_ref ref) struct blkcg blkcg = blkg->blkcg; int cpu; + /* + * A blkg that was never inserted into q->blkg_list has no hash + * entry. This happens when allocation or creation fails before + * rhashtable_insert_fast() succeeds. + / + if (!list_empty(&blkg->q_node)) + WARN_ON_ONCE(rhashtable_remove_fast(&blkg->q->blkg_hash, + &blkg->q_hash_node, + blkg_hash_params)); + / * Flush all the non-empty percpu lockless lists before releasing * us, given these stat belongs to us. @@ -209,19 +222,18 @@ static void blkg_release(struct percpu_ref ref) #ifdef CONFIG_BLK_CGROUP_PUNT_BIO static struct workqueue_struct blkcg_punt_bio_wq; -static void blkg_async_bio_workfn(struct work_struct work) +static void blkcg_async_bio_workfn(struct work_struct work) { - struct blkcg_gq blkg = container_of(work, struct blkcg_gq, - async_bio_work); + struct blkcg blkcg = container_of(work, struct blkcg, async_bio_work); struct bio_list bios = BIO_EMPTY_LIST; struct bio bio; struct blk_plug plug; bool need_plug = false; - / as long as there are pending bios, @blkg can't go away / - spin_lock(&blkg->async_bio_lock); - bio_list_merge_init(&bios, &blkg->async_bios); - spin_unlock(&blkg->async_bio_lock); + / as long as there are pending bios, @blkcg can't go away / + spin_lock(&blkcg->async_bio_lock); + bio_list_merge_init(&bios, &blkcg->async_bios); + spin_unlock(&blkcg->async_bio_lock); / start plug only when bio_list contains at least 2 bios / if (bios.head && bios.head->bi_next) { @@ -242,15 +254,15 @@ static void blkg_async_bio_workfn(struct work_struct work) / void blkcg_punt_bio_submit(struct bio bio) { - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg blkcg = bio_blkcg(bio); - if (blkg->parent) { - spin_lock(&blkg->async_bio_lock); - bio_list_add(&blkg->async_bios, bio); - spin_unlock(&blkg->async_bio_lock); - queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work); + if (blkcg && cgroup_parent(blkcg->css.cgroup)) { + spin_lock(&blkcg->async_bio_lock); + bio_list_add(&blkcg->async_bios, bio); + spin_unlock(&blkcg->async_bio_lock); + queue_work(blkcg_punt_bio_wq, &blkcg->async_bio_work); } else { - /* never bounce for the root cgroup / + / Never bounce if there is no non-root blkcg to queue on. / submit_bio(bio); } } @@ -278,9 +290,13 @@ subsys_initcall(blkcg_punt_bio_init); / struct cgroup_subsys_state bio_blkcg_css(struct bio bio) { - if (!bio \|\| !bio->bi_blkg) + struct blkcg blkcg; + + if (!bio) return NULL; - return &bio->bi_blkg->blkcg->css; + + blkcg = bio_blkcg(bio); + return blkcg ? &blkcg->css : NULL; } EXPORT_SYMBOL_GPL(bio_blkcg_css); @@ -327,12 +343,8 @@ static struct blkcg_gq blkg_alloc(struct blkcg blkcg, struct gendisk disk, blkg->q = disk->queue; INIT_LIST_HEAD(&blkg->q_node); blkg->blkcg = blkcg; + blkg->blkcg_id = blkcg->css.id; blkg->iostat.blkg = blkg; -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO - spin_lock_init(&blkg->async_bio_lock); - bio_list_init(&blkg->async_bios); - INIT_WORK(&blkg->async_bio_work, blkg_async_bio_workfn); -#endif u64_stats_init(&blkg->iostat.sync); for_each_possible_cpu(cpu) { @@ -423,7 +435,8 @@ static struct blkcg_gq blkg_create(struct blkcg blkcg, struct gendisk disk, / insert / spin_lock(&blkcg->lock); - ret = radix_tree_insert(&blkcg->blkg_tree, disk->queue->id, blkg); + ret = rhashtable_insert_fast(&disk->queue->blkg_hash, + &blkg->q_hash_node, blkg_hash_params); if (likely(!ret)) { hlist_add_head_rcu(&blkg->blkcg_node, &blkcg->blkg_list); list_add(&blkg->q_node, &disk->queue->blkg_list); @@ -454,6 +467,17 @@ static struct blkcg_gq blkg_create(struct blkcg blkcg, struct gendisk disk, return ERR_PTR(ret); } +/* + * The root blkg holds a live reference while the disk is active, so walking + * the parent chain always finds a blkg which can be pinned. + / +static struct blkcg_gq blkg_lookup_tryget(struct blkcg_gq blkg) +{ + while (!blkg_tryget(blkg)) + blkg = blkg->parent; + return blkg; +} + /* * blkg_lookup_create - lookup blkg, try to create one if not there * @blkcg: blkcg of interest @@ -461,11 +485,13 @@ static struct blkcg_gq blkg_create(struct blkcg blkcg, struct gendisk disk, * Lookup blkg for the @blkcg - @disk pair. If it doesn't exist, try to * create one. blkg creation is performed recursively from blkcg_root such - * that all non-root blkg's have access to the parent blkg. This function - * should be called under RCU read lock and takes @disk->queue->queue_lock. + * that all non-root blkg's have access to the parent blkg. * - * Returns the blkg or the closest blkg if blkg_create() fails as it walks - * down from root. + * Must be called with @disk->queue->queue_lock held. + * + * Returns the closest blkg with an extra reference acquired. If + * blkg_create() fails while walking down from root, the returned blkg may + * belong to an ancestor of @blkcg. This function never returns %NULL. / static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg, struct gendisk disk) @@ -473,12 +499,12 @@ static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg, struct request_queue q = disk->queue; struct blkcg_gq blkg; + lockdep_assert_held(&q->queue_lock); + rcu_read_lock(); blkg = blkg_lookup(blkcg, q); if (blkg) { - if (blkcg != &blkcg_root && - blkg != rcu_dereference(blkcg->blkg_hint)) - rcu_assign_pointer(blkcg->blkg_hint, blkg); + blkg = blkg_lookup_tryget(blkg); rcu_read_unlock(); return blkg; } @@ -514,7 +540,7 @@ static struct blkcg_gq blkg_lookup_create(struct blkcg blkcg, break; } - return blkg; + return blkg_lookup_tryget(blkg); } static void blkg_destroy(struct blkcg_gq blkg) @@ -546,17 +572,8 @@ static void blkg_destroy(struct blkcg_gq blkg) blkg->online = false; - radix_tree_delete(&blkcg->blkg_tree, blkg->q->id); hlist_del_init_rcu(&blkg->blkcg_node); - /* - * Both setting lookup hint to and clearing it from @blkg are done - * under queue_lock. If it's not pointing to @blkg now, it never - * will. Hint assignment itself can race safely. - / - if (rcu_access_pointer(blkcg->blkg_hint) == blkg) - rcu_assign_pointer(blkcg->blkg_hint, NULL); - / * Put the reference taken at the time of creation so that when all * queues are gone, group can be destroyed. @@ -882,18 +899,12 @@ int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol, goto fail_exit; } - if (radix_tree_preload(GFP_KERNEL)) { - blkg_free(new_blkg); - ret = -ENOMEM; - goto fail_exit; - } - spin_lock_irq(&q->queue_lock); if (!blkcg_policy_enabled(q, pol)) { blkg_free(new_blkg); ret = -EOPNOTSUPP; - goto fail_preloaded; + goto fail_unlock; } blkg = blkg_lookup(pos, q); @@ -903,12 +914,10 @@ int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol, blkg = blkg_create(pos, disk, new_blkg); if (IS_ERR(blkg)) { ret = PTR_ERR(blkg); - goto fail_preloaded; + goto fail_unlock; } } - radix_tree_preload_end(); - if (pos == blkcg) goto success; } @@ -917,8 +926,6 @@ int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol, ctx->blkg = blkg; return 0; -fail_preloaded: - radix_tree_preload_end(); fail_unlock: spin_unlock_irq(&q->queue_lock); fail_exit: @@ -1373,6 +1380,9 @@ static void blkcg_css_free(struct cgroup_subsys_state css) mutex_unlock(&blkcg_pol_mutex); +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO + WARN_ON(!bio_list_empty(&blkcg->async_bios)); +#endif free_percpu(blkcg->lhead); kfree(blkcg); } @@ -1420,8 +1430,12 @@ blkcg_css_alloc(struct cgroup_subsys_state parent_css) spin_lock_init(&blkcg->lock); refcount_set(&blkcg->online_pin, 1); - INIT_RADIX_TREE(&blkcg->blkg_tree, GFP_NOWAIT); INIT_HLIST_HEAD(&blkcg->blkg_list); +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO + spin_lock_init(&blkcg->async_bio_lock); + bio_list_init(&blkcg->async_bios); + INIT_WORK(&blkcg->async_bio_work, blkcg_async_bio_workfn); +#endif #ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD(&blkcg->cgwb_list); #endif @@ -1457,17 +1471,22 @@ static int blkcg_css_online(struct cgroup_subsys_state css) return 0; } -void blkg_init_queue(struct request_queue q) +int blkg_init_queue(struct request_queue q) { INIT_LIST_HEAD(&q->blkg_list); mutex_init(&q->blkcg_mutex); + return rhashtable_init(&q->blkg_hash, &blkg_hash_params); +} + +void blkg_exit_queue(struct request_queue q) +{ + rhashtable_destroy(&q->blkg_hash); } int blkcg_init_disk(struct gendisk disk) { struct request_queue q = disk->queue; struct blkcg_gq new_blkg, blkg; - bool preloaded; /* * If the queue is shared across disk rebind (e.g., SCSI), the @@ -1485,8 +1504,6 @@ int blkcg_init_disk(struct gendisk disk) if (!new_blkg) return -ENOMEM; - preloaded = !radix_tree_preload(GFP_KERNEL); - / Make sure the root blkg exists. / / spin_lock_irq can serve as RCU read-side critical section. / spin_lock_irq(&q->queue_lock); @@ -1496,15 +1513,10 @@ int blkcg_init_disk(struct gendisk disk) q->root_blkg = blkg; spin_unlock_irq(&q->queue_lock); - if (preloaded) - radix_tree_preload_end(); - return 0; err_unlock: spin_unlock_irq(&q->queue_lock); - if (preloaded) - radix_tree_preload_end(); return PTR_ERR(blkg); } @@ -2064,129 +2076,165 @@ void blkcg_add_delay(struct blkcg_gq blkg, u64 now, u64 delta) atomic64_add(delta, &blkg->delay_nsec); } -static inline struct blkcg_gq blkg_lookup_tryget(struct blkcg_gq blkg) +/ + * Return the blkg pinned by @bio through BIO_BLKG_REF. The returned blkg is + * already owned by @bio and no extra reference is acquired. + / +static struct blkcg_gq bio_pinned_blkg(struct bio bio) { -retry: - if (blkg_tryget(blkg)) - return blkg; + struct request_queue q = bdev_get_queue(bio->bi_bdev); + struct blkcg_gq blkg; - blkg = blkg->parent; - if (blkg) - goto retry; + rcu_read_lock(); + blkg = blkg_lookup_any(bio_blkcg(bio), q); + rcu_read_unlock(); - return NULL; + WARN_ON_ONCE(!blkg); + return blkg; +} + +static void bio_set_blkg_ref(struct bio bio, struct blkcg_gq blkg) +{ + if (bio_blkcg(bio) != blkg->blkcg) { + css_get(&blkg->blkcg->css); + bio_clear_blkcg(bio); + bio->bi_blkcg = blkg->blkcg; + } + bio_set_flag(bio, BIO_BLKG_REF); } + /* - * blkg_tryget_closest - try and get a blkg ref on the closet blkg + * bio_blkg_lookup - look up a blkg associated with a bio * @bio: target bio - * @css: target css * - * As the failure mode here is to walk up the blkg tree, this ensure that the - * blkg->parent pointers are always valid. This returns the blkg that it ended - * up taking a reference on or %NULL if no reference was taken. + * Return the blkg already pinned by @bio, or %NULL if @bio doesn't own a blkg + * reference. Call bio_blkg() instead when a missing blkg should be created. / -static inline struct blkcg_gq blkg_tryget_closest(struct bio bio, - struct cgroup_subsys_state css) +struct blkcg_gq bio_blkg_lookup(struct bio bio) { - struct request_queue q = bio->bi_bdev->bd_queue; - struct blkcg blkcg = css_to_blkcg(css); + if (!bio_flagged(bio, BIO_BLKG_REF)) + return NULL; + return bio_pinned_blkg(bio); +} +EXPORT_SYMBOL_GPL(bio_blkg_lookup); + +/** + * bio_put_blkg_ref - drop the blkg reference pinned by a bio + * @bio: target bio + * + * Drop the bio-owned blkg reference acquired by bio_blkg(), if any. + / +void bio_put_blkg_ref(struct bio bio) +{ + if (bio_flagged(bio, BIO_BLKG_REF)) { + struct blkcg_gq blkg = bio_pinned_blkg(bio); + + if (blkg) + blkg_put(blkg); + bio_clear_flag(bio, BIO_BLKG_REF); + } +} +EXPORT_SYMBOL_GPL(bio_put_blkg_ref); + +/* + * bio_blkg - look up the blkg associated with a bio + * @bio: target bio + * + * Look up the queue-local blkg for @bio's current device and blkcg. If this + * is the first policy use of @bio, create the missing blkg hierarchy if + * necessary, pin the closest available blkg, and update @bio's blkcg + * association if allocation falls back to an ancestor. + / +struct blkcg_gq bio_blkg(struct bio bio) +{ + struct blkcg blkcg = bio_blkcg(bio); + struct gendisk disk; + struct request_queue q; struct blkcg_gq blkg; - rcu_read_lock(); - blkg = blkg_lookup(blkcg, q); - if (likely(blkg)) - blkg = blkg_lookup_tryget(blkg); - rcu_read_unlock(); + if (!blkcg \|\| !bio->bi_bdev) + return NULL; - if (blkg) - return blkg; + if (bio_flagged(bio, BIO_BLKG_REF)) + return bio_pinned_blkg(bio); + + disk = bio->bi_bdev->bd_disk; + q = disk->queue; - / - * Fast path failed, we're probably issuing IO in this cgroup the first - * time, hold lock to create new blkg. - / spin_lock_irq(&q->queue_lock); - blkg = blkg_lookup_create(blkcg, bio->bi_bdev->bd_disk); - if (blkg) - blkg = blkg_lookup_tryget(blkg); + blkg = blkg_lookup_create(blkcg, disk); spin_unlock_irq(&q->queue_lock); + bio_set_blkg_ref(bio, blkg); return blkg; } +EXPORT_SYMBOL_GPL(bio_blkg); /* - * bio_associate_blkg_from_css - associate a bio with a specified css + * bio_associate_blkcg_from_css - associate a bio with a specified css * @bio: target bio * @css: target css * - * Associate @bio with the blkg found by combining the css's blkg and the - * request_queue of the @bio. An association failure is handled by walking up - * the blkg tree. Therefore, the blkg associated can be anything between @blkg - * and q->root_blkg. This situation only happens when a cgroup is dying and - * then the remaining bios will spill to the closest alive blkg. + * Associate @bio with the blkcg found from @css. The queue-local blkg is + * created and pinned by bio_blkg() when blkcg policies need it. If @css + * can't be referenced online, associate @bio with the root blkcg instead. * - * A reference will be taken on the blkg and will be released when @bio is + * A reference will be taken on the blkcg and will be released when @bio is * freed. / -void bio_associate_blkg_from_css(struct bio bio, - struct cgroup_subsys_state css) +void bio_associate_blkcg_from_css(struct bio bio, + struct cgroup_subsys_state css) { - if (bio->bi_blkg) - blkg_put(bio->bi_blkg); + struct blkcg blkcg; - if (css && css->parent) { - bio->bi_blkg = blkg_tryget_closest(bio, css); - } else { - blkg_get(bdev_get_queue(bio->bi_bdev)->root_blkg); - bio->bi_blkg = bdev_get_queue(bio->bi_bdev)->root_blkg; + if (!css \|\| !css_tryget_online(css)) { + css = &blkcg_root.css; + css_get(css); } + + blkcg = css_to_blkcg(css); + if (bio_blkcg(bio) == blkcg) { + css_put(css); + return; + } + + bio_clear_blkcg(bio); + bio->bi_blkcg = blkcg; } -EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css); +EXPORT_SYMBOL_GPL(bio_associate_blkcg_from_css); /** - * bio_associate_blkg - associate a bio with a blkg + * bio_associate_blkcg - associate a bio with a blkcg * @bio: target bio * - * Associate @bio with the blkg found from the bio's css and request_queue. - * If one is not found, bio_lookup_blkg() creates the blkg. If a blkg is - * already associated, the css is reused and association redone as the - * request_queue may have changed. + * Associate @bio with the blkcg found from the bio's css. If a blkcg is + * already associated, keep it as blkcg association is not queue-local. / -void bio_associate_blkg(struct bio bio) +void bio_associate_blkcg(struct bio bio) { - struct cgroup_subsys_state css; - if (blk_op_is_passthrough(bio->bi_opf)) return; - if (bio->bi_blkg) { - css = bio_blkcg_css(bio); - bio_associate_blkg_from_css(bio, css); - } else { - rcu_read_lock(); - css = blkcg_css(); - if (!css_tryget_online(css)) - css = NULL; - rcu_read_unlock(); + if (bio_blkcg(bio)) + return; - bio_associate_blkg_from_css(bio, css); - if (css) - css_put(css); - } + rcu_read_lock(); + bio_associate_blkcg_from_css(bio, blkcg_css()); + rcu_read_unlock(); } -EXPORT_SYMBOL_GPL(bio_associate_blkg); +EXPORT_SYMBOL_GPL(bio_associate_blkcg); /** - * bio_clone_blkg_association - clone blkg association from src to dst bio + * bio_clone_blkcg_association - clone blkcg association from src to dst bio * @dst: destination bio * @src: source bio / -void bio_clone_blkg_association(struct bio dst, struct bio src) +void bio_clone_blkcg_association(struct bio dst, struct bio src) { - if (src->bi_blkg) - bio_associate_blkg_from_css(dst, bio_blkcg_css(src)); + if (bio_blkcg(src)) + bio_associate_blkcg_from_css(dst, bio_blkcg_css(src)); } -EXPORT_SYMBOL_GPL(bio_clone_blkg_association); +EXPORT_SYMBOL_GPL(bio_clone_blkcg_association); static int blk_cgroup_io_type(struct bio bio) { @@ -2199,20 +2247,27 @@ static int blk_cgroup_io_type(struct bio bio) void blk_cgroup_bio_start(struct bio bio) { - struct blkcg blkcg = bio->bi_blkg->blkcg; + struct blkcg blkcg = bio_blkcg(bio); + struct blkcg_gq blkg; int rwd = blk_cgroup_io_type(bio), cpu; struct blkg_iostat_set bis; unsigned long flags; if (!cgroup_subsys_on_dfl(io_cgrp_subsys)) return; + if (!blkcg) + return; /* Root-level stats are sourced from system-wide IO stats / if (!cgroup_parent(blkcg->css.cgroup)) return; + blkg = bio_blkg_lookup(bio); + if (!blkg) + return; + cpu = get_cpu(); - bis = per_cpu_ptr(bio->bi_blkg->iostat_cpu, cpu); + bis = per_cpu_ptr(blkg->iostat_cpu, cpu); flags = u64_stats_update_begin_irqsave(&bis->sync); / diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index e67c698391294..eb76cc38b41ba 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -19,6 +19,8 @@ #include <linux/kthread.h> #include <linux/blk-mq.h> #include <linux/llist.h> +#include <linux/rhashtable.h> +#include <linux/rcupdate.h> #include "blk.h" struct blkcg_gq; @@ -56,9 +58,11 @@ struct blkg_iostat_set { struct blkcg_gq { /* Pointer to the associated request_queue / struct request_queue q; + struct rhash_head q_hash_node; struct list_head q_node; struct hlist_node blkcg_node; struct blkcg blkcg; + int blkcg_id; / all non-root blkcg_gq's are guaranteed to have access to parent / struct blkcg_gq parent; @@ -73,14 +77,7 @@ struct blkcg_gq { struct blkg_iostat_set iostat; struct blkg_policy_data pd[BLKCG_MAX_POLS]; -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO - spinlock_t async_bio_lock; - struct bio_list async_bios; -#endif - union { - struct work_struct async_bio_work; - struct work_struct free_work; - }; + struct work_struct free_work; atomic_t use_delay; atomic64_t delay_nsec; @@ -98,8 +95,6 @@ struct blkcg { / If there is block congestion on this cgroup. / atomic_t congestion_count; - struct radix_tree_root blkg_tree; - struct blkcg_gq __rcu blkg_hint; struct hlist_head blkg_list; struct blkcg_policy_data cpd[BLKCG_MAX_POLS]; @@ -111,6 +106,11 @@ struct blkcg { / struct llist_head __percpu lhead; +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO + spinlock_t async_bio_lock; / protects async_bios / + struct bio_list async_bios; + struct work_struct async_bio_work; +#endif #ifdef CONFIG_BLK_CGROUP_FC_APPID char fc_app_id[FC_APPID_LEN]; #endif @@ -124,6 +124,11 @@ static inline struct blkcg css_to_blkcg(struct cgroup_subsys_state css) return css ? container_of(css, struct blkcg, css) : NULL; } +static inline struct blkcg bio_blkcg(struct bio bio) +{ + return bio->bi_blkcg; +} + / * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a * request_queue (q). This is used by blkcg policies which need to track @@ -192,8 +197,10 @@ struct blkcg_policy { extern struct blkcg blkcg_root; extern bool blkcg_debug_stats; +extern const struct rhashtable_params blkg_hash_params; -void blkg_init_queue(struct request_queue q); +int blkg_init_queue(struct request_queue q); +void blkg_exit_queue(struct request_queue q); int blkcg_init_disk(struct gendisk disk); void blkcg_exit_disk(struct gendisk disk); @@ -249,11 +256,33 @@ static inline bool bio_issue_as_root_blkg(struct bio bio) } /** - * blkg_lookup - lookup blkg for the specified blkcg - q pair + * blkg_lookup_any - lookup any blkg for the specified blkcg - q pair + * @blkcg: blkcg of interest + * @q: request_queue of interest + * + * Lookup a blkg for the @blkcg - @q pair, whether it is online or dying. + * + * Must be called in a RCU critical section. + * + * This does not acquire a reference. The caller must already hold one, or + * have the blkg pinned by I/O. + / +static inline struct blkcg_gq blkg_lookup_any(struct blkcg blkcg, + struct request_queue q) +{ + RCU_LOCKDEP_WARN(!rcu_read_lock_held(), + "blkg_lookup_any() requires an RCU read lock"); + + return rhashtable_lookup(&q->blkg_hash, &blkcg->css.id, + blkg_hash_params); +} + +/** + * blkg_lookup - lookup an online blkg for the specified blkcg - q pair * @blkcg: blkcg of interest * @q: request_queue of interest * - * Lookup blkg for the @blkcg - @q pair. + * Lookup an online blkg for the @blkcg - @q pair. * * Must be called in a RCU critical section. / @@ -265,17 +294,15 @@ static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg, if (blkcg == &blkcg_root) return q->root_blkg; - blkg = rcu_dereference_check(blkcg->blkg_hint, - lockdep_is_held(&q->queue_lock)); - if (blkg && blkg->q == q) - return blkg; - - blkg = radix_tree_lookup(&blkcg->blkg_tree, q->id); - if (blkg && blkg->q != q) + blkg = blkg_lookup_any(blkcg, q); + if (blkg && !READ_ONCE(blkg->online)) blkg = NULL; return blkg; } +struct blkcg_gq bio_blkg_lookup(struct bio bio); +struct blkcg_gq bio_blkg(struct bio bio); + /* * blkg_to_pd - get policy private data * @blkg: blkg of interest @@ -343,6 +370,18 @@ static inline void blkg_put(struct blkcg_gq blkg) percpu_ref_put(&blkg->refcnt); } +static inline void bio_clear_blkcg(struct bio bio) +{ + struct blkcg blkcg = bio_blkcg(bio); + + bio_put_blkg_ref(bio); + + if (blkcg) { + css_put(&blkcg->css); + bio->bi_blkcg = NULL; + } +} + /* * blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants * @d_blkg: loop cursor pointing to the current descendant @@ -472,7 +511,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq blkg) / static inline bool blk_cgroup_mergeable(struct request rq, struct bio bio) { - return rq->bio->bi_blkg == bio->bi_blkg && + return bio_blkcg(rq->bio) == bio_blkcg(bio) && bio_issue_as_root_blkg(rq->bio) == bio_issue_as_root_blkg(bio); } @@ -498,8 +537,13 @@ struct blkcg_policy { struct blkcg { }; +static inline struct blkcg bio_blkcg(struct bio bio) { return NULL; } +static inline struct blkcg_gq bio_blkg_lookup(struct bio bio) { return NULL; } +static inline struct blkcg_gq bio_blkg(struct bio bio) { return NULL; } +static inline struct blkcg_gq blkg_lookup_any(struct blkcg blkcg, void key) { return NULL; } static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg, void key) { return NULL; } -static inline void blkg_init_queue(struct request_queue q) { } +static inline int blkg_init_queue(struct request_queue q) { return 0; } +static inline void blkg_exit_queue(struct request_queue q) { } static inline int blkcg_init_disk(struct gendisk disk) { return 0; } static inline void blkcg_exit_disk(struct gendisk disk) { } static inline int blkcg_policy_register(struct blkcg_policy pol) { return 0; } @@ -514,6 +558,7 @@ static inline struct blkg_policy_data blkg_to_pd(struct blkcg_gq blkg, static inline struct blkcg_gq pd_to_blkg(struct blkg_policy_data pd) { return NULL; } static inline void blkg_get(struct blkcg_gq blkg) { } static inline void blkg_put(struct blkcg_gq blkg) { } +static inline void bio_clear_blkcg(struct bio bio) { } static inline void blk_cgroup_bio_start(struct bio bio) { } static inline bool blk_cgroup_mergeable(struct request rq, struct bio bio) { return true; } diff --git a/block/blk-core.c b/block/blk-core.c index 196bccf27f58d..e396a316b8bf6 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -303,6 +303,7 @@ static void blk_free_queue(struct request_queue q) if (queue_is_mq(q)) blk_mq_release(q); + blkg_exit_queue(q); ida_free(&blk_queue_ida, q->id); lockdep_unregister_key(&q->io_lock_cls_key); lockdep_unregister_key(&q->q_lock_cls_key); @@ -481,7 +482,9 @@ struct request_queue blk_alloc_queue(struct queue_limits lim, int node_id) init_waitqueue_head(&q->mq_freeze_wq); mutex_init(&q->mq_freeze_lock); - blkg_init_queue(q); + error = blkg_init_queue(q); + if (error) + goto fail_stats; / * Init percpu_ref in atomic mode so that it's faster to shutdown. @@ -491,7 +494,7 @@ struct request_queue blk_alloc_queue(struct queue_limits lim, int node_id) blk_queue_usage_counter_release, PERCPU_REF_INIT_ATOMIC, GFP_KERNEL); if (error) - goto fail_stats; + goto fail_blkg; lockdep_register_key(&q->io_lock_cls_key); lockdep_register_key(&q->q_lock_cls_key); lockdep_init_map(&q->io_lockdep_map, "&q->q_usage_counter(io)", @@ -510,6 +513,8 @@ struct request_queue blk_alloc_queue(struct queue_limits lim, int node_id) return q; +fail_blkg: + blkg_exit_queue(q); fail_stats: blk_free_queue_stats(q->stats); fail_id: diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c index 2a5c52ab74b4c..5ef4baab444bb 100644 --- a/block/blk-crypto-fallback.c +++ b/block/blk-crypto-fallback.c @@ -187,7 +187,7 @@ static struct bio blk_crypto_alloc_enc_bio(struct bio bio_src, bio->bi_write_hint = bio_src->bi_write_hint; bio->bi_write_stream = bio_src->bi_write_stream; bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; - bio_clone_blkg_association(bio, bio_src); + bio_clone_blkcg_association(bio, bio_src); /* * Move page array up in the allocated memory for the bio vecs as far as diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 2745bffcd5eef..57f2b4d4af20a 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -2686,7 +2686,7 @@ iocg_handle_over_budget(struct rq_qos rqos, struct ioc_gq iocg, static void ioc_rqos_throttle(struct rq_qos rqos, struct bio bio) { - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg_gq blkg = bio_blkg(bio); struct ioc ioc = rqos_to_ioc(rqos); struct ioc_gq iocg = blkg_to_iocg(blkg); struct ioc_now now; @@ -2775,7 +2775,7 @@ static void ioc_rqos_throttle(struct rq_qos rqos, struct bio bio) static void ioc_rqos_merge(struct rq_qos rqos, struct request rq, struct bio bio) { - struct ioc_gq iocg = blkg_to_iocg(bio->bi_blkg); + struct ioc_gq iocg = blkg_to_iocg(bio_blkg(bio)); struct ioc ioc = rqos_to_ioc(rqos); sector_t bio_end = bio_end_sector(bio); struct ioc_now now; @@ -2833,9 +2833,13 @@ static void ioc_rqos_merge(struct rq_qos rqos, struct request rq, static void ioc_rqos_done_bio(struct rq_qos rqos, struct bio bio) { - struct ioc_gq iocg = blkg_to_iocg(bio->bi_blkg); + struct ioc_gq iocg; + + if (!bio->bi_iocost_cost) + return; - if (iocg && bio->bi_iocost_cost) + iocg = blkg_to_iocg(bio_blkg_lookup(bio)); + if (iocg) atomic64_add(bio->bi_iocost_cost, &iocg->done_vtime); } diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 2caa79a008ad1..7220edafd72b6 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -463,7 +463,7 @@ static void check_scale_change(struct iolatency_grp iolat) static void blkcg_iolatency_throttle(struct rq_qos rqos, struct bio bio) { struct blk_iolatency blkiolat = BLKIOLATENCY(rqos); - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg_gq blkg = bio_blkg(bio); bool issue_as_root = bio_issue_as_root_blkg(bio); if (!blkiolat->enabled) @@ -590,11 +590,14 @@ static void blkcg_iolatency_done_bio(struct rq_qos rqos, struct bio bio) bool issue_as_root = bio_issue_as_root_blkg(bio); int inflight = 0; - blkg = bio->bi_blkg; - if (!blkg \|\| !bio_flagged(bio, BIO_QOS_THROTTLED)) + if (!bio_flagged(bio, BIO_QOS_THROTTLED)) return; - iolat = blkg_to_lat(bio->bi_blkg); + blkg = bio_blkg_lookup(bio); + if (!blkg) + return; + + iolat = blkg_to_lat(blkg); if (!iolat) return; diff --git a/block/blk-ioprio.c b/block/blk-ioprio.c index 8fa8bca350629..5e3f8e49550da 100644 --- a/block/blk-ioprio.c +++ b/block/blk-ioprio.c @@ -132,7 +132,7 @@ static struct blkcg_policy ioprio_policy = { void blkcg_set_ioprio(struct bio bio) { - struct ioprio_blkcg blkcg = blkcg_to_ioprio_blkcg(bio->bi_blkg->blkcg); + struct ioprio_blkcg blkcg = blkcg_to_ioprio_blkcg(bio_blkcg(bio)); u16 prio; if (!blkcg \|\| blkcg->prio_policy == POLICY_NO_CHANGE) diff --git a/block/blk-throttle.c b/block/blk-throttle.c index ffc3b70065d4b..3828c38579002 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -1792,7 +1792,7 @@ static bool tg_within_limit(struct throtl_grp tg, struct bio bio, bool rw) bool __blk_throtl_bio(struct bio bio) { struct request_queue q = bdev_get_queue(bio->bi_bdev); - struct blkcg_gq blkg = bio->bi_blkg; + struct blkcg_gq blkg = bio_blkg(bio); struct throtl_qnode qn = NULL; struct throtl_grp tg = blkg_to_tg(blkg); struct throtl_service_queue sq; diff --git a/block/blk-throttle.h b/block/blk-throttle.h index 1b5775771e078..45f30179fd9c0 100644 --- a/block/blk-throttle.h +++ b/block/blk-throttle.h @@ -171,7 +171,7 @@ static inline bool blk_should_throtl(struct bio bio) if (!blk_throtl_activated(bio->bi_bdev->bd_queue)) return false; - tg = blkg_to_tg(bio->bi_blkg); + tg = blkg_to_tg(bio_blkg(bio)); if (!cgroup_subsys_on_dfl(io_cgrp_subsys)) { if (!bio_flagged(bio, BIO_CGROUP_ACCT)) { bio_set_flag(bio, BIO_CGROUP_ACCT); diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index 3fa3b13a410f4..c0f945b8d9413 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -848,7 +848,7 @@ static CLOSURE_CALLBACK(cached_dev_read_done) s->iop.bio->bi_iter.bi_sector = s->cache_miss->bi_iter.bi_sector; s->iop.bio->bi_iter.bi_size = s->insert_bio_sectors << 9; - bio_clone_blkg_association(s->iop.bio, s->cache_miss); + bio_clone_blkcg_association(s->iop.bio, s->cache_miss); bch_bio_map(s->iop.bio, NULL); bio_copy_data(s->cache_miss, s->iop.bio); diff --git a/drivers/md/dm.c b/drivers/md/dm.c index d413bfaf35278..cd68eec77f5a1 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1373,7 +1373,7 @@ void dm_submit_bio_remap(struct bio clone, struct bio tgt_clone) if (!tgt_clone) tgt_clone = clone; - bio_clone_blkg_association(tgt_clone, io->orig_bio); + bio_clone_blkcg_association(tgt_clone, io->orig_bio); / * Account io->origin_bio to DM dev on behalf of target diff --git a/drivers/md/md.c b/drivers/md/md.c index 680b34a63cb3d..fd78d6cc8da17 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -9386,7 +9386,7 @@ void md_submit_discard_bio(struct mddev mddev, struct md_rdev rdev, return; bio_chain(discard_bio, bio); - bio_clone_blkg_association(discard_bio, bio); + bio_clone_blkcg_association(discard_bio, bio); mddev_trace_remap(mddev, discard_bio, bio->bi_iter.bi_sector); submit_bio_noacct(discard_bio); } diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c index 4176046627beb..54e4adb0ccb7d 100644 --- a/drivers/nvdimm/nd_virtio.c +++ b/drivers/nvdimm/nd_virtio.c @@ -121,7 +121,7 @@ int async_pmem_flush(struct nd_region nd_region, struct bio bio) if (!child) return -ENOMEM; - bio_clone_blkg_association(child, bio); + bio_clone_blkcg_association(child, bio); child->bi_iter.bi_sector = -1; bio_chain(child, bio); submit_bio(child); diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c index 6dabe73ad790d..6512dbd9516fc 100644 --- a/fs/gfs2/lops.c +++ b/fs/gfs2/lops.c @@ -484,7 +484,7 @@ static struct bio gfs2_chain_bio(struct bio prev, unsigned int nr_iovecs, struct bio new; new = bio_alloc(prev->bi_bdev, nr_iovecs, opf, GFP_NOIO); - bio_clone_blkg_association(new, prev); + bio_clone_blkcg_association(new, prev); new->bi_iter.bi_sector = sector; bio_chain(new, prev); submit_bio(prev); @@ -1114,4 +1114,3 @@ const struct gfs2_log_operations gfs2_log_ops[] = { &gfs2_revoke_lops, NULL, }; - diff --git a/include/linux/bio.h b/include/linux/bio.h index bb3235497e673..11637f85c38d4 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -542,18 +542,20 @@ static inline void bio_release_pages(struct bio bio, bool mark_dirty) disk_devt((bio)->bi_bdev->bd_disk) #ifdef CONFIG_BLK_CGROUP -void bio_associate_blkg(struct bio bio); -void bio_associate_blkg_from_css(struct bio bio, - struct cgroup_subsys_state css); -void bio_clone_blkg_association(struct bio dst, struct bio src); +void bio_associate_blkcg(struct bio bio); +void bio_associate_blkcg_from_css(struct bio bio, + struct cgroup_subsys_state css); +void bio_clone_blkcg_association(struct bio dst, struct bio src); +void bio_put_blkg_ref(struct bio bio); void blkcg_punt_bio_submit(struct bio bio); #else / CONFIG_BLK_CGROUP / -static inline void bio_associate_blkg(struct bio bio) { } -static inline void bio_associate_blkg_from_css(struct bio bio, - struct cgroup_subsys_state css) +static inline void bio_associate_blkcg(struct bio bio) { } +static inline void bio_associate_blkcg_from_css(struct bio bio, + struct cgroup_subsys_state css) { } -static inline void bio_clone_blkg_association(struct bio dst, - struct bio src) { } +static inline void bio_clone_blkcg_association(struct bio dst, + struct bio src) { } +static inline void bio_put_blkg_ref(struct bio bio) { } static inline void blkcg_punt_bio_submit(struct bio bio) { submit_bio(bio); @@ -563,10 +565,12 @@ static inline void blkcg_punt_bio_submit(struct bio bio) static inline void bio_set_dev(struct bio bio, struct block_device bdev) { bio_clear_flag(bio, BIO_REMAPPED); - if (bio->bi_bdev != bdev) + if (bio->bi_bdev != bdev) { + bio_put_blkg_ref(bio); bio_clear_flag(bio, BIO_BPS_THROTTLED); + } bio->bi_bdev = bdev; - bio_associate_blkg(bio); + bio_associate_blkcg(bio); } /* diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index d49d97a050d0e..53ba307b91cb3 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -246,12 +246,10 @@ struct bio { void bi_private; #ifdef CONFIG_BLK_CGROUP / - * Represents the association of the css and request_queue for the bio. - * If a bio goes direct to device, it will not have a blkg as it will - * not have a request_queue associated with it. The reference is put - * on release of the bio. + * Represents the blkcg css association for the bio. The reference is + * put on release of the bio. / - struct blkcg_gq bi_blkg; + struct blkcg bi_blkcg; / Time that this bio was issued. / u64 issue_time_ns; #ifdef CONFIG_BLK_CGROUP_IOCOST @@ -309,6 +307,7 @@ enum { BIO_TRACE_COMPLETION, / bio_endio() should trace the final completion * of this bio. / BIO_CGROUP_ACCT, / has been accounted to a cgroup / + BIO_BLKG_REF, / bio pins the associated blkg / BIO_QOS_THROTTLED, / bio went through rq_qos throttle path / / * This bio has completed bps throttling at the single tg granularity, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 20cb8ed7d987a..f3b0e39d3f6b4 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -27,6 +27,7 @@ #include <linux/xarray.h> #include <linux/file.h> #include <linux/lockdep.h> +#include <linux/rhashtable-types.h> struct module; struct request_queue; @@ -581,6 +582,7 @@ struct request_queue { #ifdef CONFIG_BLK_CGROUP DECLARE_BITMAP (blkcg_pols, BLKCG_MAX_POLS); struct blkcg_gq root_blkg; + struct rhashtable blkg_hash; struct list_head blkg_list; struct mutex blkcg_mutex; #endif diff --git a/include/linux/writeback.h b/include/linux/writeback.h index 62552a2ce5b9e..4f869fe9cc90f 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -262,7 +262,7 @@ static inline void wbc_init_bio(struct writeback_control wbc, struct bio bio) regular writeback instead of writing things out itself. / if (wbc->wb) - bio_associate_blkg_from_css(bio, wbc->wb->blkcg_css); + bio_associate_blkcg_from_css(bio, wbc->wb->blkcg_css); } void inode_switch_wbs_work_fn(struct work_struct work); diff --git a/mm/page_io.c b/mm/page_io.c index b23f494fcc83d..a4a1b8a8a0e38 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -316,7 +316,7 @@ static inline void count_swpout_vm_event(struct folio folio) } #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) -static void bio_associate_blkg_from_page(struct bio bio, struct folio folio) +static void bio_associate_blkcg_from_page(struct bio bio, struct folio folio) { struct cgroup_subsys_state css; struct mem_cgroup memcg; @@ -331,12 +331,15 @@ static void bio_associate_blkg_from_page(struct bio bio, struct folio folio) css = NULL; rcu_read_unlock(); - bio_associate_blkg_from_css(bio, css); + bio_associate_blkcg_from_css(bio, css); if (css) css_put(css); } #else -#define bio_associate_blkg_from_page(bio, folio) do { } while (0) +static inline void bio_associate_blkcg_from_page(struct bio bio, + struct folio folio) +{ +} #endif / CONFIG_MEMCG && CONFIG_BLK_CGROUP / struct swap_iocb { @@ -436,7 +439,7 @@ static void swap_writepage_bdev_sync(struct folio folio, bio.bi_iter.bi_sector = swap_folio_sector(folio); bio_add_folio_nofail(&bio, folio, folio_size(folio), 0); - bio_associate_blkg_from_page(&bio, folio); + bio_associate_blkcg_from_page(&bio, folio); count_swpout_vm_event(folio); folio_start_writeback(folio); @@ -456,7 +459,7 @@ static void swap_writepage_bdev_async(struct folio *folio, bio->bi_end_io = end_swap_bio_write; bio_add_folio_nofail(bio, folio, folio_size(folio), 0); - bio_associate_blkg_from_page(bio, folio); + bio_associate_blkcg_from_page(bio, folio); count_swpout_vm_event(folio); folio_start_writeback(folio); folio_unlock(folio);
75/3	2026/08/18 08:09	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 21142 output: 279 thoughts: 1171 Thoughts: KMSAN Assessment: Block Cgroup Association Refactor Okay, so I've got this patch series that's overhauling how `bio` structures relate to block cgroups, a common area for performance and isolation concerns. It looks like the main thrust is to move from a direct `bio->bi_blkg` pointer to a `bio->bi_blkcg` pointer and use a hash table (`q->blkg_hash`) to manage the `blkcg_gq` (now `blkg`) lookups, instead of that radix tree. This touches a lot of code, from the block layer down to cgroups and even MD, bcache, and swap, so we need to be very careful. My primary concern, as always, is memory safety and potential leaks. First, I checked for uninitialized memory risks. The diffs show that `bio->bi_blkcg` is properly initialized to `NULL` in `bio_init()`. Then `bio_clear_blkcg()` handles the cleanup by decrementing the refcounts of the `blkcg` and `blkg`. The `blkg_hash` is initialized in `blkg_init_queue()` during queue setup. `blkcg_gq` structures are allocated with `kzalloc_node`, so they're zeroed as expected. I'm not seeing any new structures being copied to userspace, no uninitialized variables used in conditionals, and no complex data structures allocated without proper initialization. So, from the initial overview, it looks pretty safe. Next, I specifically looked for KMSAN-related concerns. KMSAN (Kernel Memory Sanitizer) is designed to flag uninitialized memory reads. This patch mostly changes pointers and uses a hash table for lookups, as well as introducing a `BIO_BLKG_REF` flag to the `bio->bi_flags` structure. Since we're just shifting how we manage the association between `bio` and cgroups, I can't see any new avenues for leaking uninitialized kernel memory or reading from it. The existing allocations are handled correctly. So, the conclusion is: This patch series should not require a dedicated KMSAN fuzzing session. The changes are largely structural refactorings concerning how the block cgroup association is managed within the kernel. The introduction of the `bi_blkcg` pointer, the hash table for lookups, and the modified initialization/cleanup routines are sound, and memory is properly initialized and cleaned up. Potential bugs would most likely appear as use-after-free conditions, memory leaks or NULL pointer dereferences that would be caught by existing KASAN and other tools. There's no indication of any uninitialized memory reads or kernel information leaking to userspace due to these changes. The `set-results` tool should handle the final outcome based on this assessment.
76/3	2026/08/18 08:09	tool	set-results	0m Args: { "KMSANReasoning": "The patch refactors the block cgroup (blkcg) association for `bio` structures, replacing the direct `bi_blkg` pointer with `bi_blkcg` and dynamically looking up or pinning the queue-local `blkcg_gq` (blkg) using a hash table instead of a radix tree.\n\nThese changes involve modifying internal kernel structures (`bio`, `request_queue`, `blkcg_gq`) and their initialization/cleanup routines (`bio_init`, `bio_uninit`, `bio_clear_blkcg`). No new structures are copied to user space, nor are there any new buffers or complex data structures allocated without proper initialization. The modified fields in `bio` are explicitly initialized to `NULL` or appropriate values in `bio_init()`.\n\nAny bugs introduced by these changes would likely manifest as NULL pointer dereferences, use-after-free (e.g., incorrect reference counting of `blkcg` or `blkg`), or memory leaks, all of which are effectively caught by standard KASAN and other default debugging tools. There is no risk of uninitialized memory reads or kernel-to-user info-leaks that would necessitate a dedicated KMSAN fuzzing session.", "NeedsKMSAN": false } Results: { "KMSANReasoning": "The patch refactors the block cgroup (blkcg) association for `bio` structures, replacing the direct `bi_blkg` pointer with `bi_blkcg` and dynamically looking up or pinning the queue-local `blkcg_gq` (blkg) using a hash table instead of a radix tree.\n\nThese changes involve modifying internal kernel structures (`bio`, `request_queue`, `blkcg_gq`) and their initialization/cleanup routines (`bio_init`, `bio_uninit`, `bio_clear_blkcg`). No new structures are copied to user space, nor are there any new buffers or complex data structures allocated without proper initialization. The modified fields in `bio` are explicitly initialized to `NULL` or appropriate values in `bio_init()`.\n\nAny bugs introduced by these changes would likely manifest as NULL pointer dereferences, use-after-free (e.g., incorrect reference counting of `blkcg` or `blkg`), or memory leaks, all of which are effectively caught by standard KASAN and other default debugging tools. There is no risk of uninitialized memory reads or kernel-to-user info-leaks that would necessitate a dedicated KMSAN fuzzing session.", "NeedsKMSAN": false }

{
  "EnableConfigs": [
    "BLK_CGROUP",
    "IOSCHED_BFQ",
    "BFQ_GROUP_IOSCHED",
    "BLK_CGROUP_IOCOST",
    "BLK_CGROUP_IOLATENCY",
    "BLK_CGROUP_IOPRIO",
    "NVME_FC",
    "BLK_CGROUP_FC_APPID",
    "BTRFS_FS"
  ],
  "FocusSymbols": [
    "blkg_conf_prep",
    "blkcg_punt_bio_submit",
    "blkg_init_queue",
    "blkg_exit_queue",
    "blkcg_init_disk",
    "blkcg_exit_disk"
  ],
  "KMSANReasoning": "The patch refactors the block cgroup (blkcg) association for `bio` structures, replacing the direct `bi_blkg` pointer with `bi_blkcg` and dynamically looking up or pinning the queue-local `blkcg_gq` (blkg) using a hash table instead of a radix tree.\n\nThese changes involve modifying internal kernel structures (`bio`, `request_queue`, `blkcg_gq`) and their initialization/cleanup routines (`bio_init`, `bio_uninit`, `bio_clear_blkcg`). No new structures are copied to user space, nor are there any new buffers or complex data structures allocated without proper initialization. The modified fields in `bio` are explicitly initialized to `NULL` or appropriate values in `bio_init()`.\n\nAny bugs introduced by these changes would likely manifest as NULL pointer dereferences, use-after-free (e.g., incorrect reference counting of `blkcg` or `blkg`), or memory leaks, all of which are effectively caught by standard KASAN and other default debugging tools. There is no risk of uninitialized memory reads or kernel-to-user info-leaks that would necessitate a dedicated KMSAN fuzzing session.",
  "NeedsKMSAN": false,
  "Reasoning": "The patch significantly refactors the block cgroup (blkcg) subsystem by changing how `bio`s are associated with block cgroups. Instead of directly storing a queue-specific `blkcg_gq` (blkg) in `bio-\u003ebi_blkg`, it now stores the generic `blkcg` in `bio-\u003ebi_blkcg` and dynamically looks up or pins the queue-specific `blkg` using a new `BIO_BLKG_REF` flag. It also replaces the `radix_tree` used for `blkg` lookups with an `rhashtable`. These are core functional changes to the block layer and cgroup IO controllers, affecting IO scheduling, throttling, and accounting.",
  "WorthFuzzing": true
}

{
  "PatchDiff": "commit abc0b1a65e87d979eec053ca54836b90fdec9729\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate:   Tue Aug 18 08:04:15 2026 +0000\n\n    syz-cluster: applied patch under review\n\ndiff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst\nindex 14b8c571c0d14..bbd79931d5ab5 100644\n--- a/Documentation/admin-guide/cgroup-v2.rst\n+++ b/Documentation/admin-guide/cgroup-v2.rst\n@@ -3235,7 +3235,7 @@ the configuration, the bio may be executed at a lower priority and if\n the writeback session is holding shared resources, e.g. a journal\n entry, may lead to priority inversion.  There is no one easy solution\n for the problem.  Filesystems can try to work around specific problem\n-cases by skipping wbc_init_bio() and using bio_associate_blkg()\n+cases by skipping wbc_init_bio() and using bio_associate_blkcg()\n directly.\n \n \ndiff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c\nindex ef9d63e8542ed..4a3975f9ff744 100644\n--- a/block/bfq-cgroup.c\n+++ b/block/bfq-cgroup.c\n@@ -363,11 +363,13 @@ void bfqg_and_blkg_put(struct bfq_group *bfqg)\n \n void bfqg_stats_update_legacy_io(struct request_queue *q, struct request *rq)\n {\n-\tstruct bfq_group *bfqg = blkg_to_bfqg(rq-\u003ebio-\u003ebi_blkg);\n+\tstruct blkcg_gq *blkg = bio_blkg_lookup(rq-\u003ebio);\n+\tstruct bfq_group *bfqg;\n \n-\tif (!bfqg)\n+\tif (!blkg)\n \t\treturn;\n \n+\tbfqg = blkg_to_bfqg(blkg);\n \tblkg_rwstat_add(\u0026bfqg-\u003estats.bytes, rq-\u003ecmd_flags, blk_rq_bytes(rq));\n \tblkg_rwstat_add(\u0026bfqg-\u003estats.ios, rq-\u003ecmd_flags, 1);\n }\n@@ -606,7 +608,7 @@ static void bfq_link_bfqg(struct bfq_data *bfqd, struct bfq_group *bfqg)\n \n struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio)\n {\n-\tstruct blkcg_gq *blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg_gq *blkg = bio_blkg_lookup(bio);\n \tstruct bfq_group *bfqg;\n \n \twhile (blkg) {\n@@ -614,15 +616,17 @@ struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio)\n \t\t\tblkg = blkg-\u003eparent;\n \t\t\tcontinue;\n \t\t}\n+\n \t\tbfqg = blkg_to_bfqg(blkg);\n \t\tif (bfqg-\u003epd.online) {\n-\t\t\tbio_associate_blkg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\n+\t\t\tbio_associate_blkcg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\n \t\t\treturn bfqg;\n \t\t}\n \t\tblkg = blkg-\u003eparent;\n \t}\n-\tbio_associate_blkg_from_css(bio,\n-\t\t\t\t\u0026bfqg_to_blkg(bfqd-\u003eroot_group)-\u003eblkcg-\u003ecss);\n+\n+\tblkg = bfqg_to_blkg(bfqd-\u003eroot_group);\n+\tbio_associate_blkcg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\n \treturn bfqd-\u003eroot_group;\n }\n \ndiff --git a/block/bfq-iosched.c b/block/bfq-iosched.c\nindex 0f75301b3115f..d4f657bc7a4e5 100644\n--- a/block/bfq-iosched.c\n+++ b/block/bfq-iosched.c\n@@ -128,6 +128,7 @@\n \n #include \"elevator.h\"\n #include \"blk.h\"\n+#include \"blk-cgroup.h\"\n #include \"blk-mq.h\"\n #include \"blk-mq-sched.h\"\n #include \"bfq-iosched.h\"\n@@ -2452,15 +2453,16 @@ static bool bfq_bio_merge(struct request_queue *q, struct bio *bio,\n \tstruct request *free = NULL;\n \tbool ret;\n \n+#ifdef CONFIG_BFQ_GROUP_IOSCHED\n+\t/* blkg creation takes q-\u003equeue_lock, so do it before bfqd-\u003elock. */\n+\tif (bic)\n+\t\tbio_blkg(bio);\n+#endif\n+\n \tspin_lock_irq(\u0026bfqd-\u003elock);\n \n \tif (bic) {\n-\t\t/*\n-\t\t * Make sure cgroup info is uptodate for current process before\n-\t\t * considering the merge.\n-\t\t */\n \t\tbfq_bic_update_cgroup(bic, bio);\n-\n \t\tbfqd-\u003ebio_bfqq = bic_to_bfqq(bic, op_is_sync(bio-\u003ebi_opf),\n \t\t\t\t\t     bfq_actuator_index(bfqd, bio));\n \t} else {\n@@ -6245,6 +6247,13 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,\n \tLIST_HEAD(free);\n \n #ifdef CONFIG_BFQ_GROUP_IOSCHED\n+\t/*\n+\t * Pin the blkg used to look up bfqg.  If this is the first IO for\n+\t * the blkcg on this queue, create the bfqg before holding bfqd-\u003elock.\n+\t */\n+\tif (rq-\u003ebio \u0026\u0026 !bio_flagged(rq-\u003ebio, BIO_BLKG_REF))\n+\t\tbio_blkg(rq-\u003ebio);\n+\n \tif (!cgroup_subsys_on_dfl(io_cgrp_subsys) \u0026\u0026 rq-\u003ebio)\n \t\tbfqg_stats_update_legacy_io(q, rq);\n #endif\ndiff --git a/block/bio.c b/block/bio.c\nindex 898b2f5ef8c81..6c84042adf5dd 100644\n--- a/block/bio.c\n+++ b/block/bio.c\n@@ -179,12 +179,7 @@ static inline gfp_t try_alloc_gfp(gfp_t gfp)\n \n void bio_uninit(struct bio *bio)\n {\n-#ifdef CONFIG_BLK_CGROUP\n-\tif (bio-\u003ebi_blkg) {\n-\t\tblkg_put(bio-\u003ebi_blkg);\n-\t\tbio-\u003ebi_blkg = NULL;\n-\t}\n-#endif\n+\tbio_clear_blkcg(bio);\n \tif (bio_integrity(bio))\n \t\tbio_integrity_free(bio);\n \n@@ -233,10 +228,10 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,\n \tbio-\u003ebi_end_io = NULL;\n \tbio-\u003ebi_private = NULL;\n #ifdef CONFIG_BLK_CGROUP\n-\tbio-\u003ebi_blkg = NULL;\n+\tbio-\u003ebi_blkcg = NULL;\n \tbio-\u003eissue_time_ns = 0;\n \tif (bdev)\n-\t\tbio_associate_blkg(bio);\n+\t\tbio_associate_blkcg(bio);\n #ifdef CONFIG_BLK_CGROUP_IOCOST\n \tbio-\u003ebi_iocost_cost = 0;\n #endif\n@@ -281,7 +276,7 @@ void bio_reset(struct bio *bio, struct block_device *bdev, blk_opf_t opf)\n \tbio-\u003ebi_io_vec = bv;\n \tbio-\u003ebi_bdev = bdev;\n \tif (bio-\u003ebi_bdev)\n-\t\tbio_associate_blkg(bio);\n+\t\tbio_associate_blkcg(bio);\n \tbio-\u003ebi_opf = opf;\n }\n EXPORT_SYMBOL(bio_reset);\n@@ -866,7 +861,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)\n \t\tif (bio-\u003ebi_bdev == bio_src-\u003ebi_bdev \u0026\u0026\n \t\t    bio_flagged(bio_src, BIO_REMAPPED))\n \t\t\tbio_set_flag(bio, BIO_REMAPPED);\n-\t\tbio_clone_blkg_association(bio, bio_src);\n+\t\tbio_clone_blkcg_association(bio, bio_src);\n \t}\n \n \tif (bio_crypt_clone(bio, bio_src, gfp) \u003c 0)\n@@ -1910,17 +1905,12 @@ void bio_endio(struct bio *bio)\n \t\tgoto again;\n \t}\n \n-#ifdef CONFIG_BLK_CGROUP\n \t/*\n \t * Release cgroup info.  We shouldn't have to do this here, but quite\n \t * a few callers of bio_init fail to call bio_uninit, so we cover up\n \t * for that here at least for now.\n \t */\n-\tif (bio-\u003ebi_blkg) {\n-\t\tblkg_put(bio-\u003ebi_blkg);\n-\t\tbio-\u003ebi_blkg = NULL;\n-\t}\n-#endif\n+\tbio_clear_blkcg(bio);\n \n \tif (bio_flagged(bio, BIO_COMPLETE_IN_TASK) \u0026\u0026 bio_in_atomic())\n \t\t__bio_complete_in_task(bio);\ndiff --git a/block/blk-cgroup-fc-appid.c b/block/blk-cgroup-fc-appid.c\nindex 3ec21333f393b..7589c62099898 100644\n--- a/block/blk-cgroup-fc-appid.c\n+++ b/block/blk-cgroup-fc-appid.c\n@@ -50,8 +50,14 @@ EXPORT_SYMBOL_GPL(blkcg_set_fc_appid);\n  */\n char *blkcg_get_fc_appid(struct bio *bio)\n {\n-\tif (!bio-\u003ebi_blkg || bio-\u003ebi_blkg-\u003eblkcg-\u003efc_app_id[0] == '\\0')\n+\tstruct blkcg *blkcg = bio_blkcg(bio);\n+\n+\tif (!blkcg)\n+\t\treturn NULL;\n+\n+\tif (blkcg-\u003efc_app_id[0] == '\\0')\n \t\treturn NULL;\n-\treturn bio-\u003ebi_blkg-\u003eblkcg-\u003efc_app_id;\n+\n+\treturn blkcg-\u003efc_app_id;\n }\n EXPORT_SYMBOL_GPL(blkcg_get_fc_appid);\ndiff --git a/block/blk-cgroup.c b/block/blk-cgroup.c\nindex 2b5c29434e426..2c758c26bce60 100644\n--- a/block/blk-cgroup.c\n+++ b/block/blk-cgroup.c\n@@ -66,6 +66,13 @@ static DEFINE_RAW_SPINLOCK(blkg_stat_lock);\n \n #define BLKG_DESTROY_BATCH_SIZE  64\n \n+const struct rhashtable_params blkg_hash_params = {\n+\t.key_len\t\t= sizeof_field(struct blkcg_gq, blkcg_id),\n+\t.key_offset\t\t= offsetof(struct blkcg_gq, blkcg_id),\n+\t.head_offset\t\t= offsetof(struct blkcg_gq, q_hash_node),\n+\t.automatic_shrinking\t= true,\n+};\n+\n /*\n  * Lockless lists for tracking IO stats update\n  *\n@@ -173,10 +180,6 @@ static void __blkg_release(struct rcu_head *rcu)\n {\n \tstruct blkcg_gq *blkg = container_of(rcu, struct blkcg_gq, rcu_head);\n \n-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n-\tWARN_ON(!bio_list_empty(\u0026blkg-\u003easync_bios));\n-#endif\n-\n \tblkg_free(blkg);\n }\n \n@@ -194,6 +197,16 @@ static void blkg_release(struct percpu_ref *ref)\n \tstruct blkcg *blkcg = blkg-\u003eblkcg;\n \tint cpu;\n \n+\t/*\n+\t * A blkg that was never inserted into q-\u003eblkg_list has no hash\n+\t * entry.  This happens when allocation or creation fails before\n+\t * rhashtable_insert_fast() succeeds.\n+\t */\n+\tif (!list_empty(\u0026blkg-\u003eq_node))\n+\t\tWARN_ON_ONCE(rhashtable_remove_fast(\u0026blkg-\u003eq-\u003eblkg_hash,\n+\t\t\t\t\t\t    \u0026blkg-\u003eq_hash_node,\n+\t\t\t\t\t\t    blkg_hash_params));\n+\n \t/*\n \t * Flush all the non-empty percpu lockless lists before releasing\n \t * us, given these stat belongs to us.\n@@ -209,19 +222,18 @@ static void blkg_release(struct percpu_ref *ref)\n #ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n static struct workqueue_struct *blkcg_punt_bio_wq;\n \n-static void blkg_async_bio_workfn(struct work_struct *work)\n+static void blkcg_async_bio_workfn(struct work_struct *work)\n {\n-\tstruct blkcg_gq *blkg = container_of(work, struct blkcg_gq,\n-\t\t\t\t\t     async_bio_work);\n+\tstruct blkcg *blkcg = container_of(work, struct blkcg, async_bio_work);\n \tstruct bio_list bios = BIO_EMPTY_LIST;\n \tstruct bio *bio;\n \tstruct blk_plug plug;\n \tbool need_plug = false;\n \n-\t/* as long as there are pending bios, @blkg can't go away */\n-\tspin_lock(\u0026blkg-\u003easync_bio_lock);\n-\tbio_list_merge_init(\u0026bios, \u0026blkg-\u003easync_bios);\n-\tspin_unlock(\u0026blkg-\u003easync_bio_lock);\n+\t/* as long as there are pending bios, @blkcg can't go away */\n+\tspin_lock(\u0026blkcg-\u003easync_bio_lock);\n+\tbio_list_merge_init(\u0026bios, \u0026blkcg-\u003easync_bios);\n+\tspin_unlock(\u0026blkcg-\u003easync_bio_lock);\n \n \t/* start plug only when bio_list contains at least 2 bios */\n \tif (bios.head \u0026\u0026 bios.head-\u003ebi_next) {\n@@ -242,15 +254,15 @@ static void blkg_async_bio_workfn(struct work_struct *work)\n  */\n void blkcg_punt_bio_submit(struct bio *bio)\n {\n-\tstruct blkcg_gq *blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg *blkcg = bio_blkcg(bio);\n \n-\tif (blkg-\u003eparent) {\n-\t\tspin_lock(\u0026blkg-\u003easync_bio_lock);\n-\t\tbio_list_add(\u0026blkg-\u003easync_bios, bio);\n-\t\tspin_unlock(\u0026blkg-\u003easync_bio_lock);\n-\t\tqueue_work(blkcg_punt_bio_wq, \u0026blkg-\u003easync_bio_work);\n+\tif (blkcg \u0026\u0026 cgroup_parent(blkcg-\u003ecss.cgroup)) {\n+\t\tspin_lock(\u0026blkcg-\u003easync_bio_lock);\n+\t\tbio_list_add(\u0026blkcg-\u003easync_bios, bio);\n+\t\tspin_unlock(\u0026blkcg-\u003easync_bio_lock);\n+\t\tqueue_work(blkcg_punt_bio_wq, \u0026blkcg-\u003easync_bio_work);\n \t} else {\n-\t\t/* never bounce for the root cgroup */\n+\t\t/* Never bounce if there is no non-root blkcg to queue on. */\n \t\tsubmit_bio(bio);\n \t}\n }\n@@ -278,9 +290,13 @@ subsys_initcall(blkcg_punt_bio_init);\n  */\n struct cgroup_subsys_state *bio_blkcg_css(struct bio *bio)\n {\n-\tif (!bio || !bio-\u003ebi_blkg)\n+\tstruct blkcg *blkcg;\n+\n+\tif (!bio)\n \t\treturn NULL;\n-\treturn \u0026bio-\u003ebi_blkg-\u003eblkcg-\u003ecss;\n+\n+\tblkcg = bio_blkcg(bio);\n+\treturn blkcg ? \u0026blkcg-\u003ecss : NULL;\n }\n EXPORT_SYMBOL_GPL(bio_blkcg_css);\n \n@@ -327,12 +343,8 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct gendisk *disk,\n \tblkg-\u003eq = disk-\u003equeue;\n \tINIT_LIST_HEAD(\u0026blkg-\u003eq_node);\n \tblkg-\u003eblkcg = blkcg;\n+\tblkg-\u003eblkcg_id = blkcg-\u003ecss.id;\n \tblkg-\u003eiostat.blkg = blkg;\n-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n-\tspin_lock_init(\u0026blkg-\u003easync_bio_lock);\n-\tbio_list_init(\u0026blkg-\u003easync_bios);\n-\tINIT_WORK(\u0026blkg-\u003easync_bio_work, blkg_async_bio_workfn);\n-#endif\n \n \tu64_stats_init(\u0026blkg-\u003eiostat.sync);\n \tfor_each_possible_cpu(cpu) {\n@@ -423,7 +435,8 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,\n \n \t/* insert */\n \tspin_lock(\u0026blkcg-\u003elock);\n-\tret = radix_tree_insert(\u0026blkcg-\u003eblkg_tree, disk-\u003equeue-\u003eid, blkg);\n+\tret = rhashtable_insert_fast(\u0026disk-\u003equeue-\u003eblkg_hash,\n+\t\t\t\t     \u0026blkg-\u003eq_hash_node, blkg_hash_params);\n \tif (likely(!ret)) {\n \t\thlist_add_head_rcu(\u0026blkg-\u003eblkcg_node, \u0026blkcg-\u003eblkg_list);\n \t\tlist_add(\u0026blkg-\u003eq_node, \u0026disk-\u003equeue-\u003eblkg_list);\n@@ -454,6 +467,17 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,\n \treturn ERR_PTR(ret);\n }\n \n+/*\n+ * The root blkg holds a live reference while the disk is active, so walking\n+ * the parent chain always finds a blkg which can be pinned.\n+ */\n+static struct blkcg_gq *blkg_lookup_tryget(struct blkcg_gq *blkg)\n+{\n+\twhile (!blkg_tryget(blkg))\n+\t\tblkg = blkg-\u003eparent;\n+\treturn blkg;\n+}\n+\n /**\n  * blkg_lookup_create - lookup blkg, try to create one if not there\n  * @blkcg: blkcg of interest\n@@ -461,11 +485,13 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,\n  *\n  * Lookup blkg for the @blkcg - @disk pair.  If it doesn't exist, try to\n  * create one.  blkg creation is performed recursively from blkcg_root such\n- * that all non-root blkg's have access to the parent blkg.  This function\n- * should be called under RCU read lock and takes @disk-\u003equeue-\u003equeue_lock.\n+ * that all non-root blkg's have access to the parent blkg.\n  *\n- * Returns the blkg or the closest blkg if blkg_create() fails as it walks\n- * down from root.\n+ * Must be called with @disk-\u003equeue-\u003equeue_lock held.\n+ *\n+ * Returns the closest blkg with an extra reference acquired.  If\n+ * blkg_create() fails while walking down from root, the returned blkg may\n+ * belong to an ancestor of @blkcg.  This function never returns %NULL.\n  */\n static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,\n \t\tstruct gendisk *disk)\n@@ -473,12 +499,12 @@ static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,\n \tstruct request_queue *q = disk-\u003equeue;\n \tstruct blkcg_gq *blkg;\n \n+\tlockdep_assert_held(\u0026q-\u003equeue_lock);\n+\n \trcu_read_lock();\n \tblkg = blkg_lookup(blkcg, q);\n \tif (blkg) {\n-\t\tif (blkcg != \u0026blkcg_root \u0026\u0026\n-\t\t    blkg != rcu_dereference(blkcg-\u003eblkg_hint))\n-\t\t\trcu_assign_pointer(blkcg-\u003eblkg_hint, blkg);\n+\t\tblkg = blkg_lookup_tryget(blkg);\n \t\trcu_read_unlock();\n \t\treturn blkg;\n \t}\n@@ -514,7 +540,7 @@ static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,\n \t\t\tbreak;\n \t}\n \n-\treturn blkg;\n+\treturn blkg_lookup_tryget(blkg);\n }\n \n static void blkg_destroy(struct blkcg_gq *blkg)\n@@ -546,17 +572,8 @@ static void blkg_destroy(struct blkcg_gq *blkg)\n \n \tblkg-\u003eonline = false;\n \n-\tradix_tree_delete(\u0026blkcg-\u003eblkg_tree, blkg-\u003eq-\u003eid);\n \thlist_del_init_rcu(\u0026blkg-\u003eblkcg_node);\n \n-\t/*\n-\t * Both setting lookup hint to and clearing it from @blkg are done\n-\t * under queue_lock.  If it's not pointing to @blkg now, it never\n-\t * will.  Hint assignment itself can race safely.\n-\t */\n-\tif (rcu_access_pointer(blkcg-\u003eblkg_hint) == blkg)\n-\t\trcu_assign_pointer(blkcg-\u003eblkg_hint, NULL);\n-\n \t/*\n \t * Put the reference taken at the time of creation so that when all\n \t * queues are gone, group can be destroyed.\n@@ -882,18 +899,12 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,\n \t\t\tgoto fail_exit;\n \t\t}\n \n-\t\tif (radix_tree_preload(GFP_KERNEL)) {\n-\t\t\tblkg_free(new_blkg);\n-\t\t\tret = -ENOMEM;\n-\t\t\tgoto fail_exit;\n-\t\t}\n-\n \t\tspin_lock_irq(\u0026q-\u003equeue_lock);\n \n \t\tif (!blkcg_policy_enabled(q, pol)) {\n \t\t\tblkg_free(new_blkg);\n \t\t\tret = -EOPNOTSUPP;\n-\t\t\tgoto fail_preloaded;\n+\t\t\tgoto fail_unlock;\n \t\t}\n \n \t\tblkg = blkg_lookup(pos, q);\n@@ -903,12 +914,10 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,\n \t\t\tblkg = blkg_create(pos, disk, new_blkg);\n \t\t\tif (IS_ERR(blkg)) {\n \t\t\t\tret = PTR_ERR(blkg);\n-\t\t\t\tgoto fail_preloaded;\n+\t\t\t\tgoto fail_unlock;\n \t\t\t}\n \t\t}\n \n-\t\tradix_tree_preload_end();\n-\n \t\tif (pos == blkcg)\n \t\t\tgoto success;\n \t}\n@@ -917,8 +926,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,\n \tctx-\u003eblkg = blkg;\n \treturn 0;\n \n-fail_preloaded:\n-\tradix_tree_preload_end();\n fail_unlock:\n \tspin_unlock_irq(\u0026q-\u003equeue_lock);\n fail_exit:\n@@ -1373,6 +1380,9 @@ static void blkcg_css_free(struct cgroup_subsys_state *css)\n \n \tmutex_unlock(\u0026blkcg_pol_mutex);\n \n+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n+\tWARN_ON(!bio_list_empty(\u0026blkcg-\u003easync_bios));\n+#endif\n \tfree_percpu(blkcg-\u003elhead);\n \tkfree(blkcg);\n }\n@@ -1420,8 +1430,12 @@ blkcg_css_alloc(struct cgroup_subsys_state *parent_css)\n \n \tspin_lock_init(\u0026blkcg-\u003elock);\n \trefcount_set(\u0026blkcg-\u003eonline_pin, 1);\n-\tINIT_RADIX_TREE(\u0026blkcg-\u003eblkg_tree, GFP_NOWAIT);\n \tINIT_HLIST_HEAD(\u0026blkcg-\u003eblkg_list);\n+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n+\tspin_lock_init(\u0026blkcg-\u003easync_bio_lock);\n+\tbio_list_init(\u0026blkcg-\u003easync_bios);\n+\tINIT_WORK(\u0026blkcg-\u003easync_bio_work, blkcg_async_bio_workfn);\n+#endif\n #ifdef CONFIG_CGROUP_WRITEBACK\n \tINIT_LIST_HEAD(\u0026blkcg-\u003ecgwb_list);\n #endif\n@@ -1457,17 +1471,22 @@ static int blkcg_css_online(struct cgroup_subsys_state *css)\n \treturn 0;\n }\n \n-void blkg_init_queue(struct request_queue *q)\n+int blkg_init_queue(struct request_queue *q)\n {\n \tINIT_LIST_HEAD(\u0026q-\u003eblkg_list);\n \tmutex_init(\u0026q-\u003eblkcg_mutex);\n+\treturn rhashtable_init(\u0026q-\u003eblkg_hash, \u0026blkg_hash_params);\n+}\n+\n+void blkg_exit_queue(struct request_queue *q)\n+{\n+\trhashtable_destroy(\u0026q-\u003eblkg_hash);\n }\n \n int blkcg_init_disk(struct gendisk *disk)\n {\n \tstruct request_queue *q = disk-\u003equeue;\n \tstruct blkcg_gq *new_blkg, *blkg;\n-\tbool preloaded;\n \n \t/*\n \t * If the queue is shared across disk rebind (e.g., SCSI), the\n@@ -1485,8 +1504,6 @@ int blkcg_init_disk(struct gendisk *disk)\n \tif (!new_blkg)\n \t\treturn -ENOMEM;\n \n-\tpreloaded = !radix_tree_preload(GFP_KERNEL);\n-\n \t/* Make sure the root blkg exists. */\n \t/* spin_lock_irq can serve as RCU read-side critical section. */\n \tspin_lock_irq(\u0026q-\u003equeue_lock);\n@@ -1496,15 +1513,10 @@ int blkcg_init_disk(struct gendisk *disk)\n \tq-\u003eroot_blkg = blkg;\n \tspin_unlock_irq(\u0026q-\u003equeue_lock);\n \n-\tif (preloaded)\n-\t\tradix_tree_preload_end();\n-\n \treturn 0;\n \n err_unlock:\n \tspin_unlock_irq(\u0026q-\u003equeue_lock);\n-\tif (preloaded)\n-\t\tradix_tree_preload_end();\n \treturn PTR_ERR(blkg);\n }\n \n@@ -2064,129 +2076,165 @@ void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta)\n \tatomic64_add(delta, \u0026blkg-\u003edelay_nsec);\n }\n \n-static inline struct blkcg_gq *blkg_lookup_tryget(struct blkcg_gq *blkg)\n+/*\n+ * Return the blkg pinned by @bio through BIO_BLKG_REF.  The returned blkg is\n+ * already owned by @bio and no extra reference is acquired.\n+ */\n+static struct blkcg_gq *bio_pinned_blkg(struct bio *bio)\n {\n-retry:\n-\tif (blkg_tryget(blkg))\n-\t\treturn blkg;\n+\tstruct request_queue *q = bdev_get_queue(bio-\u003ebi_bdev);\n+\tstruct blkcg_gq *blkg;\n \n-\tblkg = blkg-\u003eparent;\n-\tif (blkg)\n-\t\tgoto retry;\n+\trcu_read_lock();\n+\tblkg = blkg_lookup_any(bio_blkcg(bio), q);\n+\trcu_read_unlock();\n \n-\treturn NULL;\n+\tWARN_ON_ONCE(!blkg);\n+\treturn blkg;\n+}\n+\n+static void bio_set_blkg_ref(struct bio *bio, struct blkcg_gq *blkg)\n+{\n+\tif (bio_blkcg(bio) != blkg-\u003eblkcg) {\n+\t\tcss_get(\u0026blkg-\u003eblkcg-\u003ecss);\n+\t\tbio_clear_blkcg(bio);\n+\t\tbio-\u003ebi_blkcg = blkg-\u003eblkcg;\n+\t}\n+\tbio_set_flag(bio, BIO_BLKG_REF);\n }\n+\n /**\n- * blkg_tryget_closest - try and get a blkg ref on the closet blkg\n+ * bio_blkg_lookup - look up a blkg associated with a bio\n  * @bio: target bio\n- * @css: target css\n  *\n- * As the failure mode here is to walk up the blkg tree, this ensure that the\n- * blkg-\u003eparent pointers are always valid.  This returns the blkg that it ended\n- * up taking a reference on or %NULL if no reference was taken.\n+ * Return the blkg already pinned by @bio, or %NULL if @bio doesn't own a blkg\n+ * reference.  Call bio_blkg() instead when a missing blkg should be created.\n  */\n-static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio,\n-\t\tstruct cgroup_subsys_state *css)\n+struct blkcg_gq *bio_blkg_lookup(struct bio *bio)\n {\n-\tstruct request_queue *q = bio-\u003ebi_bdev-\u003ebd_queue;\n-\tstruct blkcg *blkcg = css_to_blkcg(css);\n+\tif (!bio_flagged(bio, BIO_BLKG_REF))\n+\t\treturn NULL;\n+\treturn bio_pinned_blkg(bio);\n+}\n+EXPORT_SYMBOL_GPL(bio_blkg_lookup);\n+\n+/**\n+ * bio_put_blkg_ref - drop the blkg reference pinned by a bio\n+ * @bio: target bio\n+ *\n+ * Drop the bio-owned blkg reference acquired by bio_blkg(), if any.\n+ */\n+void bio_put_blkg_ref(struct bio *bio)\n+{\n+\tif (bio_flagged(bio, BIO_BLKG_REF)) {\n+\t\tstruct blkcg_gq *blkg = bio_pinned_blkg(bio);\n+\n+\t\tif (blkg)\n+\t\t\tblkg_put(blkg);\n+\t\tbio_clear_flag(bio, BIO_BLKG_REF);\n+\t}\n+}\n+EXPORT_SYMBOL_GPL(bio_put_blkg_ref);\n+\n+/**\n+ * bio_blkg - look up the blkg associated with a bio\n+ * @bio: target bio\n+ *\n+ * Look up the queue-local blkg for @bio's current device and blkcg.  If this\n+ * is the first policy use of @bio, create the missing blkg hierarchy if\n+ * necessary, pin the closest available blkg, and update @bio's blkcg\n+ * association if allocation falls back to an ancestor.\n+ */\n+struct blkcg_gq *bio_blkg(struct bio *bio)\n+{\n+\tstruct blkcg *blkcg = bio_blkcg(bio);\n+\tstruct gendisk *disk;\n+\tstruct request_queue *q;\n \tstruct blkcg_gq *blkg;\n \n-\trcu_read_lock();\n-\tblkg = blkg_lookup(blkcg, q);\n-\tif (likely(blkg))\n-\t\tblkg = blkg_lookup_tryget(blkg);\n-\trcu_read_unlock();\n+\tif (!blkcg || !bio-\u003ebi_bdev)\n+\t\treturn NULL;\n \n-\tif (blkg)\n-\t\treturn blkg;\n+\tif (bio_flagged(bio, BIO_BLKG_REF))\n+\t\treturn bio_pinned_blkg(bio);\n+\n+\tdisk = bio-\u003ebi_bdev-\u003ebd_disk;\n+\tq = disk-\u003equeue;\n \n-\t/*\n-\t * Fast path failed, we're probably issuing IO in this cgroup the first\n-\t * time, hold lock to create new blkg.\n-\t */\n \tspin_lock_irq(\u0026q-\u003equeue_lock);\n-\tblkg = blkg_lookup_create(blkcg, bio-\u003ebi_bdev-\u003ebd_disk);\n-\tif (blkg)\n-\t\tblkg = blkg_lookup_tryget(blkg);\n+\tblkg = blkg_lookup_create(blkcg, disk);\n \tspin_unlock_irq(\u0026q-\u003equeue_lock);\n \n+\tbio_set_blkg_ref(bio, blkg);\n \treturn blkg;\n }\n+EXPORT_SYMBOL_GPL(bio_blkg);\n \n /**\n- * bio_associate_blkg_from_css - associate a bio with a specified css\n+ * bio_associate_blkcg_from_css - associate a bio with a specified css\n  * @bio: target bio\n  * @css: target css\n  *\n- * Associate @bio with the blkg found by combining the css's blkg and the\n- * request_queue of the @bio.  An association failure is handled by walking up\n- * the blkg tree.  Therefore, the blkg associated can be anything between @blkg\n- * and q-\u003eroot_blkg.  This situation only happens when a cgroup is dying and\n- * then the remaining bios will spill to the closest alive blkg.\n+ * Associate @bio with the blkcg found from @css.  The queue-local blkg is\n+ * created and pinned by bio_blkg() when blkcg policies need it.  If @css\n+ * can't be referenced online, associate @bio with the root blkcg instead.\n  *\n- * A reference will be taken on the blkg and will be released when @bio is\n+ * A reference will be taken on the blkcg and will be released when @bio is\n  * freed.\n  */\n-void bio_associate_blkg_from_css(struct bio *bio,\n-\t\t\t\t struct cgroup_subsys_state *css)\n+void bio_associate_blkcg_from_css(struct bio *bio,\n+\t\t\t\t  struct cgroup_subsys_state *css)\n {\n-\tif (bio-\u003ebi_blkg)\n-\t\tblkg_put(bio-\u003ebi_blkg);\n+\tstruct blkcg *blkcg;\n \n-\tif (css \u0026\u0026 css-\u003eparent) {\n-\t\tbio-\u003ebi_blkg = blkg_tryget_closest(bio, css);\n-\t} else {\n-\t\tblkg_get(bdev_get_queue(bio-\u003ebi_bdev)-\u003eroot_blkg);\n-\t\tbio-\u003ebi_blkg = bdev_get_queue(bio-\u003ebi_bdev)-\u003eroot_blkg;\n+\tif (!css || !css_tryget_online(css)) {\n+\t\tcss = \u0026blkcg_root.css;\n+\t\tcss_get(css);\n \t}\n+\n+\tblkcg = css_to_blkcg(css);\n+\tif (bio_blkcg(bio) == blkcg) {\n+\t\tcss_put(css);\n+\t\treturn;\n+\t}\n+\n+\tbio_clear_blkcg(bio);\n+\tbio-\u003ebi_blkcg = blkcg;\n }\n-EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css);\n+EXPORT_SYMBOL_GPL(bio_associate_blkcg_from_css);\n \n /**\n- * bio_associate_blkg - associate a bio with a blkg\n+ * bio_associate_blkcg - associate a bio with a blkcg\n  * @bio: target bio\n  *\n- * Associate @bio with the blkg found from the bio's css and request_queue.\n- * If one is not found, bio_lookup_blkg() creates the blkg.  If a blkg is\n- * already associated, the css is reused and association redone as the\n- * request_queue may have changed.\n+ * Associate @bio with the blkcg found from the bio's css.  If a blkcg is\n+ * already associated, keep it as blkcg association is not queue-local.\n  */\n-void bio_associate_blkg(struct bio *bio)\n+void bio_associate_blkcg(struct bio *bio)\n {\n-\tstruct cgroup_subsys_state *css;\n-\n \tif (blk_op_is_passthrough(bio-\u003ebi_opf))\n \t\treturn;\n \n-\tif (bio-\u003ebi_blkg) {\n-\t\tcss = bio_blkcg_css(bio);\n-\t\tbio_associate_blkg_from_css(bio, css);\n-\t} else {\n-\t\trcu_read_lock();\n-\t\tcss = blkcg_css();\n-\t\tif (!css_tryget_online(css))\n-\t\t\tcss = NULL;\n-\t\trcu_read_unlock();\n+\tif (bio_blkcg(bio))\n+\t\treturn;\n \n-\t\tbio_associate_blkg_from_css(bio, css);\n-\t\tif (css)\n-\t\t\tcss_put(css);\n-\t}\n+\trcu_read_lock();\n+\tbio_associate_blkcg_from_css(bio, blkcg_css());\n+\trcu_read_unlock();\n }\n-EXPORT_SYMBOL_GPL(bio_associate_blkg);\n+EXPORT_SYMBOL_GPL(bio_associate_blkcg);\n \n /**\n- * bio_clone_blkg_association - clone blkg association from src to dst bio\n+ * bio_clone_blkcg_association - clone blkcg association from src to dst bio\n  * @dst: destination bio\n  * @src: source bio\n  */\n-void bio_clone_blkg_association(struct bio *dst, struct bio *src)\n+void bio_clone_blkcg_association(struct bio *dst, struct bio *src)\n {\n-\tif (src-\u003ebi_blkg)\n-\t\tbio_associate_blkg_from_css(dst, bio_blkcg_css(src));\n+\tif (bio_blkcg(src))\n+\t\tbio_associate_blkcg_from_css(dst, bio_blkcg_css(src));\n }\n-EXPORT_SYMBOL_GPL(bio_clone_blkg_association);\n+EXPORT_SYMBOL_GPL(bio_clone_blkcg_association);\n \n static int blk_cgroup_io_type(struct bio *bio)\n {\n@@ -2199,20 +2247,27 @@ static int blk_cgroup_io_type(struct bio *bio)\n \n void blk_cgroup_bio_start(struct bio *bio)\n {\n-\tstruct blkcg *blkcg = bio-\u003ebi_blkg-\u003eblkcg;\n+\tstruct blkcg *blkcg = bio_blkcg(bio);\n+\tstruct blkcg_gq *blkg;\n \tint rwd = blk_cgroup_io_type(bio), cpu;\n \tstruct blkg_iostat_set *bis;\n \tunsigned long flags;\n \n \tif (!cgroup_subsys_on_dfl(io_cgrp_subsys))\n \t\treturn;\n+\tif (!blkcg)\n+\t\treturn;\n \n \t/* Root-level stats are sourced from system-wide IO stats */\n \tif (!cgroup_parent(blkcg-\u003ecss.cgroup))\n \t\treturn;\n \n+\tblkg = bio_blkg_lookup(bio);\n+\tif (!blkg)\n+\t\treturn;\n+\n \tcpu = get_cpu();\n-\tbis = per_cpu_ptr(bio-\u003ebi_blkg-\u003eiostat_cpu, cpu);\n+\tbis = per_cpu_ptr(blkg-\u003eiostat_cpu, cpu);\n \tflags = u64_stats_update_begin_irqsave(\u0026bis-\u003esync);\n \n \t/*\ndiff --git a/block/blk-cgroup.h b/block/blk-cgroup.h\nindex e67c698391294..eb76cc38b41ba 100644\n--- a/block/blk-cgroup.h\n+++ b/block/blk-cgroup.h\n@@ -19,6 +19,8 @@\n #include \u003clinux/kthread.h\u003e\n #include \u003clinux/blk-mq.h\u003e\n #include \u003clinux/llist.h\u003e\n+#include \u003clinux/rhashtable.h\u003e\n+#include \u003clinux/rcupdate.h\u003e\n #include \"blk.h\"\n \n struct blkcg_gq;\n@@ -56,9 +58,11 @@ struct blkg_iostat_set {\n struct blkcg_gq {\n \t/* Pointer to the associated request_queue */\n \tstruct request_queue\t\t*q;\n+\tstruct rhash_head\t\tq_hash_node;\n \tstruct list_head\t\tq_node;\n \tstruct hlist_node\t\tblkcg_node;\n \tstruct blkcg\t\t\t*blkcg;\n+\tint\t\t\t\tblkcg_id;\n \n \t/* all non-root blkcg_gq's are guaranteed to have access to parent */\n \tstruct blkcg_gq\t\t\t*parent;\n@@ -73,14 +77,7 @@ struct blkcg_gq {\n \tstruct blkg_iostat_set\t\tiostat;\n \n \tstruct blkg_policy_data\t\t*pd[BLKCG_MAX_POLS];\n-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n-\tspinlock_t\t\t\tasync_bio_lock;\n-\tstruct bio_list\t\t\tasync_bios;\n-#endif\n-\tunion {\n-\t\tstruct work_struct\tasync_bio_work;\n-\t\tstruct work_struct\tfree_work;\n-\t};\n+\tstruct work_struct\t\tfree_work;\n \n \tatomic_t\t\t\tuse_delay;\n \tatomic64_t\t\t\tdelay_nsec;\n@@ -98,8 +95,6 @@ struct blkcg {\n \t/* If there is block congestion on this cgroup. */\n \tatomic_t\t\t\tcongestion_count;\n \n-\tstruct radix_tree_root\t\tblkg_tree;\n-\tstruct blkcg_gq\t__rcu\t\t*blkg_hint;\n \tstruct hlist_head\t\tblkg_list;\n \n \tstruct blkcg_policy_data\t*cpd[BLKCG_MAX_POLS];\n@@ -111,6 +106,11 @@ struct blkcg {\n \t */\n \tstruct llist_head __percpu\t*lhead;\n \n+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO\n+\tspinlock_t\t\t\tasync_bio_lock; /* protects async_bios */\n+\tstruct bio_list\t\t\tasync_bios;\n+\tstruct work_struct\t\tasync_bio_work;\n+#endif\n #ifdef CONFIG_BLK_CGROUP_FC_APPID\n \tchar                            fc_app_id[FC_APPID_LEN];\n #endif\n@@ -124,6 +124,11 @@ static inline struct blkcg *css_to_blkcg(struct cgroup_subsys_state *css)\n \treturn css ? container_of(css, struct blkcg, css) : NULL;\n }\n \n+static inline struct blkcg *bio_blkcg(struct bio *bio)\n+{\n+\treturn bio-\u003ebi_blkcg;\n+}\n+\n /*\n  * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a\n  * request_queue (q).  This is used by blkcg policies which need to track\n@@ -192,8 +197,10 @@ struct blkcg_policy {\n \n extern struct blkcg blkcg_root;\n extern bool blkcg_debug_stats;\n+extern const struct rhashtable_params blkg_hash_params;\n \n-void blkg_init_queue(struct request_queue *q);\n+int blkg_init_queue(struct request_queue *q);\n+void blkg_exit_queue(struct request_queue *q);\n int blkcg_init_disk(struct gendisk *disk);\n void blkcg_exit_disk(struct gendisk *disk);\n \n@@ -249,11 +256,33 @@ static inline bool bio_issue_as_root_blkg(struct bio *bio)\n }\n \n /**\n- * blkg_lookup - lookup blkg for the specified blkcg - q pair\n+ * blkg_lookup_any - lookup any blkg for the specified blkcg - q pair\n+ * @blkcg: blkcg of interest\n+ * @q: request_queue of interest\n+ *\n+ * Lookup a blkg for the @blkcg - @q pair, whether it is online or dying.\n+ *\n+ * Must be called in a RCU critical section.\n+ *\n+ * This does not acquire a reference.  The caller must already hold one, or\n+ * have the blkg pinned by I/O.\n+ */\n+static inline struct blkcg_gq *blkg_lookup_any(struct blkcg *blkcg,\n+\t\t\t\t\t       struct request_queue *q)\n+{\n+\tRCU_LOCKDEP_WARN(!rcu_read_lock_held(),\n+\t\t\t \"blkg_lookup_any() requires an RCU read lock\");\n+\n+\treturn rhashtable_lookup(\u0026q-\u003eblkg_hash, \u0026blkcg-\u003ecss.id,\n+\t\t\t\t blkg_hash_params);\n+}\n+\n+/**\n+ * blkg_lookup - lookup an online blkg for the specified blkcg - q pair\n  * @blkcg: blkcg of interest\n  * @q: request_queue of interest\n  *\n- * Lookup blkg for the @blkcg - @q pair.\n+ * Lookup an online blkg for the @blkcg - @q pair.\n  *\n  * Must be called in a RCU critical section.\n  */\n@@ -265,17 +294,15 @@ static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg,\n \tif (blkcg == \u0026blkcg_root)\n \t\treturn q-\u003eroot_blkg;\n \n-\tblkg = rcu_dereference_check(blkcg-\u003eblkg_hint,\n-\t\t\tlockdep_is_held(\u0026q-\u003equeue_lock));\n-\tif (blkg \u0026\u0026 blkg-\u003eq == q)\n-\t\treturn blkg;\n-\n-\tblkg = radix_tree_lookup(\u0026blkcg-\u003eblkg_tree, q-\u003eid);\n-\tif (blkg \u0026\u0026 blkg-\u003eq != q)\n+\tblkg = blkg_lookup_any(blkcg, q);\n+\tif (blkg \u0026\u0026 !READ_ONCE(blkg-\u003eonline))\n \t\tblkg = NULL;\n \treturn blkg;\n }\n \n+struct blkcg_gq *bio_blkg_lookup(struct bio *bio);\n+struct blkcg_gq *bio_blkg(struct bio *bio);\n+\n /**\n  * blkg_to_pd - get policy private data\n  * @blkg: blkg of interest\n@@ -343,6 +370,18 @@ static inline void blkg_put(struct blkcg_gq *blkg)\n \tpercpu_ref_put(\u0026blkg-\u003erefcnt);\n }\n \n+static inline void bio_clear_blkcg(struct bio *bio)\n+{\n+\tstruct blkcg *blkcg = bio_blkcg(bio);\n+\n+\tbio_put_blkg_ref(bio);\n+\n+\tif (blkcg) {\n+\t\tcss_put(\u0026blkcg-\u003ecss);\n+\t\tbio-\u003ebi_blkcg = NULL;\n+\t}\n+}\n+\n /**\n  * blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants\n  * @d_blkg: loop cursor pointing to the current descendant\n@@ -472,7 +511,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq *blkg)\n  */\n static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bio)\n {\n-\treturn rq-\u003ebio-\u003ebi_blkg == bio-\u003ebi_blkg \u0026\u0026\n+\treturn bio_blkcg(rq-\u003ebio) == bio_blkcg(bio) \u0026\u0026\n \t\tbio_issue_as_root_blkg(rq-\u003ebio) == bio_issue_as_root_blkg(bio);\n }\n \n@@ -498,8 +537,13 @@ struct blkcg_policy {\n struct blkcg {\n };\n \n+static inline struct blkcg *bio_blkcg(struct bio *bio) { return NULL; }\n+static inline struct blkcg_gq *bio_blkg_lookup(struct bio *bio) { return NULL; }\n+static inline struct blkcg_gq *bio_blkg(struct bio *bio) { return NULL; }\n+static inline struct blkcg_gq *blkg_lookup_any(struct blkcg *blkcg, void *key) { return NULL; }\n static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, void *key) { return NULL; }\n-static inline void blkg_init_queue(struct request_queue *q) { }\n+static inline int blkg_init_queue(struct request_queue *q) { return 0; }\n+static inline void blkg_exit_queue(struct request_queue *q) { }\n static inline int blkcg_init_disk(struct gendisk *disk) { return 0; }\n static inline void blkcg_exit_disk(struct gendisk *disk) { }\n static inline int blkcg_policy_register(struct blkcg_policy *pol) { return 0; }\n@@ -514,6 +558,7 @@ static inline struct blkg_policy_data *blkg_to_pd(struct blkcg_gq *blkg,\n static inline struct blkcg_gq *pd_to_blkg(struct blkg_policy_data *pd) { return NULL; }\n static inline void blkg_get(struct blkcg_gq *blkg) { }\n static inline void blkg_put(struct blkcg_gq *blkg) { }\n+static inline void bio_clear_blkcg(struct bio *bio) { }\n static inline void blk_cgroup_bio_start(struct bio *bio) { }\n static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bio) { return true; }\n \ndiff --git a/block/blk-core.c b/block/blk-core.c\nindex 196bccf27f58d..e396a316b8bf6 100644\n--- a/block/blk-core.c\n+++ b/block/blk-core.c\n@@ -303,6 +303,7 @@ static void blk_free_queue(struct request_queue *q)\n \tif (queue_is_mq(q))\n \t\tblk_mq_release(q);\n \n+\tblkg_exit_queue(q);\n \tida_free(\u0026blk_queue_ida, q-\u003eid);\n \tlockdep_unregister_key(\u0026q-\u003eio_lock_cls_key);\n \tlockdep_unregister_key(\u0026q-\u003eq_lock_cls_key);\n@@ -481,7 +482,9 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)\n \tinit_waitqueue_head(\u0026q-\u003emq_freeze_wq);\n \tmutex_init(\u0026q-\u003emq_freeze_lock);\n \n-\tblkg_init_queue(q);\n+\terror = blkg_init_queue(q);\n+\tif (error)\n+\t\tgoto fail_stats;\n \n \t/*\n \t * Init percpu_ref in atomic mode so that it's faster to shutdown.\n@@ -491,7 +494,7 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)\n \t\t\t\tblk_queue_usage_counter_release,\n \t\t\t\tPERCPU_REF_INIT_ATOMIC, GFP_KERNEL);\n \tif (error)\n-\t\tgoto fail_stats;\n+\t\tgoto fail_blkg;\n \tlockdep_register_key(\u0026q-\u003eio_lock_cls_key);\n \tlockdep_register_key(\u0026q-\u003eq_lock_cls_key);\n \tlockdep_init_map(\u0026q-\u003eio_lockdep_map, \"\u0026q-\u003eq_usage_counter(io)\",\n@@ -510,6 +513,8 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)\n \n \treturn q;\n \n+fail_blkg:\n+\tblkg_exit_queue(q);\n fail_stats:\n \tblk_free_queue_stats(q-\u003estats);\n fail_id:\ndiff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c\nindex 2a5c52ab74b4c..5ef4baab444bb 100644\n--- a/block/blk-crypto-fallback.c\n+++ b/block/blk-crypto-fallback.c\n@@ -187,7 +187,7 @@ static struct bio *blk_crypto_alloc_enc_bio(struct bio *bio_src,\n \tbio-\u003ebi_write_hint\t= bio_src-\u003ebi_write_hint;\n \tbio-\u003ebi_write_stream\t= bio_src-\u003ebi_write_stream;\n \tbio-\u003ebi_iter.bi_sector\t= bio_src-\u003ebi_iter.bi_sector;\n-\tbio_clone_blkg_association(bio, bio_src);\n+\tbio_clone_blkcg_association(bio, bio_src);\n \n \t/*\n \t * Move page array up in the allocated memory for the bio vecs as far as\ndiff --git a/block/blk-iocost.c b/block/blk-iocost.c\nindex 2745bffcd5eef..57f2b4d4af20a 100644\n--- a/block/blk-iocost.c\n+++ b/block/blk-iocost.c\n@@ -2686,7 +2686,7 @@ iocg_handle_over_budget(struct rq_qos *rqos, struct ioc_gq *iocg,\n \n static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)\n {\n-\tstruct blkcg_gq *blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg_gq *blkg = bio_blkg(bio);\n \tstruct ioc *ioc = rqos_to_ioc(rqos);\n \tstruct ioc_gq *iocg = blkg_to_iocg(blkg);\n \tstruct ioc_now now;\n@@ -2775,7 +2775,7 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)\n static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq,\n \t\t\t   struct bio *bio)\n {\n-\tstruct ioc_gq *iocg = blkg_to_iocg(bio-\u003ebi_blkg);\n+\tstruct ioc_gq *iocg = blkg_to_iocg(bio_blkg(bio));\n \tstruct ioc *ioc = rqos_to_ioc(rqos);\n \tsector_t bio_end = bio_end_sector(bio);\n \tstruct ioc_now now;\n@@ -2833,9 +2833,13 @@ static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq,\n \n static void ioc_rqos_done_bio(struct rq_qos *rqos, struct bio *bio)\n {\n-\tstruct ioc_gq *iocg = blkg_to_iocg(bio-\u003ebi_blkg);\n+\tstruct ioc_gq *iocg;\n+\n+\tif (!bio-\u003ebi_iocost_cost)\n+\t\treturn;\n \n-\tif (iocg \u0026\u0026 bio-\u003ebi_iocost_cost)\n+\tiocg = blkg_to_iocg(bio_blkg_lookup(bio));\n+\tif (iocg)\n \t\tatomic64_add(bio-\u003ebi_iocost_cost, \u0026iocg-\u003edone_vtime);\n }\n \ndiff --git a/block/blk-iolatency.c b/block/blk-iolatency.c\nindex 2caa79a008ad1..7220edafd72b6 100644\n--- a/block/blk-iolatency.c\n+++ b/block/blk-iolatency.c\n@@ -463,7 +463,7 @@ static void check_scale_change(struct iolatency_grp *iolat)\n static void blkcg_iolatency_throttle(struct rq_qos *rqos, struct bio *bio)\n {\n \tstruct blk_iolatency *blkiolat = BLKIOLATENCY(rqos);\n-\tstruct blkcg_gq *blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg_gq *blkg = bio_blkg(bio);\n \tbool issue_as_root = bio_issue_as_root_blkg(bio);\n \n \tif (!blkiolat-\u003eenabled)\n@@ -590,11 +590,14 @@ static void blkcg_iolatency_done_bio(struct rq_qos *rqos, struct bio *bio)\n \tbool issue_as_root = bio_issue_as_root_blkg(bio);\n \tint inflight = 0;\n \n-\tblkg = bio-\u003ebi_blkg;\n-\tif (!blkg || !bio_flagged(bio, BIO_QOS_THROTTLED))\n+\tif (!bio_flagged(bio, BIO_QOS_THROTTLED))\n \t\treturn;\n \n-\tiolat = blkg_to_lat(bio-\u003ebi_blkg);\n+\tblkg = bio_blkg_lookup(bio);\n+\tif (!blkg)\n+\t\treturn;\n+\n+\tiolat = blkg_to_lat(blkg);\n \tif (!iolat)\n \t\treturn;\n \ndiff --git a/block/blk-ioprio.c b/block/blk-ioprio.c\nindex 8fa8bca350629..5e3f8e49550da 100644\n--- a/block/blk-ioprio.c\n+++ b/block/blk-ioprio.c\n@@ -132,7 +132,7 @@ static struct blkcg_policy ioprio_policy = {\n \n void blkcg_set_ioprio(struct bio *bio)\n {\n-\tstruct ioprio_blkcg *blkcg = blkcg_to_ioprio_blkcg(bio-\u003ebi_blkg-\u003eblkcg);\n+\tstruct ioprio_blkcg *blkcg = blkcg_to_ioprio_blkcg(bio_blkcg(bio));\n \tu16 prio;\n \n \tif (!blkcg || blkcg-\u003eprio_policy == POLICY_NO_CHANGE)\ndiff --git a/block/blk-throttle.c b/block/blk-throttle.c\nindex ffc3b70065d4b..3828c38579002 100644\n--- a/block/blk-throttle.c\n+++ b/block/blk-throttle.c\n@@ -1792,7 +1792,7 @@ static bool tg_within_limit(struct throtl_grp *tg, struct bio *bio, bool rw)\n bool __blk_throtl_bio(struct bio *bio)\n {\n \tstruct request_queue *q = bdev_get_queue(bio-\u003ebi_bdev);\n-\tstruct blkcg_gq *blkg = bio-\u003ebi_blkg;\n+\tstruct blkcg_gq *blkg = bio_blkg(bio);\n \tstruct throtl_qnode *qn = NULL;\n \tstruct throtl_grp *tg = blkg_to_tg(blkg);\n \tstruct throtl_service_queue *sq;\ndiff --git a/block/blk-throttle.h b/block/blk-throttle.h\nindex 1b5775771e078..45f30179fd9c0 100644\n--- a/block/blk-throttle.h\n+++ b/block/blk-throttle.h\n@@ -171,7 +171,7 @@ static inline bool blk_should_throtl(struct bio *bio)\n \tif (!blk_throtl_activated(bio-\u003ebi_bdev-\u003ebd_queue))\n \t\treturn false;\n \n-\ttg = blkg_to_tg(bio-\u003ebi_blkg);\n+\ttg = blkg_to_tg(bio_blkg(bio));\n \tif (!cgroup_subsys_on_dfl(io_cgrp_subsys)) {\n \t\tif (!bio_flagged(bio, BIO_CGROUP_ACCT)) {\n \t\t\tbio_set_flag(bio, BIO_CGROUP_ACCT);\ndiff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c\nindex 3fa3b13a410f4..c0f945b8d9413 100644\n--- a/drivers/md/bcache/request.c\n+++ b/drivers/md/bcache/request.c\n@@ -848,7 +848,7 @@ static CLOSURE_CALLBACK(cached_dev_read_done)\n \t\ts-\u003eiop.bio-\u003ebi_iter.bi_sector =\n \t\t\ts-\u003ecache_miss-\u003ebi_iter.bi_sector;\n \t\ts-\u003eiop.bio-\u003ebi_iter.bi_size = s-\u003einsert_bio_sectors \u003c\u003c 9;\n-\t\tbio_clone_blkg_association(s-\u003eiop.bio, s-\u003ecache_miss);\n+\t\tbio_clone_blkcg_association(s-\u003eiop.bio, s-\u003ecache_miss);\n \t\tbch_bio_map(s-\u003eiop.bio, NULL);\n \n \t\tbio_copy_data(s-\u003ecache_miss, s-\u003eiop.bio);\ndiff --git a/drivers/md/dm.c b/drivers/md/dm.c\nindex d413bfaf35278..cd68eec77f5a1 100644\n--- a/drivers/md/dm.c\n+++ b/drivers/md/dm.c\n@@ -1373,7 +1373,7 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone)\n \tif (!tgt_clone)\n \t\ttgt_clone = clone;\n \n-\tbio_clone_blkg_association(tgt_clone, io-\u003eorig_bio);\n+\tbio_clone_blkcg_association(tgt_clone, io-\u003eorig_bio);\n \n \t/*\n \t * Account io-\u003eorigin_bio to DM dev on behalf of target\ndiff --git a/drivers/md/md.c b/drivers/md/md.c\nindex 680b34a63cb3d..fd78d6cc8da17 100644\n--- a/drivers/md/md.c\n+++ b/drivers/md/md.c\n@@ -9386,7 +9386,7 @@ void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,\n \t\treturn;\n \n \tbio_chain(discard_bio, bio);\n-\tbio_clone_blkg_association(discard_bio, bio);\n+\tbio_clone_blkcg_association(discard_bio, bio);\n \tmddev_trace_remap(mddev, discard_bio, bio-\u003ebi_iter.bi_sector);\n \tsubmit_bio_noacct(discard_bio);\n }\ndiff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c\nindex 4176046627beb..54e4adb0ccb7d 100644\n--- a/drivers/nvdimm/nd_virtio.c\n+++ b/drivers/nvdimm/nd_virtio.c\n@@ -121,7 +121,7 @@ int async_pmem_flush(struct nd_region *nd_region, struct bio *bio)\n \n \t\tif (!child)\n \t\t\treturn -ENOMEM;\n-\t\tbio_clone_blkg_association(child, bio);\n+\t\tbio_clone_blkcg_association(child, bio);\n \t\tchild-\u003ebi_iter.bi_sector = -1;\n \t\tbio_chain(child, bio);\n \t\tsubmit_bio(child);\ndiff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c\nindex 6dabe73ad790d..6512dbd9516fc 100644\n--- a/fs/gfs2/lops.c\n+++ b/fs/gfs2/lops.c\n@@ -484,7 +484,7 @@ static struct bio *gfs2_chain_bio(struct bio *prev, unsigned int nr_iovecs,\n \tstruct bio *new;\n \n \tnew = bio_alloc(prev-\u003ebi_bdev, nr_iovecs, opf, GFP_NOIO);\n-\tbio_clone_blkg_association(new, prev);\n+\tbio_clone_blkcg_association(new, prev);\n \tnew-\u003ebi_iter.bi_sector = sector;\n \tbio_chain(new, prev);\n \tsubmit_bio(prev);\n@@ -1114,4 +1114,3 @@ const struct gfs2_log_operations *gfs2_log_ops[] = {\n \t\u0026gfs2_revoke_lops,\n \tNULL,\n };\n-\ndiff --git a/include/linux/bio.h b/include/linux/bio.h\nindex bb3235497e673..11637f85c38d4 100644\n--- a/include/linux/bio.h\n+++ b/include/linux/bio.h\n@@ -542,18 +542,20 @@ static inline void bio_release_pages(struct bio *bio, bool mark_dirty)\n \tdisk_devt((bio)-\u003ebi_bdev-\u003ebd_disk)\n \n #ifdef CONFIG_BLK_CGROUP\n-void bio_associate_blkg(struct bio *bio);\n-void bio_associate_blkg_from_css(struct bio *bio,\n-\t\t\t\t struct cgroup_subsys_state *css);\n-void bio_clone_blkg_association(struct bio *dst, struct bio *src);\n+void bio_associate_blkcg(struct bio *bio);\n+void bio_associate_blkcg_from_css(struct bio *bio,\n+\t\t\t\t  struct cgroup_subsys_state *css);\n+void bio_clone_blkcg_association(struct bio *dst, struct bio *src);\n+void bio_put_blkg_ref(struct bio *bio);\n void blkcg_punt_bio_submit(struct bio *bio);\n #else\t/* CONFIG_BLK_CGROUP */\n-static inline void bio_associate_blkg(struct bio *bio) { }\n-static inline void bio_associate_blkg_from_css(struct bio *bio,\n-\t\t\t\t\t       struct cgroup_subsys_state *css)\n+static inline void bio_associate_blkcg(struct bio *bio) { }\n+static inline void bio_associate_blkcg_from_css(struct bio *bio,\n+\t\t\t\t\t\tstruct cgroup_subsys_state *css)\n { }\n-static inline void bio_clone_blkg_association(struct bio *dst,\n-\t\t\t\t\t      struct bio *src) { }\n+static inline void bio_clone_blkcg_association(struct bio *dst,\n+\t\t\t\t\t       struct bio *src) { }\n+static inline void bio_put_blkg_ref(struct bio *bio) { }\n static inline void blkcg_punt_bio_submit(struct bio *bio)\n {\n \tsubmit_bio(bio);\n@@ -563,10 +565,12 @@ static inline void blkcg_punt_bio_submit(struct bio *bio)\n static inline void bio_set_dev(struct bio *bio, struct block_device *bdev)\n {\n \tbio_clear_flag(bio, BIO_REMAPPED);\n-\tif (bio-\u003ebi_bdev != bdev)\n+\tif (bio-\u003ebi_bdev != bdev) {\n+\t\tbio_put_blkg_ref(bio);\n \t\tbio_clear_flag(bio, BIO_BPS_THROTTLED);\n+\t}\n \tbio-\u003ebi_bdev = bdev;\n-\tbio_associate_blkg(bio);\n+\tbio_associate_blkcg(bio);\n }\n \n /*\ndiff --git a/include/linux/blk_types.h b/include/linux/blk_types.h\nindex d49d97a050d0e..53ba307b91cb3 100644\n--- a/include/linux/blk_types.h\n+++ b/include/linux/blk_types.h\n@@ -246,12 +246,10 @@ struct bio {\n \tvoid\t\t\t*bi_private;\n #ifdef CONFIG_BLK_CGROUP\n \t/*\n-\t * Represents the association of the css and request_queue for the bio.\n-\t * If a bio goes direct to device, it will not have a blkg as it will\n-\t * not have a request_queue associated with it.  The reference is put\n-\t * on release of the bio.\n+\t * Represents the blkcg css association for the bio.  The reference is\n+\t * put on release of the bio.\n \t */\n-\tstruct blkcg_gq\t\t*bi_blkg;\n+\tstruct blkcg\t\t*bi_blkcg;\n \t/* Time that this bio was issued. */\n \tu64\t\t\tissue_time_ns;\n #ifdef CONFIG_BLK_CGROUP_IOCOST\n@@ -309,6 +307,7 @@ enum {\n \tBIO_TRACE_COMPLETION,\t/* bio_endio() should trace the final completion\n \t\t\t\t * of this bio. */\n \tBIO_CGROUP_ACCT,\t/* has been accounted to a cgroup */\n+\tBIO_BLKG_REF,\t\t/* bio pins the associated blkg */\n \tBIO_QOS_THROTTLED,\t/* bio went through rq_qos throttle path */\n \t/*\n \t * This bio has completed bps throttling at the single tg granularity,\ndiff --git a/include/linux/blkdev.h b/include/linux/blkdev.h\nindex 20cb8ed7d987a..f3b0e39d3f6b4 100644\n--- a/include/linux/blkdev.h\n+++ b/include/linux/blkdev.h\n@@ -27,6 +27,7 @@\n #include \u003clinux/xarray.h\u003e\n #include \u003clinux/file.h\u003e\n #include \u003clinux/lockdep.h\u003e\n+#include \u003clinux/rhashtable-types.h\u003e\n \n struct module;\n struct request_queue;\n@@ -581,6 +582,7 @@ struct request_queue {\n #ifdef CONFIG_BLK_CGROUP\n \tDECLARE_BITMAP\t\t(blkcg_pols, BLKCG_MAX_POLS);\n \tstruct blkcg_gq\t\t*root_blkg;\n+\tstruct rhashtable\tblkg_hash;\n \tstruct list_head\tblkg_list;\n \tstruct mutex\t\tblkcg_mutex;\n #endif\ndiff --git a/include/linux/writeback.h b/include/linux/writeback.h\nindex 62552a2ce5b9e..4f869fe9cc90f 100644\n--- a/include/linux/writeback.h\n+++ b/include/linux/writeback.h\n@@ -262,7 +262,7 @@ static inline void wbc_init_bio(struct writeback_control *wbc, struct bio *bio)\n \t * regular writeback instead of writing things out itself.\n \t */\n \tif (wbc-\u003ewb)\n-\t\tbio_associate_blkg_from_css(bio, wbc-\u003ewb-\u003eblkcg_css);\n+\t\tbio_associate_blkcg_from_css(bio, wbc-\u003ewb-\u003eblkcg_css);\n }\n \n void inode_switch_wbs_work_fn(struct work_struct *work);\ndiff --git a/mm/page_io.c b/mm/page_io.c\nindex b23f494fcc83d..a4a1b8a8a0e38 100644\n--- a/mm/page_io.c\n+++ b/mm/page_io.c\n@@ -316,7 +316,7 @@ static inline void count_swpout_vm_event(struct folio *folio)\n }\n \n #if defined(CONFIG_MEMCG) \u0026\u0026 defined(CONFIG_BLK_CGROUP)\n-static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)\n+static void bio_associate_blkcg_from_page(struct bio *bio, struct folio *folio)\n {\n \tstruct cgroup_subsys_state *css;\n \tstruct mem_cgroup *memcg;\n@@ -331,12 +331,15 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)\n \t\tcss = NULL;\n \trcu_read_unlock();\n \n-\tbio_associate_blkg_from_css(bio, css);\n+\tbio_associate_blkcg_from_css(bio, css);\n \tif (css)\n \t\tcss_put(css);\n }\n #else\n-#define bio_associate_blkg_from_page(bio, folio)\t\tdo { } while (0)\n+static inline void bio_associate_blkcg_from_page(struct bio *bio,\n+\t\t\t\t\t\t struct folio *folio)\n+{\n+}\n #endif /* CONFIG_MEMCG \u0026\u0026 CONFIG_BLK_CGROUP */\n \n struct swap_iocb {\n@@ -436,7 +439,7 @@ static void swap_writepage_bdev_sync(struct folio *folio,\n \tbio.bi_iter.bi_sector = swap_folio_sector(folio);\n \tbio_add_folio_nofail(\u0026bio, folio, folio_size(folio), 0);\n \n-\tbio_associate_blkg_from_page(\u0026bio, folio);\n+\tbio_associate_blkcg_from_page(\u0026bio, folio);\n \tcount_swpout_vm_event(folio);\n \n \tfolio_start_writeback(folio);\n@@ -456,7 +459,7 @@ static void swap_writepage_bdev_async(struct folio *folio,\n \tbio-\u003ebi_end_io = end_swap_bio_write;\n \tbio_add_folio_nofail(bio, folio, folio_size(folio), 0);\n \n-\tbio_associate_blkg_from_page(bio, folio);\n+\tbio_associate_blkcg_from_page(bio, folio);\n \tcount_swpout_vm_event(folio);\n \tfolio_start_writeback(folio);\n \tfolio_unlock(folio);\n"
}

{
  "EnableConfigs": [
    "BLK_CGROUP",
    "IOSCHED_BFQ",
    "BFQ_GROUP_IOSCHED",
    "BLK_CGROUP_IOCOST",
    "BLK_CGROUP_IOLATENCY",
    "BLK_CGROUP_IOPRIO",
    "NVME_FC",
    "BLK_CGROUP_FC_APPID",
    "BTRFS_FS"
  ],
  "FocusSymbols": [
    "blkg_conf_prep",
    "blkcg_punt_bio_submit",
    "blkg_init_queue",
    "blkg_exit_queue",
    "blkcg_init_disk",
    "blkcg_exit_disk"
  ],
  "Reasoning": "The patch significantly refactors the block cgroup (blkcg) subsystem by changing how `bio`s are associated with block cgroups. Instead of directly storing a queue-specific `blkcg_gq` (blkg) in `bio-\u003ebi_blkg`, it now stores the generic `blkcg` in `bio-\u003ebi_blkcg` and dynamically looks up or pins the queue-specific `blkg` using a new `BIO_BLKG_REF` flag. It also replaces the `radix_tree` used for `blkg` lookups with an `rhashtable`. These are core functional changes to the block layer and cgroup IO controllers, affecting IO scheduling, throttling, and accounting.",
  "WorthFuzzing": true
}

You are an expert Linux kernel maintainer.
Your job is to review a provided patch series and determine
if it makes functional changes to the kernel that should be fuzzed.

IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively
use your code access tools to examine the actual source code and confirm any assumptions.

Return WorthFuzzing=false if the patch only contains:
- Modifications to Documentation/, Kconfig files, or code comments.
- Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints.
- Changes to numeric constants or macros that do not functionally alter execution flow.
- Code paths that are impossible to reach in virtualized environments like GCE or QEMU,
  even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim).
- Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed,
  ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU.
- Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove)
  that are executed only during PCI hot-unplug or sysfs driver unbind operations.

If it modifies reachable core kernel logic, drivers, or architectures, use your code search
tools to verify the code can be executed, then return WorthFuzzing=true.

When returning WorthFuzzing=true, you MUST ALSO:
1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols.
   Avoid listing generic hot-path functions to prevent skewed test distributions.
   Prefer non-static, non-inlined API entrypoint functions over internal static helper functions
   (which are inlined by the compiler and do not have distinct symbol addresses).
2. Identify any specific CONFIG_ options required to properly test this new/modified feature.
   Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions.
   Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs
   needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs
   output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit abc0b1a65e87d979eec053ca54836b90fdec9729
Author: syz-cluster <triage@syzkaller.com>
Date:   Tue Aug 18 08:04:15 2026 +0000

    syz-cluster: applied patch under review

diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 14b8c571c0d14..bbd79931d5ab5 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -3235,7 +3235,7 @@ the configuration, the bio may be executed at a lower priority and if
 the writeback session is holding shared resources, e.g. a journal
 entry, may lead to priority inversion.  There is no one easy solution
 for the problem.  Filesystems can try to work around specific problem
-cases by skipping wbc_init_bio() and using bio_associate_blkg()
+cases by skipping wbc_init_bio() and using bio_associate_blkcg()
 directly.
 
 
diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
index ef9d63e8542ed..4a3975f9ff744 100644
--- a/block/bfq-cgroup.c
+++ b/block/bfq-cgroup.c
@@ -363,11 +363,13 @@ void bfqg_and_blkg_put(struct bfq_group *bfqg)
 
 void bfqg_stats_update_legacy_io(struct request_queue *q, struct request *rq)
 {
-	struct bfq_group *bfqg = blkg_to_bfqg(rq->bio->bi_blkg);
+	struct blkcg_gq *blkg = bio_blkg_lookup(rq->bio);
+	struct bfq_group *bfqg;
 
-	if (!bfqg)
+	if (!blkg)
 		return;
 
+	bfqg = blkg_to_bfqg(blkg);
 	blkg_rwstat_add(&bfqg->stats.bytes, rq->cmd_flags, blk_rq_bytes(rq));
 	blkg_rwstat_add(&bfqg->stats.ios, rq->cmd_flags, 1);
 }
@@ -606,7 +608,7 @@ static void bfq_link_bfqg(struct bfq_data *bfqd, struct bfq_group *bfqg)
 
 struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio)
 {
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg_gq *blkg = bio_blkg_lookup(bio);
 	struct bfq_group *bfqg;
 
 	while (blkg) {
@@ -614,15 +616,17 @@ struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio)
 			blkg = blkg->parent;
 			continue;
 		}
+
 		bfqg = blkg_to_bfqg(blkg);
 		if (bfqg->pd.online) {
-			bio_associate_blkg_from_css(bio, &blkg->blkcg->css);
+			bio_associate_blkcg_from_css(bio, &blkg->blkcg->css);
 			return bfqg;
 		}
 		blkg = blkg->parent;
 	}
-	bio_associate_blkg_from_css(bio,
-				&bfqg_to_blkg(bfqd->root_group)->blkcg->css);
+
+	blkg = bfqg_to_blkg(bfqd->root_group);
+	bio_associate_blkcg_from_css(bio, &blkg->blkcg->css);
 	return bfqd->root_group;
 }
 
diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 0f75301b3115f..d4f657bc7a4e5 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -128,6 +128,7 @@
 
 #include "elevator.h"
 #include "blk.h"
+#include "blk-cgroup.h"
 #include "blk-mq.h"
 #include "blk-mq-sched.h"
 #include "bfq-iosched.h"
@@ -2452,15 +2453,16 @@ static bool bfq_bio_merge(struct request_queue *q, struct bio *bio,
 	struct request *free = NULL;
 	bool ret;
 
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+	/* blkg creation takes q->queue_lock, so do it before bfqd->lock. */
+	if (bic)
+		bio_blkg(bio);
+#endif
+
 	spin_lock_irq(&bfqd->lock);
 
 	if (bic) {
-		/*
-		 * Make sure cgroup info is uptodate for current process before
-		 * considering the merge.
-		 */
 		bfq_bic_update_cgroup(bic, bio);
-
 		bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf),
 					     bfq_actuator_index(bfqd, bio));
 	} else {
@@ -6245,6 +6247,13 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,
 	LIST_HEAD(free);
 
 #ifdef CONFIG_BFQ_GROUP_IOSCHED
+	/*
+	 * Pin the blkg used to look up bfqg.  If this is the first IO for
+	 * the blkcg on this queue, create the bfqg before holding bfqd->lock.
+	 */
+	if (rq->bio && !bio_flagged(rq->bio, BIO_BLKG_REF))
+		bio_blkg(rq->bio);
+
 	if (!cgroup_subsys_on_dfl(io_cgrp_subsys) && rq->bio)
 		bfqg_stats_update_legacy_io(q, rq);
 #endif
diff --git a/block/bio.c b/block/bio.c
index 898b2f5ef8c81..6c84042adf5dd 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -179,12 +179,7 @@ static inline gfp_t try_alloc_gfp(gfp_t gfp)
 
 void bio_uninit(struct bio *bio)
 {
-#ifdef CONFIG_BLK_CGROUP
-	if (bio->bi_blkg) {
-		blkg_put(bio->bi_blkg);
-		bio->bi_blkg = NULL;
-	}
-#endif
+	bio_clear_blkcg(bio);
 	if (bio_integrity(bio))
 		bio_integrity_free(bio);
 
@@ -233,10 +228,10 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,
 	bio->bi_end_io = NULL;
 	bio->bi_private = NULL;
 #ifdef CONFIG_BLK_CGROUP
-	bio->bi_blkg = NULL;
+	bio->bi_blkcg = NULL;
 	bio->issue_time_ns = 0;
 	if (bdev)
-		bio_associate_blkg(bio);
+		bio_associate_blkcg(bio);
 #ifdef CONFIG_BLK_CGROUP_IOCOST
 	bio->bi_iocost_cost = 0;
 #endif
@@ -281,7 +276,7 @@ void bio_reset(struct bio *bio, struct block_device *bdev, blk_opf_t opf)
 	bio->bi_io_vec = bv;
 	bio->bi_bdev = bdev;
 	if (bio->bi_bdev)
-		bio_associate_blkg(bio);
+		bio_associate_blkcg(bio);
 	bio->bi_opf = opf;
 }
 EXPORT_SYMBOL(bio_reset);
@@ -866,7 +861,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
 		if (bio->bi_bdev == bio_src->bi_bdev &&
 		    bio_flagged(bio_src, BIO_REMAPPED))
 			bio_set_flag(bio, BIO_REMAPPED);
-		bio_clone_blkg_association(bio, bio_src);
+		bio_clone_blkcg_association(bio, bio_src);
 	}
 
 	if (bio_crypt_clone(bio, bio_src, gfp) < 0)
@@ -1910,17 +1905,12 @@ void bio_endio(struct bio *bio)
 		goto again;
 	}
 
-#ifdef CONFIG_BLK_CGROUP
 	/*
 	 * Release cgroup info.  We shouldn't have to do this here, but quite
 	 * a few callers of bio_init fail to call bio_uninit, so we cover up
 	 * for that here at least for now.
 	 */
-	if (bio->bi_blkg) {
-		blkg_put(bio->bi_blkg);
-		bio->bi_blkg = NULL;
-	}
-#endif
+	bio_clear_blkcg(bio);
 
 	if (bio_flagged(bio, BIO_COMPLETE_IN_TASK) && bio_in_atomic())
 		__bio_complete_in_task(bio);
diff --git a/block/blk-cgroup-fc-appid.c b/block/blk-cgroup-fc-appid.c
index 3ec21333f393b..7589c62099898 100644
--- a/block/blk-cgroup-fc-appid.c
+++ b/block/blk-cgroup-fc-appid.c
@@ -50,8 +50,14 @@ EXPORT_SYMBOL_GPL(blkcg_set_fc_appid);
  */
 char *blkcg_get_fc_appid(struct bio *bio)
 {
-	if (!bio->bi_blkg || bio->bi_blkg->blkcg->fc_app_id[0] == '\0')
+	struct blkcg *blkcg = bio_blkcg(bio);
+
+	if (!blkcg)
+		return NULL;
+
+	if (blkcg->fc_app_id[0] == '\0')
 		return NULL;
-	return bio->bi_blkg->blkcg->fc_app_id;
+
+	return blkcg->fc_app_id;
 }
 EXPORT_SYMBOL_GPL(blkcg_get_fc_appid);
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 2b5c29434e426..2c758c26bce60 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -66,6 +66,13 @@ static DEFINE_RAW_SPINLOCK(blkg_stat_lock);
 
 #define BLKG_DESTROY_BATCH_SIZE  64
 
+const struct rhashtable_params blkg_hash_params = {
+	.key_len		= sizeof_field(struct blkcg_gq, blkcg_id),
+	.key_offset		= offsetof(struct blkcg_gq, blkcg_id),
+	.head_offset		= offsetof(struct blkcg_gq, q_hash_node),
+	.automatic_shrinking	= true,
+};
+
 /*
  * Lockless lists for tracking IO stats update
  *
@@ -173,10 +180,6 @@ static void __blkg_release(struct rcu_head *rcu)
 {
 	struct blkcg_gq *blkg = container_of(rcu, struct blkcg_gq, rcu_head);
 
-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
-	WARN_ON(!bio_list_empty(&blkg->async_bios));
-#endif
-
 	blkg_free(blkg);
 }
 
@@ -194,6 +197,16 @@ static void blkg_release(struct percpu_ref *ref)
 	struct blkcg *blkcg = blkg->blkcg;
 	int cpu;
 
+	/*
+	 * A blkg that was never inserted into q->blkg_list has no hash
+	 * entry.  This happens when allocation or creation fails before
+	 * rhashtable_insert_fast() succeeds.
+	 */
+	if (!list_empty(&blkg->q_node))
+		WARN_ON_ONCE(rhashtable_remove_fast(&blkg->q->blkg_hash,
+						    &blkg->q_hash_node,
+						    blkg_hash_params));
+
 	/*
 	 * Flush all the non-empty percpu lockless lists before releasing
 	 * us, given these stat belongs to us.
@@ -209,19 +222,18 @@ static void blkg_release(struct percpu_ref *ref)
 #ifdef CONFIG_BLK_CGROUP_PUNT_BIO
 static struct workqueue_struct *blkcg_punt_bio_wq;
 
-static void blkg_async_bio_workfn(struct work_struct *work)
+static void blkcg_async_bio_workfn(struct work_struct *work)
 {
-	struct blkcg_gq *blkg = container_of(work, struct blkcg_gq,
-					     async_bio_work);
+	struct blkcg *blkcg = container_of(work, struct blkcg, async_bio_work);
 	struct bio_list bios = BIO_EMPTY_LIST;
 	struct bio *bio;
 	struct blk_plug plug;
 	bool need_plug = false;
 
-	/* as long as there are pending bios, @blkg can't go away */
-	spin_lock(&blkg->async_bio_lock);
-	bio_list_merge_init(&bios, &blkg->async_bios);
-	spin_unlock(&blkg->async_bio_lock);
+	/* as long as there are pending bios, @blkcg can't go away */
+	spin_lock(&blkcg->async_bio_lock);
+	bio_list_merge_init(&bios, &blkcg->async_bios);
+	spin_unlock(&blkcg->async_bio_lock);
 
 	/* start plug only when bio_list contains at least 2 bios */
 	if (bios.head && bios.head->bi_next) {
@@ -242,15 +254,15 @@ static void blkg_async_bio_workfn(struct work_struct *work)
  */
 void blkcg_punt_bio_submit(struct bio *bio)
 {
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg *blkcg = bio_blkcg(bio);
 
-	if (blkg->parent) {
-		spin_lock(&blkg->async_bio_lock);
-		bio_list_add(&blkg->async_bios, bio);
-		spin_unlock(&blkg->async_bio_lock);
-		queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work);
+	if (blkcg && cgroup_parent(blkcg->css.cgroup)) {
+		spin_lock(&blkcg->async_bio_lock);
+		bio_list_add(&blkcg->async_bios, bio);
+		spin_unlock(&blkcg->async_bio_lock);
+		queue_work(blkcg_punt_bio_wq, &blkcg->async_bio_work);
 	} else {
-		/* never bounce for the root cgroup */
+		/* Never bounce if there is no non-root blkcg to queue on. */
 		submit_bio(bio);
 	}
 }
@@ -278,9 +290,13 @@ subsys_initcall(blkcg_punt_bio_init);
  */
 struct cgroup_subsys_state *bio_blkcg_css(struct bio *bio)
 {
-	if (!bio || !bio->bi_blkg)
+	struct blkcg *blkcg;
+
+	if (!bio)
 		return NULL;
-	return &bio->bi_blkg->blkcg->css;
+
+	blkcg = bio_blkcg(bio);
+	return blkcg ? &blkcg->css : NULL;
 }
 EXPORT_SYMBOL_GPL(bio_blkcg_css);
 
@@ -327,12 +343,8 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct gendisk *disk,
 	blkg->q = disk->queue;
 	INIT_LIST_HEAD(&blkg->q_node);
 	blkg->blkcg = blkcg;
+	blkg->blkcg_id = blkcg->css.id;
 	blkg->iostat.blkg = blkg;
-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
-	spin_lock_init(&blkg->async_bio_lock);
-	bio_list_init(&blkg->async_bios);
-	INIT_WORK(&blkg->async_bio_work, blkg_async_bio_workfn);
-#endif
 
 	u64_stats_init(&blkg->iostat.sync);
 	for_each_possible_cpu(cpu) {
@@ -423,7 +435,8 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,
 
 	/* insert */
 	spin_lock(&blkcg->lock);
-	ret = radix_tree_insert(&blkcg->blkg_tree, disk->queue->id, blkg);
+	ret = rhashtable_insert_fast(&disk->queue->blkg_hash,
+				     &blkg->q_hash_node, blkg_hash_params);
 	if (likely(!ret)) {
 		hlist_add_head_rcu(&blkg->blkcg_node, &blkcg->blkg_list);
 		list_add(&blkg->q_node, &disk->queue->blkg_list);
@@ -454,6 +467,17 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,
 	return ERR_PTR(ret);
 }
 
+/*
+ * The root blkg holds a live reference while the disk is active, so walking
+ * the parent chain always finds a blkg which can be pinned.
+ */
+static struct blkcg_gq *blkg_lookup_tryget(struct blkcg_gq *blkg)
+{
+	while (!blkg_tryget(blkg))
+		blkg = blkg->parent;
+	return blkg;
+}
+
 /**
  * blkg_lookup_create - lookup blkg, try to create one if not there
  * @blkcg: blkcg of interest
@@ -461,11 +485,13 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,
  *
  * Lookup blkg for the @blkcg - @disk pair.  If it doesn't exist, try to
  * create one.  blkg creation is performed recursively from blkcg_root such
- * that all non-root blkg's have access to the parent blkg.  This function
- * should be called under RCU read lock and takes @disk->queue->queue_lock.
+ * that all non-root blkg's have access to the parent blkg.
  *
- * Returns the blkg or the closest blkg if blkg_create() fails as it walks
- * down from root.
+ * Must be called with @disk->queue->queue_lock held.
+ *
+ * Returns the closest blkg with an extra reference acquired.  If
+ * blkg_create() fails while walking down from root, the returned blkg may
+ * belong to an ancestor of @blkcg.  This function never returns %NULL.
  */
 static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,
 		struct gendisk *disk)
@@ -473,12 +499,12 @@ static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,
 	struct request_queue *q = disk->queue;
 	struct blkcg_gq *blkg;
 
+	lockdep_assert_held(&q->queue_lock);
+
 	rcu_read_lock();
 	blkg = blkg_lookup(blkcg, q);
 	if (blkg) {
-		if (blkcg != &blkcg_root &&
-		    blkg != rcu_dereference(blkcg->blkg_hint))
-			rcu_assign_pointer(blkcg->blkg_hint, blkg);
+		blkg = blkg_lookup_tryget(blkg);
 		rcu_read_unlock();
 		return blkg;
 	}
@@ -514,7 +540,7 @@ static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,
 			break;
 	}
 
-	return blkg;
+	return blkg_lookup_tryget(blkg);
 }
 
 static void blkg_destroy(struct blkcg_gq *blkg)
@@ -546,17 +572,8 @@ static void blkg_destroy(struct blkcg_gq *blkg)
 
 	blkg->online = false;
 
-	radix_tree_delete(&blkcg->blkg_tree, blkg->q->id);
 	hlist_del_init_rcu(&blkg->blkcg_node);
 
-	/*
-	 * Both setting lookup hint to and clearing it from @blkg are done
-	 * under queue_lock.  If it's not pointing to @blkg now, it never
-	 * will.  Hint assignment itself can race safely.
-	 */
-	if (rcu_access_pointer(blkcg->blkg_hint) == blkg)
-		rcu_assign_pointer(blkcg->blkg_hint, NULL);
-
 	/*
 	 * Put the reference taken at the time of creation so that when all
 	 * queues are gone, group can be destroyed.
@@ -882,18 +899,12 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 			goto fail_exit;
 		}
 
-		if (radix_tree_preload(GFP_KERNEL)) {
-			blkg_free(new_blkg);
-			ret = -ENOMEM;
-			goto fail_exit;
-		}
-
 		spin_lock_irq(&q->queue_lock);
 
 		if (!blkcg_policy_enabled(q, pol)) {
 			blkg_free(new_blkg);
 			ret = -EOPNOTSUPP;
-			goto fail_preloaded;
+			goto fail_unlock;
 		}
 
 		blkg = blkg_lookup(pos, q);
@@ -903,12 +914,10 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 			blkg = blkg_create(pos, disk, new_blkg);
 			if (IS_ERR(blkg)) {
 				ret = PTR_ERR(blkg);
-				goto fail_preloaded;
+				goto fail_unlock;
 			}
 		}
 
-		radix_tree_preload_end();
-
 		if (pos == blkcg)
 			goto success;
 	}
@@ -917,8 +926,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 	ctx->blkg = blkg;
 	return 0;
 
-fail_preloaded:
-	radix_tree_preload_end();
 fail_unlock:
 	spin_unlock_irq(&q->queue_lock);
 fail_exit:
@@ -1373,6 +1380,9 @@ static void blkcg_css_free(struct cgroup_subsys_state *css)
 
 	mutex_unlock(&blkcg_pol_mutex);
 
+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
+	WARN_ON(!bio_list_empty(&blkcg->async_bios));
+#endif
 	free_percpu(blkcg->lhead);
 	kfree(blkcg);
 }
@@ -1420,8 +1430,12 @@ blkcg_css_alloc(struct cgroup_subsys_state *parent_css)
 
 	spin_lock_init(&blkcg->lock);
 	refcount_set(&blkcg->online_pin, 1);
-	INIT_RADIX_TREE(&blkcg->blkg_tree, GFP_NOWAIT);
 	INIT_HLIST_HEAD(&blkcg->blkg_list);
+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
+	spin_lock_init(&blkcg->async_bio_lock);
+	bio_list_init(&blkcg->async_bios);
+	INIT_WORK(&blkcg->async_bio_work, blkcg_async_bio_workfn);
+#endif
 #ifdef CONFIG_CGROUP_WRITEBACK
 	INIT_LIST_HEAD(&blkcg->cgwb_list);
 #endif
@@ -1457,17 +1471,22 @@ static int blkcg_css_online(struct cgroup_subsys_state *css)
 	return 0;
 }
 
-void blkg_init_queue(struct request_queue *q)
+int blkg_init_queue(struct request_queue *q)
 {
 	INIT_LIST_HEAD(&q->blkg_list);
 	mutex_init(&q->blkcg_mutex);
+	return rhashtable_init(&q->blkg_hash, &blkg_hash_params);
+}
+
+void blkg_exit_queue(struct request_queue *q)
+{
+	rhashtable_destroy(&q->blkg_hash);
 }
 
 int blkcg_init_disk(struct gendisk *disk)
 {
 	struct request_queue *q = disk->queue;
 	struct blkcg_gq *new_blkg, *blkg;
-	bool preloaded;
 
 	/*
 	 * If the queue is shared across disk rebind (e.g., SCSI), the
@@ -1485,8 +1504,6 @@ int blkcg_init_disk(struct gendisk *disk)
 	if (!new_blkg)
 		return -ENOMEM;
 
-	preloaded = !radix_tree_preload(GFP_KERNEL);
-
 	/* Make sure the root blkg exists. */
 	/* spin_lock_irq can serve as RCU read-side critical section. */
 	spin_lock_irq(&q->queue_lock);
@@ -1496,15 +1513,10 @@ int blkcg_init_disk(struct gendisk *disk)
 	q->root_blkg = blkg;
 	spin_unlock_irq(&q->queue_lock);
 
-	if (preloaded)
-		radix_tree_preload_end();
-
 	return 0;
 
 err_unlock:
 	spin_unlock_irq(&q->queue_lock);
-	if (preloaded)
-		radix_tree_preload_end();
 	return PTR_ERR(blkg);
 }
 
@@ -2064,129 +2076,165 @@ void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta)
 	atomic64_add(delta, &blkg->delay_nsec);
 }
 
-static inline struct blkcg_gq *blkg_lookup_tryget(struct blkcg_gq *blkg)
+/*
+ * Return the blkg pinned by @bio through BIO_BLKG_REF.  The returned blkg is
+ * already owned by @bio and no extra reference is acquired.
+ */
+static struct blkcg_gq *bio_pinned_blkg(struct bio *bio)
 {
-retry:
-	if (blkg_tryget(blkg))
-		return blkg;
+	struct request_queue *q = bdev_get_queue(bio->bi_bdev);
+	struct blkcg_gq *blkg;
 
-	blkg = blkg->parent;
-	if (blkg)
-		goto retry;
+	rcu_read_lock();
+	blkg = blkg_lookup_any(bio_blkcg(bio), q);
+	rcu_read_unlock();
 
-	return NULL;
+	WARN_ON_ONCE(!blkg);
+	return blkg;
+}
+
+static void bio_set_blkg_ref(struct bio *bio, struct blkcg_gq *blkg)
+{
+	if (bio_blkcg(bio) != blkg->blkcg) {
+		css_get(&blkg->blkcg->css);
+		bio_clear_blkcg(bio);
+		bio->bi_blkcg = blkg->blkcg;
+	}
+	bio_set_flag(bio, BIO_BLKG_REF);
 }
+
 /**
- * blkg_tryget_closest - try and get a blkg ref on the closet blkg
+ * bio_blkg_lookup - look up a blkg associated with a bio
  * @bio: target bio
- * @css: target css
  *
- * As the failure mode here is to walk up the blkg tree, this ensure that the
- * blkg->parent pointers are always valid.  This returns the blkg that it ended
- * up taking a reference on or %NULL if no reference was taken.
+ * Return the blkg already pinned by @bio, or %NULL if @bio doesn't own a blkg
+ * reference.  Call bio_blkg() instead when a missing blkg should be created.
  */
-static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio,
-		struct cgroup_subsys_state *css)
+struct blkcg_gq *bio_blkg_lookup(struct bio *bio)
 {
-	struct request_queue *q = bio->bi_bdev->bd_queue;
-	struct blkcg *blkcg = css_to_blkcg(css);
+	if (!bio_flagged(bio, BIO_BLKG_REF))
+		return NULL;
+	return bio_pinned_blkg(bio);
+}
+EXPORT_SYMBOL_GPL(bio_blkg_lookup);
+
+/**
+ * bio_put_blkg_ref - drop the blkg reference pinned by a bio
+ * @bio: target bio
+ *
+ * Drop the bio-owned blkg reference acquired by bio_blkg(), if any.
+ */
+void bio_put_blkg_ref(struct bio *bio)
+{
+	if (bio_flagged(bio, BIO_BLKG_REF)) {
+		struct blkcg_gq *blkg = bio_pinned_blkg(bio);
+
+		if (blkg)
+			blkg_put(blkg);
+		bio_clear_flag(bio, BIO_BLKG_REF);
+	}
+}
+EXPORT_SYMBOL_GPL(bio_put_blkg_ref);
+
+/**
+ * bio_blkg - look up the blkg associated with a bio
+ * @bio: target bio
+ *
+ * Look up the queue-local blkg for @bio's current device and blkcg.  If this
+ * is the first policy use of @bio, create the missing blkg hierarchy if
+ * necessary, pin the closest available blkg, and update @bio's blkcg
+ * association if allocation falls back to an ancestor.
+ */
+struct blkcg_gq *bio_blkg(struct bio *bio)
+{
+	struct blkcg *blkcg = bio_blkcg(bio);
+	struct gendisk *disk;
+	struct request_queue *q;
 	struct blkcg_gq *blkg;
 
-	rcu_read_lock();
-	blkg = blkg_lookup(blkcg, q);
-	if (likely(blkg))
-		blkg = blkg_lookup_tryget(blkg);
-	rcu_read_unlock();
+	if (!blkcg || !bio->bi_bdev)
+		return NULL;
 
-	if (blkg)
-		return blkg;
+	if (bio_flagged(bio, BIO_BLKG_REF))
+		return bio_pinned_blkg(bio);
+
+	disk = bio->bi_bdev->bd_disk;
+	q = disk->queue;
 
-	/*
-	 * Fast path failed, we're probably issuing IO in this cgroup the first
-	 * time, hold lock to create new blkg.
-	 */
 	spin_lock_irq(&q->queue_lock);
-	blkg = blkg_lookup_create(blkcg, bio->bi_bdev->bd_disk);
-	if (blkg)
-		blkg = blkg_lookup_tryget(blkg);
+	blkg = blkg_lookup_create(blkcg, disk);
 	spin_unlock_irq(&q->queue_lock);
 
+	bio_set_blkg_ref(bio, blkg);
 	return blkg;
 }
+EXPORT_SYMBOL_GPL(bio_blkg);
 
 /**
- * bio_associate_blkg_from_css - associate a bio with a specified css
+ * bio_associate_blkcg_from_css - associate a bio with a specified css
  * @bio: target bio
  * @css: target css
  *
- * Associate @bio with the blkg found by combining the css's blkg and the
- * request_queue of the @bio.  An association failure is handled by walking up
- * the blkg tree.  Therefore, the blkg associated can be anything between @blkg
- * and q->root_blkg.  This situation only happens when a cgroup is dying and
- * then the remaining bios will spill to the closest alive blkg.
+ * Associate @bio with the blkcg found from @css.  The queue-local blkg is
+ * created and pinned by bio_blkg() when blkcg policies need it.  If @css
+ * can't be referenced online, associate @bio with the root blkcg instead.
  *
- * A reference will be taken on the blkg and will be released when @bio is
+ * A reference will be taken on the blkcg and will be released when @bio is
  * freed.
  */
-void bio_associate_blkg_from_css(struct bio *bio,
-				 struct cgroup_subsys_state *css)
+void bio_associate_blkcg_from_css(struct bio *bio,
+				  struct cgroup_subsys_state *css)
 {
-	if (bio->bi_blkg)
-		blkg_put(bio->bi_blkg);
+	struct blkcg *blkcg;
 
-	if (css && css->parent) {
-		bio->bi_blkg = blkg_tryget_closest(bio, css);
-	} else {
-		blkg_get(bdev_get_queue(bio->bi_bdev)->root_blkg);
-		bio->bi_blkg = bdev_get_queue(bio->bi_bdev)->root_blkg;
+	if (!css || !css_tryget_online(css)) {
+		css = &blkcg_root.css;
+		css_get(css);
 	}
+
+	blkcg = css_to_blkcg(css);
+	if (bio_blkcg(bio) == blkcg) {
+		css_put(css);
+		return;
+	}
+
+	bio_clear_blkcg(bio);
+	bio->bi_blkcg = blkcg;
 }
-EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css);
+EXPORT_SYMBOL_GPL(bio_associate_blkcg_from_css);
 
 /**
- * bio_associate_blkg - associate a bio with a blkg
+ * bio_associate_blkcg - associate a bio with a blkcg
  * @bio: target bio
  *
- * Associate @bio with the blkg found from the bio's css and request_queue.
- * If one is not found, bio_lookup_blkg() creates the blkg.  If a blkg is
- * already associated, the css is reused and association redone as the
- * request_queue may have changed.
+ * Associate @bio with the blkcg found from the bio's css.  If a blkcg is
+ * already associated, keep it as blkcg association is not queue-local.
  */
-void bio_associate_blkg(struct bio *bio)
+void bio_associate_blkcg(struct bio *bio)
 {
-	struct cgroup_subsys_state *css;
-
 	if (blk_op_is_passthrough(bio->bi_opf))
 		return;
 
-	if (bio->bi_blkg) {
-		css = bio_blkcg_css(bio);
-		bio_associate_blkg_from_css(bio, css);
-	} else {
-		rcu_read_lock();
-		css = blkcg_css();
-		if (!css_tryget_online(css))
-			css = NULL;
-		rcu_read_unlock();
+	if (bio_blkcg(bio))
+		return;
 
-		bio_associate_blkg_from_css(bio, css);
-		if (css)
-			css_put(css);
-	}
+	rcu_read_lock();
+	bio_associate_blkcg_from_css(bio, blkcg_css());
+	rcu_read_unlock();
 }
-EXPORT_SYMBOL_GPL(bio_associate_blkg);
+EXPORT_SYMBOL_GPL(bio_associate_blkcg);
 
 /**
- * bio_clone_blkg_association - clone blkg association from src to dst bio
+ * bio_clone_blkcg_association - clone blkcg association from src to dst bio
  * @dst: destination bio
  * @src: source bio
  */
-void bio_clone_blkg_association(struct bio *dst, struct bio *src)
+void bio_clone_blkcg_association(struct bio *dst, struct bio *src)
 {
-	if (src->bi_blkg)
-		bio_associate_blkg_from_css(dst, bio_blkcg_css(src));
+	if (bio_blkcg(src))
+		bio_associate_blkcg_from_css(dst, bio_blkcg_css(src));
 }
-EXPORT_SYMBOL_GPL(bio_clone_blkg_association);
+EXPORT_SYMBOL_GPL(bio_clone_blkcg_association);
 
 static int blk_cgroup_io_type(struct bio *bio)
 {
@@ -2199,20 +2247,27 @@ static int blk_cgroup_io_type(struct bio *bio)
 
 void blk_cgroup_bio_start(struct bio *bio)
 {
-	struct blkcg *blkcg = bio->bi_blkg->blkcg;
+	struct blkcg *blkcg = bio_blkcg(bio);
+	struct blkcg_gq *blkg;
 	int rwd = blk_cgroup_io_type(bio), cpu;
 	struct blkg_iostat_set *bis;
 	unsigned long flags;
 
 	if (!cgroup_subsys_on_dfl(io_cgrp_subsys))
 		return;
+	if (!blkcg)
+		return;
 
 	/* Root-level stats are sourced from system-wide IO stats */
 	if (!cgroup_parent(blkcg->css.cgroup))
 		return;
 
+	blkg = bio_blkg_lookup(bio);
+	if (!blkg)
+		return;
+
 	cpu = get_cpu();
-	bis = per_cpu_ptr(bio->bi_blkg->iostat_cpu, cpu);
+	bis = per_cpu_ptr(blkg->iostat_cpu, cpu);
 	flags = u64_stats_update_begin_irqsave(&bis->sync);
 
 	/*
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index e67c698391294..eb76cc38b41ba 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -19,6 +19,8 @@
 #include <linux/kthread.h>
 #include <linux/blk-mq.h>
 #include <linux/llist.h>
+#include <linux/rhashtable.h>
+#include <linux/rcupdate.h>
 #include "blk.h"
 
 struct blkcg_gq;
@@ -56,9 +58,11 @@ struct blkg_iostat_set {
 struct blkcg_gq {
 	/* Pointer to the associated request_queue */
 	struct request_queue		*q;
+	struct rhash_head		q_hash_node;
 	struct list_head		q_node;
 	struct hlist_node		blkcg_node;
 	struct blkcg			*blkcg;
+	int				blkcg_id;
 
 	/* all non-root blkcg_gq's are guaranteed to have access to parent */
 	struct blkcg_gq			*parent;
@@ -73,14 +77,7 @@ struct blkcg_gq {
 	struct blkg_iostat_set		iostat;
 
 	struct blkg_policy_data		*pd[BLKCG_MAX_POLS];
-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
-	spinlock_t			async_bio_lock;
-	struct bio_list			async_bios;
-#endif
-	union {
-		struct work_struct	async_bio_work;
-		struct work_struct	free_work;
-	};
+	struct work_struct		free_work;
 
 	atomic_t			use_delay;
 	atomic64_t			delay_nsec;
@@ -98,8 +95,6 @@ struct blkcg {
 	/* If there is block congestion on this cgroup. */
 	atomic_t			congestion_count;
 
-	struct radix_tree_root		blkg_tree;
-	struct blkcg_gq	__rcu		*blkg_hint;
 	struct hlist_head		blkg_list;
 
 	struct blkcg_policy_data	*cpd[BLKCG_MAX_POLS];
@@ -111,6 +106,11 @@ struct blkcg {
 	 */
 	struct llist_head __percpu	*lhead;
 
+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
+	spinlock_t			async_bio_lock; /* protects async_bios */
+	struct bio_list			async_bios;
+	struct work_struct		async_bio_work;
+#endif
 #ifdef CONFIG_BLK_CGROUP_FC_APPID
 	char                            fc_app_id[FC_APPID_LEN];
 #endif
@@ -124,6 +124,11 @@ static inline struct blkcg *css_to_blkcg(struct cgroup_subsys_state *css)
 	return css ? container_of(css, struct blkcg, css) : NULL;
 }
 
+static inline struct blkcg *bio_blkcg(struct bio *bio)
+{
+	return bio->bi_blkcg;
+}
+
 /*
  * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a
  * request_queue (q).  This is used by blkcg policies which need to track
@@ -192,8 +197,10 @@ struct blkcg_policy {
 
 extern struct blkcg blkcg_root;
 extern bool blkcg_debug_stats;
+extern const struct rhashtable_params blkg_hash_params;
 
-void blkg_init_queue(struct request_queue *q);
+int blkg_init_queue(struct request_queue *q);
+void blkg_exit_queue(struct request_queue *q);
 int blkcg_init_disk(struct gendisk *disk);
 void blkcg_exit_disk(struct gendisk *disk);
 
@@ -249,11 +256,33 @@ static inline bool bio_issue_as_root_blkg(struct bio *bio)
 }
 
 /**
- * blkg_lookup - lookup blkg for the specified blkcg - q pair
+ * blkg_lookup_any - lookup any blkg for the specified blkcg - q pair
+ * @blkcg: blkcg of interest
+ * @q: request_queue of interest
+ *
+ * Lookup a blkg for the @blkcg - @q pair, whether it is online or dying.
+ *
+ * Must be called in a RCU critical section.
+ *
+ * This does not acquire a reference.  The caller must already hold one, or
+ * have the blkg pinned by I/O.
+ */
+static inline struct blkcg_gq *blkg_lookup_any(struct blkcg *blkcg,
+					       struct request_queue *q)
+{
+	RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
+			 "blkg_lookup_any() requires an RCU read lock");
+
+	return rhashtable_lookup(&q->blkg_hash, &blkcg->css.id,
+				 blkg_hash_params);
+}
+
+/**
+ * blkg_lookup - lookup an online blkg for the specified blkcg - q pair
  * @blkcg: blkcg of interest
  * @q: request_queue of interest
  *
- * Lookup blkg for the @blkcg - @q pair.
+ * Lookup an online blkg for the @blkcg - @q pair.
  *
  * Must be called in a RCU critical section.
  */
@@ -265,17 +294,15 @@ static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg,
 	if (blkcg == &blkcg_root)
 		return q->root_blkg;
 
-	blkg = rcu_dereference_check(blkcg->blkg_hint,
-			lockdep_is_held(&q->queue_lock));
-	if (blkg && blkg->q == q)
-		return blkg;
-
-	blkg = radix_tree_lookup(&blkcg->blkg_tree, q->id);
-	if (blkg && blkg->q != q)
+	blkg = blkg_lookup_any(blkcg, q);
+	if (blkg && !READ_ONCE(blkg->online))
 		blkg = NULL;
 	return blkg;
 }
 
+struct blkcg_gq *bio_blkg_lookup(struct bio *bio);
+struct blkcg_gq *bio_blkg(struct bio *bio);
+
 /**
  * blkg_to_pd - get policy private data
  * @blkg: blkg of interest
@@ -343,6 +370,18 @@ static inline void blkg_put(struct blkcg_gq *blkg)
 	percpu_ref_put(&blkg->refcnt);
 }
 
+static inline void bio_clear_blkcg(struct bio *bio)
+{
+	struct blkcg *blkcg = bio_blkcg(bio);
+
+	bio_put_blkg_ref(bio);
+
+	if (blkcg) {
+		css_put(&blkcg->css);
+		bio->bi_blkcg = NULL;
+	}
+}
+
 /**
  * blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants
  * @d_blkg: loop cursor pointing to the current descendant
@@ -472,7 +511,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq *blkg)
  */
 static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bio)
 {
-	return rq->bio->bi_blkg == bio->bi_blkg &&
+	return bio_blkcg(rq->bio) == bio_blkcg(bio) &&
 		bio_issue_as_root_blkg(rq->bio) == bio_issue_as_root_blkg(bio);
 }
 
@@ -498,8 +537,13 @@ struct blkcg_policy {
 struct blkcg {
 };
 
+static inline struct blkcg *bio_blkcg(struct bio *bio) { return NULL; }
+static inline struct blkcg_gq *bio_blkg_lookup(struct bio *bio) { return NULL; }
+static inline struct blkcg_gq *bio_blkg(struct bio *bio) { return NULL; }
+static inline struct blkcg_gq *blkg_lookup_any(struct blkcg *blkcg, void *key) { return NULL; }
 static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, void *key) { return NULL; }
-static inline void blkg_init_queue(struct request_queue *q) { }
+static inline int blkg_init_queue(struct request_queue *q) { return 0; }
+static inline void blkg_exit_queue(struct request_queue *q) { }
 static inline int blkcg_init_disk(struct gendisk *disk) { return 0; }
 static inline void blkcg_exit_disk(struct gendisk *disk) { }
 static inline int blkcg_policy_register(struct blkcg_policy *pol) { return 0; }
@@ -514,6 +558,7 @@ static inline struct blkg_policy_data *blkg_to_pd(struct blkcg_gq *blkg,
 static inline struct blkcg_gq *pd_to_blkg(struct blkg_policy_data *pd) { return NULL; }
 static inline void blkg_get(struct blkcg_gq *blkg) { }
 static inline void blkg_put(struct blkcg_gq *blkg) { }
+static inline void bio_clear_blkcg(struct bio *bio) { }
 static inline void blk_cgroup_bio_start(struct bio *bio) { }
 static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bio) { return true; }
 
diff --git a/block/blk-core.c b/block/blk-core.c
index 196bccf27f58d..e396a316b8bf6 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -303,6 +303,7 @@ static void blk_free_queue(struct request_queue *q)
 	if (queue_is_mq(q))
 		blk_mq_release(q);
 
+	blkg_exit_queue(q);
 	ida_free(&blk_queue_ida, q->id);
 	lockdep_unregister_key(&q->io_lock_cls_key);
 	lockdep_unregister_key(&q->q_lock_cls_key);
@@ -481,7 +482,9 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)
 	init_waitqueue_head(&q->mq_freeze_wq);
 	mutex_init(&q->mq_freeze_lock);
 
-	blkg_init_queue(q);
+	error = blkg_init_queue(q);
+	if (error)
+		goto fail_stats;
 
 	/*
 	 * Init percpu_ref in atomic mode so that it's faster to shutdown.
@@ -491,7 +494,7 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)
 				blk_queue_usage_counter_release,
 				PERCPU_REF_INIT_ATOMIC, GFP_KERNEL);
 	if (error)
-		goto fail_stats;
+		goto fail_blkg;
 	lockdep_register_key(&q->io_lock_cls_key);
 	lockdep_register_key(&q->q_lock_cls_key);
 	lockdep_init_map(&q->io_lockdep_map, "&q->q_usage_counter(io)",
@@ -510,6 +513,8 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)
 
 	return q;
 
+fail_blkg:
+	blkg_exit_queue(q);
 fail_stats:
 	blk_free_queue_stats(q->stats);
 fail_id:
diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c
index 2a5c52ab74b4c..5ef4baab444bb 100644
--- a/block/blk-crypto-fallback.c
+++ b/block/blk-crypto-fallback.c
@@ -187,7 +187,7 @@ static struct bio *blk_crypto_alloc_enc_bio(struct bio *bio_src,
 	bio->bi_write_hint	= bio_src->bi_write_hint;
 	bio->bi_write_stream	= bio_src->bi_write_stream;
 	bio->bi_iter.bi_sector	= bio_src->bi_iter.bi_sector;
-	bio_clone_blkg_association(bio, bio_src);
+	bio_clone_blkcg_association(bio, bio_src);
 
 	/*
 	 * Move page array up in the allocated memory for the bio vecs as far as
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 2745bffcd5eef..57f2b4d4af20a 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -2686,7 +2686,7 @@ iocg_handle_over_budget(struct rq_qos *rqos, struct ioc_gq *iocg,
 
 static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)
 {
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg_gq *blkg = bio_blkg(bio);
 	struct ioc *ioc = rqos_to_ioc(rqos);
 	struct ioc_gq *iocg = blkg_to_iocg(blkg);
 	struct ioc_now now;
@@ -2775,7 +2775,7 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)
 static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq,
 			   struct bio *bio)
 {
-	struct ioc_gq *iocg = blkg_to_iocg(bio->bi_blkg);
+	struct ioc_gq *iocg = blkg_to_iocg(bio_blkg(bio));
 	struct ioc *ioc = rqos_to_ioc(rqos);
 	sector_t bio_end = bio_end_sector(bio);
 	struct ioc_now now;
@@ -2833,9 +2833,13 @@ static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq,
 
 static void ioc_rqos_done_bio(struct rq_qos *rqos, struct bio *bio)
 {
-	struct ioc_gq *iocg = blkg_to_iocg(bio->bi_blkg);
+	struct ioc_gq *iocg;
+
+	if (!bio->bi_iocost_cost)
+		return;
 
-	if (iocg && bio->bi_iocost_cost)
+	iocg = blkg_to_iocg(bio_blkg_lookup(bio));
+	if (iocg)
 		atomic64_add(bio->bi_iocost_cost, &iocg->done_vtime);
 }
 
diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index 2caa79a008ad1..7220edafd72b6 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -463,7 +463,7 @@ static void check_scale_change(struct iolatency_grp *iolat)
 static void blkcg_iolatency_throttle(struct rq_qos *rqos, struct bio *bio)
 {
 	struct blk_iolatency *blkiolat = BLKIOLATENCY(rqos);
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg_gq *blkg = bio_blkg(bio);
 	bool issue_as_root = bio_issue_as_root_blkg(bio);
 
 	if (!blkiolat->enabled)
@@ -590,11 +590,14 @@ static void blkcg_iolatency_done_bio(struct rq_qos *rqos, struct bio *bio)
 	bool issue_as_root = bio_issue_as_root_blkg(bio);
 	int inflight = 0;
 
-	blkg = bio->bi_blkg;
-	if (!blkg || !bio_flagged(bio, BIO_QOS_THROTTLED))
+	if (!bio_flagged(bio, BIO_QOS_THROTTLED))
 		return;
 
-	iolat = blkg_to_lat(bio->bi_blkg);
+	blkg = bio_blkg_lookup(bio);
+	if (!blkg)
+		return;
+
+	iolat = blkg_to_lat(blkg);
 	if (!iolat)
 		return;
 
diff --git a/block/blk-ioprio.c b/block/blk-ioprio.c
index 8fa8bca350629..5e3f8e49550da 100644
--- a/block/blk-ioprio.c
+++ b/block/blk-ioprio.c
@@ -132,7 +132,7 @@ static struct blkcg_policy ioprio_policy = {
 
 void blkcg_set_ioprio(struct bio *bio)
 {
-	struct ioprio_blkcg *blkcg = blkcg_to_ioprio_blkcg(bio->bi_blkg->blkcg);
+	struct ioprio_blkcg *blkcg = blkcg_to_ioprio_blkcg(bio_blkcg(bio));
 	u16 prio;
 
 	if (!blkcg || blkcg->prio_policy == POLICY_NO_CHANGE)
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index ffc3b70065d4b..3828c38579002 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1792,7 +1792,7 @@ static bool tg_within_limit(struct throtl_grp *tg, struct bio *bio, bool rw)
 bool __blk_throtl_bio(struct bio *bio)
 {
 	struct request_queue *q = bdev_get_queue(bio->bi_bdev);
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg_gq *blkg = bio_blkg(bio);
 	struct throtl_qnode *qn = NULL;
 	struct throtl_grp *tg = blkg_to_tg(blkg);
 	struct throtl_service_queue *sq;
diff --git a/block/blk-throttle.h b/block/blk-throttle.h
index 1b5775771e078..45f30179fd9c0 100644
--- a/block/blk-throttle.h
+++ b/block/blk-throttle.h
@@ -171,7 +171,7 @@ static inline bool blk_should_throtl(struct bio *bio)
 	if (!blk_throtl_activated(bio->bi_bdev->bd_queue))
 		return false;
 
-	tg = blkg_to_tg(bio->bi_blkg);
+	tg = blkg_to_tg(bio_blkg(bio));
 	if (!cgroup_subsys_on_dfl(io_cgrp_subsys)) {
 		if (!bio_flagged(bio, BIO_CGROUP_ACCT)) {
 			bio_set_flag(bio, BIO_CGROUP_ACCT);
diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
index 3fa3b13a410f4..c0f945b8d9413 100644
--- a/drivers/md/bcache/request.c
+++ b/drivers/md/bcache/request.c
@@ -848,7 +848,7 @@ static CLOSURE_CALLBACK(cached_dev_read_done)
 		s->iop.bio->bi_iter.bi_sector =
 			s->cache_miss->bi_iter.bi_sector;
 		s->iop.bio->bi_iter.bi_size = s->insert_bio_sectors << 9;
-		bio_clone_blkg_association(s->iop.bio, s->cache_miss);
+		bio_clone_blkcg_association(s->iop.bio, s->cache_miss);
 		bch_bio_map(s->iop.bio, NULL);
 
 		bio_copy_data(s->cache_miss, s->iop.bio);
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index d413bfaf35278..cd68eec77f5a1 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1373,7 +1373,7 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone)
 	if (!tgt_clone)
 		tgt_clone = clone;
 
-	bio_clone_blkg_association(tgt_clone, io->orig_bio);
+	bio_clone_blkcg_association(tgt_clone, io->orig_bio);
 
 	/*
 	 * Account io->origin_bio to DM dev on behalf of target
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 680b34a63cb3d..fd78d6cc8da17 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -9386,7 +9386,7 @@ void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,
 		return;
 
 	bio_chain(discard_bio, bio);
-	bio_clone_blkg_association(discard_bio, bio);
+	bio_clone_blkcg_association(discard_bio, bio);
 	mddev_trace_remap(mddev, discard_bio, bio->bi_iter.bi_sector);
 	submit_bio_noacct(discard_bio);
 }
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 4176046627beb..54e4adb0ccb7d 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -121,7 +121,7 @@ int async_pmem_flush(struct nd_region *nd_region, struct bio *bio)
 
 		if (!child)
 			return -ENOMEM;
-		bio_clone_blkg_association(child, bio);
+		bio_clone_blkcg_association(child, bio);
 		child->bi_iter.bi_sector = -1;
 		bio_chain(child, bio);
 		submit_bio(child);
diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c
index 6dabe73ad790d..6512dbd9516fc 100644
--- a/fs/gfs2/lops.c
+++ b/fs/gfs2/lops.c
@@ -484,7 +484,7 @@ static struct bio *gfs2_chain_bio(struct bio *prev, unsigned int nr_iovecs,
 	struct bio *new;
 
 	new = bio_alloc(prev->bi_bdev, nr_iovecs, opf, GFP_NOIO);
-	bio_clone_blkg_association(new, prev);
+	bio_clone_blkcg_association(new, prev);
 	new->bi_iter.bi_sector = sector;
 	bio_chain(new, prev);
 	submit_bio(prev);
@@ -1114,4 +1114,3 @@ const struct gfs2_log_operations *gfs2_log_ops[] = {
 	&gfs2_revoke_lops,
 	NULL,
 };
-
diff --git a/include/linux/bio.h b/include/linux/bio.h
index bb3235497e673..11637f85c38d4 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -542,18 +542,20 @@ static inline void bio_release_pages(struct bio *bio, bool mark_dirty)
 	disk_devt((bio)->bi_bdev->bd_disk)
 
 #ifdef CONFIG_BLK_CGROUP
-void bio_associate_blkg(struct bio *bio);
-void bio_associate_blkg_from_css(struct bio *bio,
-				 struct cgroup_subsys_state *css);
-void bio_clone_blkg_association(struct bio *dst, struct bio *src);
+void bio_associate_blkcg(struct bio *bio);
+void bio_associate_blkcg_from_css(struct bio *bio,
+				  struct cgroup_subsys_state *css);
+void bio_clone_blkcg_association(struct bio *dst, struct bio *src);
+void bio_put_blkg_ref(struct bio *bio);
 void blkcg_punt_bio_submit(struct bio *bio);
 #else	/* CONFIG_BLK_CGROUP */
-static inline void bio_associate_blkg(struct bio *bio) { }
-static inline void bio_associate_blkg_from_css(struct bio *bio,
-					       struct cgroup_subsys_state *css)
+static inline void bio_associate_blkcg(struct bio *bio) { }
+static inline void bio_associate_blkcg_from_css(struct bio *bio,
+						struct cgroup_subsys_state *css)
 { }
-static inline void bio_clone_blkg_association(struct bio *dst,
-					      struct bio *src) { }
+static inline void bio_clone_blkcg_association(struct bio *dst,
+					       struct bio *src) { }
+static inline void bio_put_blkg_ref(struct bio *bio) { }
 static inline void blkcg_punt_bio_submit(struct bio *bio)
 {
 	submit_bio(bio);
@@ -563,10 +565,12 @@ static inline void blkcg_punt_bio_submit(struct bio *bio)
 static inline void bio_set_dev(struct bio *bio, struct block_device *bdev)
 {
 	bio_clear_flag(bio, BIO_REMAPPED);
-	if (bio->bi_bdev != bdev)
+	if (bio->bi_bdev != bdev) {
+		bio_put_blkg_ref(bio);
 		bio_clear_flag(bio, BIO_BPS_THROTTLED);
+	}
 	bio->bi_bdev = bdev;
-	bio_associate_blkg(bio);
+	bio_associate_blkcg(bio);
 }
 
 /*
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index d49d97a050d0e..53ba307b91cb3 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -246,12 +246,10 @@ struct bio {
 	void			*bi_private;
 #ifdef CONFIG_BLK_CGROUP
 	/*
-	 * Represents the association of the css and request_queue for the bio.
-	 * If a bio goes direct to device, it will not have a blkg as it will
-	 * not have a request_queue associated with it.  The reference is put
-	 * on release of the bio.
+	 * Represents the blkcg css association for the bio.  The reference is
+	 * put on release of the bio.
 	 */
-	struct blkcg_gq		*bi_blkg;
+	struct blkcg		*bi_blkcg;
 	/* Time that this bio was issued. */
 	u64			issue_time_ns;
 #ifdef CONFIG_BLK_CGROUP_IOCOST
@@ -309,6 +307,7 @@ enum {
 	BIO_TRACE_COMPLETION,	/* bio_endio() should trace the final completion
 				 * of this bio. */
 	BIO_CGROUP_ACCT,	/* has been accounted to a cgroup */
+	BIO_BLKG_REF,		/* bio pins the associated blkg */
 	BIO_QOS_THROTTLED,	/* bio went through rq_qos throttle path */
 	/*
 	 * This bio has completed bps throttling at the single tg granularity,
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 20cb8ed7d987a..f3b0e39d3f6b4 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -27,6 +27,7 @@
 #include <linux/xarray.h>
 #include <linux/file.h>
 #include <linux/lockdep.h>
+#include <linux/rhashtable-types.h>
 
 struct module;
 struct request_queue;
@@ -581,6 +582,7 @@ struct request_queue {
 #ifdef CONFIG_BLK_CGROUP
 	DECLARE_BITMAP		(blkcg_pols, BLKCG_MAX_POLS);
 	struct blkcg_gq		*root_blkg;
+	struct rhashtable	blkg_hash;
 	struct list_head	blkg_list;
 	struct mutex		blkcg_mutex;
 #endif
diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index 62552a2ce5b9e..4f869fe9cc90f 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -262,7 +262,7 @@ static inline void wbc_init_bio(struct writeback_control *wbc, struct bio *bio)
 	 * regular writeback instead of writing things out itself.
 	 */
 	if (wbc->wb)
-		bio_associate_blkg_from_css(bio, wbc->wb->blkcg_css);
+		bio_associate_blkcg_from_css(bio, wbc->wb->blkcg_css);
 }
 
 void inode_switch_wbs_work_fn(struct work_struct *work);
diff --git a/mm/page_io.c b/mm/page_io.c
index b23f494fcc83d..a4a1b8a8a0e38 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -316,7 +316,7 @@ static inline void count_swpout_vm_event(struct folio *folio)
 }
 
 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
-static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)
+static void bio_associate_blkcg_from_page(struct bio *bio, struct folio *folio)
 {
 	struct cgroup_subsys_state *css;
 	struct mem_cgroup *memcg;
@@ -331,12 +331,15 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)
 		css = NULL;
 	rcu_read_unlock();
 
-	bio_associate_blkg_from_css(bio, css);
+	bio_associate_blkcg_from_css(bio, css);
 	if (css)
 		css_put(css);
 }
 #else
-#define bio_associate_blkg_from_page(bio, folio)		do { } while (0)
+static inline void bio_associate_blkcg_from_page(struct bio *bio,
+						 struct folio *folio)
+{
+}
 #endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */
 
 struct swap_iocb {
@@ -436,7 +439,7 @@ static void swap_writepage_bdev_sync(struct folio *folio,
 	bio.bi_iter.bi_sector = swap_folio_sector(folio);
 	bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
 
-	bio_associate_blkg_from_page(&bio, folio);
+	bio_associate_blkcg_from_page(&bio, folio);
 	count_swpout_vm_event(folio);
 
 	folio_start_writeback(folio);
@@ -456,7 +459,7 @@ static void swap_writepage_bdev_async(struct folio *folio,
 	bio->bi_end_io = end_swap_bio_write;
 	bio_add_folio_nofail(bio, folio, folio_size(folio), 0);
 
-	bio_associate_blkg_from_page(bio, folio);
+	bio_associate_blkcg_from_page(bio, folio);
 	count_swpout_vm_event(folio);
 	folio_start_writeback(folio);
 	folio_unlock(folio);

gemini-3.1-pro-preview

							input: 21145
							output: 31
							thoughts: 519

**Analyzing a Block Cgroup Patch: Reachability and Configuration**

Okay, so I'm looking at this patch that fundamentally alters how `bio`s are linked to block cgroups – a significant change to the `blkcg` and `bio` association logic. Instead of the current `bio->bi_blkg` using queue-specific `blkcg_gq`, it's switching to `bio->bi_blkcg` holding the general `blkcg`. The `blkcg_gq` (blkg) is now either looked up dynamically or pinned via the new `BIO_BLKG_REF` flag. This is a big deal; this impacts the core block layer.

My initial thoughts are, let's nail down reachability and configurations. The code's spread across `block/blk-cgroup.c`, `block/bio.c`, `block/bfq-cgroup.c`, and likely others. Given it touches core block layer components, there's no question about reachability here.

Now, configuration is next. I see these configurations involved: `CONFIG_BLK_CGROUP`, `CONFIG_BFQ_GROUP_IOSCHED`, `CONFIG_BLK_CGROUP_PUNT_BIO`, `CONFIG_BLK_CGROUP_IOCOST`, and `CONFIG_BLK_CGROUP_FC_APPID`.

Next step, I need to zero in on the functions that have been modified or introduced. I see: `bio_blkg_lookup`, `bio_blkg`, `bio_associate_blkcg_from_css`, `bio_associate_blkcg`, `bio_clone_blkcg_association`, and `bio_put_blkg_ref`.

I'll start by checking `block/blk-cgroup.c` to see if those functions are defined there. I confirm, they are all defined there.

Alright, let's do a quick grep to make sure they're exported. Yep, they are using `EXPORT_SYMBOL_GPL`, which is as expected.

Finally, I need to check the Kconfig for `BLK_CGROUP`. Looks like it depends on both `CGROUPS` and `BLOCK`, which is standard. `BFQ_GROUP_IOSCHED` in turn depends on `IOSCHED_BFQ` and, as expected, `BLK_CGROUP`. What about `BLK_CGROUP_PUNT_BIO`? That needs closer examination to see if it's an internal config or if it has any additional dependencies.

{
  "Expression": "config BLK_CGROUP",
  "PathPrefix": "block/Kconfig"
}

{
  "Output": "block/Kconfig=39=config BLK_RQ_ALLOC_TIME\n--\nblock/Kconfig-41-\nblock/Kconfig:42:config BLK_CGROUP_RWSTAT\nblock/Kconfig-43-\tbool\nblock/Kconfig-44-\nblock/Kconfig:45:config BLK_CGROUP_PUNT_BIO\nblock/Kconfig-46-\tbool\n--\nblock/Kconfig=127=config BLK_WBT_MQ\n--\nblock/Kconfig-133-\nblock/Kconfig:134:config BLK_CGROUP_IOLATENCY\nblock/Kconfig-135-\tbool \"Enable support for latency based cgroup IO protection\"\n--\nblock/Kconfig-144-\nblock/Kconfig:145:config BLK_CGROUP_FC_APPID\nblock/Kconfig-146-\tbool \"Enable support to track FC I/O Traffic across cgroup applications\"\n--\nblock/Kconfig-153-\nblock/Kconfig:154:config BLK_CGROUP_IOCOST\nblock/Kconfig-155-\tbool \"Enable support for cost model based cgroup IO controller\"\n--\nblock/Kconfig-163-\nblock/Kconfig:164:config BLK_CGROUP_IOPRIO\nblock/Kconfig-165-\tbool \"Cgroup I/O controller for assigning an I/O priority class\"\n"
}

gemini-3.1-pro-preview

							input: 22609
							output: 27
							thoughts: 25

Here's my thought process, as if I were considering this myself:

**Investigating the BLK_CGROUP Configuration Option**

Okay, so I need to track down where the `BLK_CGROUP` configuration option is defined. My first instinct, given its relation to block devices and control groups, is to look in the kernel configuration files. The two most likely candidates are `init/Kconfig` and `block/Kconfig`. `init/Kconfig` often handles fundamental initialization options, and `block/Kconfig` obviously deals with block device-related configurations. I'll start by checking those two files – that should be a good starting point to locate the definition or dependency of `BLK_CGROUP`. It's a pretty standard approach. I expect to find it declared with its help text and default value. From there, I can trace any dependencies if needed. Let's dig in.

{
  "Expression": "config BLK_CGROUP",
  "PathPrefix": ""
}

{
  "Output": "block/Kconfig=39=config BLK_RQ_ALLOC_TIME\n--\nblock/Kconfig-41-\nblock/Kconfig:42:config BLK_CGROUP_RWSTAT\nblock/Kconfig-43-\tbool\nblock/Kconfig-44-\nblock/Kconfig:45:config BLK_CGROUP_PUNT_BIO\nblock/Kconfig-46-\tbool\n--\nblock/Kconfig=127=config BLK_WBT_MQ\n--\nblock/Kconfig-133-\nblock/Kconfig:134:config BLK_CGROUP_IOLATENCY\nblock/Kconfig-135-\tbool \"Enable support for latency based cgroup IO protection\"\n--\nblock/Kconfig-144-\nblock/Kconfig:145:config BLK_CGROUP_FC_APPID\nblock/Kconfig-146-\tbool \"Enable support to track FC I/O Traffic across cgroup applications\"\n--\nblock/Kconfig-153-\nblock/Kconfig:154:config BLK_CGROUP_IOCOST\nblock/Kconfig-155-\tbool \"Enable support for cost model based cgroup IO controller\"\n--\nblock/Kconfig-163-\nblock/Kconfig:164:config BLK_CGROUP_IOPRIO\nblock/Kconfig-165-\tbool \"Cgroup I/O controller for assigning an I/O priority class\"\n--\ninit/Kconfig=1099=config MEMCG_V1\n--\ninit/Kconfig-1115-\ninit/Kconfig:1116:config BLK_CGROUP\ninit/Kconfig-1117-\tbool \"IO controller\"\n"
}

gemini-3.1-pro-preview

							input: 23288
							output: 33
							thoughts: 18

Here's a summary of my thought process, as requested:

**Checking BLK_CGROUP Configuration**

Okay, so I need to quickly ascertain the status of `BLK_CGROUP` within the kernel configuration. My immediate instinct is to check `init/Kconfig`. That's the most logical place to start digging for definitions related to block layer control groups, especially since it's a core configuration file. Let's see if it's defined there, and if so, what the dependencies and default value are. This should give me a good initial overview of how `BLK_CGROUP` is handled within the build system and if it is enabled or disabled by default. Understanding the configuration is critical to moving forward.

{
  "File": "init/Kconfig",
  "FirstLine": 1110,
  "LineCount": 20
}

{
  "Contents": "1110:\t\t  Please note that feature set of the legacy memory controller is likely\n1111:\t\t  going to shrink due to deprecation process. New deployments with v1\n1112:\t\t  controller are highly discouraged.\n1113:\t\n1114:\t\t  Say N if unsure.\n1115:\t\n1116:\tconfig BLK_CGROUP\n1117:\t\tbool \"IO controller\"\n1118:\t\tdepends on BLOCK\n1119:\t\tdefault n\n1120:\t\thelp\n1121:\t\tGeneric block IO controller cgroup interface. This is the common\n1122:\t\tcgroup interface which should be used by various IO controlling\n1123:\t\tpolicies.\n1124:\t\n1125:\t\tCurrently, CFQ IO scheduler uses it to recognize task groups and\n1126:\t\tcontrol disk bandwidth allocation (proportional time slice allocation)\n1127:\t\tto such task groups. It is also used by bio throttling logic in\n1128:\t\tblock layer to implement upper limit in IO rates on a device.\n1129:\t\n"
}

gemini-3.1-pro-preview

							input: 23781
							output: 29
							thoughts: 42

Okay, here's my summary, taking on the persona you've described:

**Understanding the Block IO Scheduler Dependencies**

Alright, let's break this down. From what I'm seeing here, `BLK_CGROUP` is a core dependency for the block layer, that's not surprising. But more interestingly,  `BFQ_GROUP_IOSCHED` requires `IOSCHED_BFQ` and, critically, `BLK_CGROUP`. So, to get the BFQ group I/O scheduler up and running, I need both the base BFQ scheduler *and* the control group infrastructure. This confirms my understanding. I need to make sure I have the necessary configurations enabled in my kernel to use this correctly. These dependencies are crucial to managing and prioritizing block I/O at a granular, group-specific level. Okay, that's straightforward.

{
  "Expression": "config BFQ_GROUP_IOSCHED",
  "PathPrefix": ""
}

{
  "Output": "block/Kconfig.iosched=19=config IOSCHED_BFQ\n--\nblock/Kconfig.iosched-29-\nblock/Kconfig.iosched:30:config BFQ_GROUP_IOSCHED\nblock/Kconfig.iosched-31-       bool \"BFQ hierarchical scheduling support\"\n"
}

gemini-3.1-pro-preview

							input: 24125
							output: 34
							thoughts: 24

Here's a summary of my immediate thought process:

**Kconfig Dive for BFQ Group Scheduling**

Okay, time to get to the heart of this. I need to figure out if `BFQ_GROUP_IOSCHED` is enabled and how it's configured. That means I need to start digging into the kernel configuration. The logical place to start is the `block/Kconfig.iosched` file. It's the standard location for I/O scheduler-related configurations, and `BFQ` is an I/O scheduler, so that's where I'd expect to find this particular option. Let's get to it. Time to open that file and see what's what.

{
  "File": "block/Kconfig.iosched",
  "FirstLine": 25,
  "LineCount": 20
}

{
  "Contents": "  25:\t\tregardless of the device parameters and with any workload. It\n  26:\t\talso guarantees a low latency to interactive and soft\n  27:\t\treal-time applications.  Details in\n  28:\t\tDocumentation/block/bfq-iosched.rst\n  29:\t\n  30:\tconfig BFQ_GROUP_IOSCHED\n  31:\t       bool \"BFQ hierarchical scheduling support\"\n  32:\t       depends on IOSCHED_BFQ \u0026\u0026 BLK_CGROUP\n  33:\t       default y\n  34:\t       select BLK_CGROUP_RWSTAT\n  35:\t\thelp\n  36:\t\n  37:\t       Enable hierarchical scheduling in BFQ, using the blkio\n  38:\t       (cgroups-v1) or io (cgroups-v2) controller.\n  39:\t\n  40:\tconfig BFQ_CGROUP_DEBUG\n  41:\t\tbool \"BFQ IO controller debugging\"\n  42:\t\tdepends on BFQ_GROUP_IOSCHED\n  43:\t\thelp\n  44:\t\tEnable some debugging help. Currently it exports additional stat\n"
}

gemini-3.1-pro-preview

							input: 24625
							output: 26
							thoughts: 522

**Fuzzing Target Identification: Block Cgroup Changes**

Alright, let's break this down. This patch touches a whole slew of files within the `block/` directory: `blk-cgroup.c`, `bio.c`, `bfq-cgroup.c`, `blk-iocost.c`, `blk-iolatency.c`, `blk-ioprio.c`, and `blk-throttle.c`.  All block cgroup related, naturally.  The changes seem functional, which makes them prime fuzzing targets.

Specifically, I need to zero in on these functions: `bio_blkg_lookup`, `bio_blkg`, `bio_associate_blkcg_from_css`, `bio_associate_blkcg`, `bio_clone_blkcg_association`, and `bio_put_blkg_ref`. Now, the key is to determine which of these are likely to be in the hot path, and therefore more impactful to fuzz.

`bio_associate_blkcg` is called within both `bio_init` and `bio_reset` *if* a `bdev` is set.  `bio_init` is definitely hot-path. Which means `bio_associate_blkcg`, which calls `bio_associate_blkcg_from_css`, is hot path. I think these should be the first ones I focus on.

`bio_blkg` is also interesting because it's called within `bfq_bio_merge`, `bfq_insert_request`, `ioc_rqos_throttle`, `blkcg_iolatency_throttle`, and `__blk_throtl_bio`, which are all hot paths since they are called with every bio. While `bio_blkg` is a new function, it is being called in hot-path, so it's a good place to concentrate on.

I also need to consider: `blkg_lookup_create`, `blkg_conf_prep`, `blkcg_css_alloc`, `blkg_init_queue`, and `blkcg_init_disk`. However, a quick scan suggests these are not as frequently accessed: `blkg_conf_prep` is involved in configuring blkcg policies (not hot-path), `blkg_init_queue` is used during queue allocation (also not hot-path), and `blkcg_init_disk` during disk initialization (again, not hot-path).  `bio_associate_blkcg_from_css` also gets called in `bio_clone_blkcg_association`.

Therefore, I'll prioritize fuzzing `bio_blkg` and `bio_associate_blkcg`, `bio_associate_blkcg_from_css` and `bio_clone_blkcg_association` and dig deeper into those. Let's see how they work.

{
  "Expression": "bio_blkg",
  "PathPrefix": "block/"
}

{
  "Output": "block/bfq-cgroup.c=364=void bfqg_stats_update_legacy_io(struct request_queue *q, struct request *rq)\nblock/bfq-cgroup.c-365-{\nblock/bfq-cgroup.c:366:\tstruct blkcg_gq *blkg = bio_blkg_lookup(rq-\u003ebio);\nblock/bfq-cgroup.c-367-\tstruct bfq_group *bfqg;\n--\nblock/bfq-cgroup.c=609=struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio)\nblock/bfq-cgroup.c-610-{\nblock/bfq-cgroup.c:611:\tstruct blkcg_gq *blkg = bio_blkg_lookup(bio);\nblock/bfq-cgroup.c-612-\tstruct bfq_group *bfqg;\n--\nblock/bfq-iosched.c=2448=static bool bfq_bio_merge(struct request_queue *q, struct bio *bio,\n--\nblock/bfq-iosched.c-2458-\tif (bic)\nblock/bfq-iosched.c:2459:\t\tbio_blkg(bio);\nblock/bfq-iosched.c-2460-#endif\n--\nblock/bfq-iosched.c=6239=static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,\n--\nblock/bfq-iosched.c-6254-\tif (rq-\u003ebio \u0026\u0026 !bio_flagged(rq-\u003ebio, BIO_BLKG_REF))\nblock/bfq-iosched.c:6255:\t\tbio_blkg(rq-\u003ebio);\nblock/bfq-iosched.c-6256-\n--\nblock/blk-cgroup.c=2096=static void bio_set_blkg_ref(struct bio *bio, struct blkcg_gq *blkg)\n--\nblock/blk-cgroup.c-2106-/**\nblock/blk-cgroup.c:2107: * bio_blkg_lookup - look up a blkg associated with a bio\nblock/blk-cgroup.c-2108- * @bio: target bio\n--\nblock/blk-cgroup.c-2110- * Return the blkg already pinned by @bio, or %NULL if @bio doesn't own a blkg\nblock/blk-cgroup.c:2111: * reference.  Call bio_blkg() instead when a missing blkg should be created.\nblock/blk-cgroup.c-2112- */\nblock/blk-cgroup.c:2113:struct blkcg_gq *bio_blkg_lookup(struct bio *bio)\nblock/blk-cgroup.c-2114-{\n--\nblock/blk-cgroup.c-2118-}\nblock/blk-cgroup.c:2119:EXPORT_SYMBOL_GPL(bio_blkg_lookup);\nblock/blk-cgroup.c-2120-\n--\nblock/blk-cgroup.c-2124- *\nblock/blk-cgroup.c:2125: * Drop the bio-owned blkg reference acquired by bio_blkg(), if any.\nblock/blk-cgroup.c-2126- */\n--\nblock/blk-cgroup.c=2137=EXPORT_SYMBOL_GPL(bio_put_blkg_ref);\n--\nblock/blk-cgroup.c-2139-/**\nblock/blk-cgroup.c:2140: * bio_blkg - look up the blkg associated with a bio\nblock/blk-cgroup.c-2141- * @bio: target bio\n--\nblock/blk-cgroup.c-2147- */\nblock/blk-cgroup.c:2148:struct blkcg_gq *bio_blkg(struct bio *bio)\nblock/blk-cgroup.c-2149-{\n--\nblock/blk-cgroup.c-2170-}\nblock/blk-cgroup.c:2171:EXPORT_SYMBOL_GPL(bio_blkg);\nblock/blk-cgroup.c-2172-\n--\nblock/blk-cgroup.c-2178- * Associate @bio with the blkcg found from @css.  The queue-local blkg is\nblock/blk-cgroup.c:2179: * created and pinned by bio_blkg() when blkcg policies need it.  If @css\nblock/blk-cgroup.c-2180- * can't be referenced online, associate @bio with the root blkcg instead.\n--\nblock/blk-cgroup.c=2248=void blk_cgroup_bio_start(struct bio *bio)\n--\nblock/blk-cgroup.c-2264-\nblock/blk-cgroup.c:2265:\tblkg = bio_blkg_lookup(bio);\nblock/blk-cgroup.c-2266-\tif (!blkg)\n--\nblock/blk-cgroup.h=289=static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg,\n--\nblock/blk-cgroup.h-302-\nblock/blk-cgroup.h:303:struct blkcg_gq *bio_blkg_lookup(struct bio *bio);\nblock/blk-cgroup.h:304:struct blkcg_gq *bio_blkg(struct bio *bio);\nblock/blk-cgroup.h-305-\n--\nblock/blk-cgroup.h=540=static inline struct blkcg *bio_blkcg(struct bio *bio) { return NULL; }\nblock/blk-cgroup.h:541:static inline struct blkcg_gq *bio_blkg_lookup(struct bio *bio) { return NULL; }\nblock/blk-cgroup.h:542:static inline struct blkcg_gq *bio_blkg(struct bio *bio) { return NULL; }\nblock/blk-cgroup.h-543-static inline struct blkcg_gq *blkg_lookup_any(struct blkcg *blkcg, void *key) { return NULL; }\n--\nblock/blk-iocost.c=2687=static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)\nblock/blk-iocost.c-2688-{\nblock/blk-iocost.c:2689:\tstruct blkcg_gq *blkg = bio_blkg(bio);\nblock/blk-iocost.c-2690-\tstruct ioc *ioc = rqos_to_ioc(rqos);\n--\nblock/blk-iocost.c=2775=static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq,\n--\nblock/blk-iocost.c-2777-{\nblock/blk-iocost.c:2778:\tstruct ioc_gq *iocg = blkg_to_iocg(bio_blkg(bio));\nblock/blk-iocost.c-2779-\tstruct ioc *ioc = rqos_to_ioc(rqos);\n--\nblock/blk-iocost.c=2834=static void ioc_rqos_done_bio(struct rq_qos *rqos, struct bio *bio)\n--\nblock/blk-iocost.c-2840-\nblock/blk-iocost.c:2841:\tiocg = blkg_to_iocg(bio_blkg_lookup(bio));\nblock/blk-iocost.c-2842-\tif (iocg)\n--\nblock/blk-iolatency.c=463=static void blkcg_iolatency_throttle(struct rq_qos *rqos, struct bio *bio)\n--\nblock/blk-iolatency.c-465-\tstruct blk_iolatency *blkiolat = BLKIOLATENCY(rqos);\nblock/blk-iolatency.c:466:\tstruct blkcg_gq *blkg = bio_blkg(bio);\nblock/blk-iolatency.c-467-\tbool issue_as_root = bio_issue_as_root_blkg(bio);\n--\nblock/blk-iolatency.c=583=static void blkcg_iolatency_done_bio(struct rq_qos *rqos, struct bio *bio)\n--\nblock/blk-iolatency.c-595-\nblock/blk-iolatency.c:596:\tblkg = bio_blkg_lookup(bio);\nblock/blk-iolatency.c-597-\tif (!blkg)\n--\nblock/blk-throttle.c=1792=bool __blk_throtl_bio(struct bio *bio)\n--\nblock/blk-throttle.c-1794-\tstruct request_queue *q = bdev_get_queue(bio-\u003ebi_bdev);\nblock/blk-throttle.c:1795:\tstruct blkcg_gq *blkg = bio_blkg(bio);\nblock/blk-throttle.c-1796-\tstruct throtl_qnode *qn = NULL;\n--\nblock/blk-throttle.h=166=static inline bool blk_should_throtl(struct bio *bio)\n--\nblock/blk-throttle.h-173-\nblock/blk-throttle.h:174:\ttg = blkg_to_tg(bio_blkg(bio));\nblock/blk-throttle.h-175-\tif (!cgroup_subsys_on_dfl(io_cgrp_subsys)) {\n"
}

gemini-3.1-pro-preview

							input: 28262
							output: 28
							thoughts: 449

**Analyzing Block I/O Cgroup Function Usage**

Okay, so I'm looking at these exported functions related to block I/O cgroups. I see `bio_blkg`, `bio_blkg_lookup`, `bio_associate_blkcg`, `bio_associate_blkcg_from_css`, `bio_clone_blkcg_association`, and `bio_put_blkg_ref` are all exposed. That's the baseline. Now, I want to see which of these are *not* in a hot path, since those are the ones that would be more interesting to potentially optimize or refactor.

I've identified a few that seem less performance-critical: `blkg_init_queue` during queue allocation, `blkcg_init_disk` during disk initialization, `blkg_conf_prep` when configuring policies, `blkcg_css_alloc`, `blkcg_css_online`, `blkcg_css_free` relating to cgroup creation and destruction, `blkg_destroy`, `blkg_release` which handles refcounts, and `blkcg_punt_bio_submit` for asynchronous bio submission. This list seems like a decent starting point.

But wait, I need to double-check my assumptions. `bio_associate_blkcg_from_css` is on the list, but it's called by `bio_associate_blkcg`, and *that* is called by `bio_init` which *is* hot-path! Same with `bio_clone_blkcg_association`, which is called by `__bio_clone`. So, those are actually not good candidates. Hmmm, now I need to check `blkcg_punt_bio_submit`. I think that's invoked from `bio_endio` or something similar, and that would make that a hot-path. Gotta confirm that before I dig deeper.

{
  "Expression": "blkcg_punt_bio_submit",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=225=static void blkcg_async_bio_workfn(struct work_struct *work)\n--\nblock/blk-cgroup.c-254- */\nblock/blk-cgroup.c:255:void blkcg_punt_bio_submit(struct bio *bio)\nblock/blk-cgroup.c-256-{\n--\nblock/blk-cgroup.c-268-}\nblock/blk-cgroup.c:269:EXPORT_SYMBOL_GPL(blkcg_punt_bio_submit);\nblock/blk-cgroup.c-270-\n--\nfs/btrfs/bio.c=495=static void btrfs_submit_dev_bio(struct btrfs_device *dev, struct bio *bio)\n--\nfs/btrfs/bio.c-535-\tif (bio-\u003ebi_opf \u0026 REQ_BTRFS_CGROUP_PUNT)\nfs/btrfs/bio.c:536:\t\tblkcg_punt_bio_submit(bio);\nfs/btrfs/bio.c-537-\telse\n--\nfs/btrfs/bio.h=122=void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status);\nfs/btrfs/bio.h-123-\nfs/btrfs/bio.h:124:/* Submit using blkcg_punt_bio_submit. */\nfs/btrfs/bio.h-125-#define REQ_BTRFS_CGROUP_PUNT\t\t\tREQ_FS_PRIVATE\n--\ninclude/linux/bio.h=549=void bio_put_blkg_ref(struct bio *bio);\ninclude/linux/bio.h:550:void blkcg_punt_bio_submit(struct bio *bio);\ninclude/linux/bio.h-551-#else\t/* CONFIG_BLK_CGROUP */\n--\ninclude/linux/bio.h=558=static inline void bio_put_blkg_ref(struct bio *bio) { }\ninclude/linux/bio.h:559:static inline void blkcg_punt_bio_submit(struct bio *bio)\ninclude/linux/bio.h-560-{\n"
}

gemini-3.1-pro-preview

							input: 29725
							output: 26
							thoughts: 50

Here's my summary, as though I'm thinking through this:

**Analyzing Btrfs IO Submission**

Okay, so I'm looking at `blkcg_punt_bio_submit` and immediately, my brain flags it as something Btrfs-specific. It's not a general-purpose, high-frequency core function, but it *is* used for IO submission, which is critical. Given its role in IO submission, I need to understand its relationship to block I/O controller configuration, and where the resource management fits in. The next thing I need to investigate is `blkg_conf_prep`. I need to figure out how I need to configure the IO submission in this function.

{
  "Expression": "blkg_conf_prep",
  "PathPrefix": ""
}

{
  "Output": "block/bfq-cgroup.c=1063=static ssize_t bfq_io_set_device_weight(struct kernfs_open_file *of,\n--\nblock/bfq-cgroup.c-1078-\nblock/bfq-cgroup.c:1079:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_bfq, \u0026ctx);\nblock/bfq-cgroup.c-1080-\tif (ret)\n--\nblock/blk-cgroup.c=770=EXPORT_SYMBOL_GPL(__blkg_prfill_u64);\n--\nblock/blk-cgroup.c-778- * Once initialized, @ctx can be used with blkg_conf_open_bdev() and\nblock/blk-cgroup.c:779: * blkg_conf_prep().\nblock/blk-cgroup.c-780- */\n--\nblock/blk-cgroup.c=834=EXPORT_SYMBOL_GPL(blkg_conf_open_bdev);\n--\nblock/blk-cgroup.c-836-/**\nblock/blk-cgroup.c:837: * blkg_conf_prep - parse and prepare for per-blkg config update\nblock/blk-cgroup.c-838- * @blkcg: target block cgroup\n--\nblock/blk-cgroup.c-850- */\nblock/blk-cgroup.c:851:int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,\nblock/blk-cgroup.c-852-\t\t   struct blkg_conf_ctx *ctx)\n--\nblock/blk-cgroup.c-944-}\nblock/blk-cgroup.c:945:EXPORT_SYMBOL_GPL(blkg_conf_prep);\nblock/blk-cgroup.c-946-\nblock/blk-cgroup.c-947-/**\nblock/blk-cgroup.c:948: * blkg_conf_unprep - counterpart of blkg_conf_prep()\nblock/blk-cgroup.c-949- * @ctx: blkg_conf_ctx initialized with blkg_conf_init()\n--\nblock/blk-cgroup.h=230=int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx)\nblock/blk-cgroup.h-231-\t__cond_acquires(0, \u0026ctx-\u003ebdev-\u003ebd_queue-\u003erq_qos_mutex);\nblock/blk-cgroup.h:232:int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,\nblock/blk-cgroup.h-233-\t\t   struct blkg_conf_ctx *ctx)\n--\nblock/blk-iocost.c=3149=static ssize_t ioc_weight_write(struct kernfs_open_file *of, char *buf,\n--\nblock/blk-iocost.c-3191-\nblock/blk-iocost.c:3192:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_iocost, \u0026ctx);\nblock/blk-iocost.c-3193-\tif (ret)\n--\nblock/blk-iolatency.c=830=static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf,\n--\nblock/blk-iolatency.c-857-\nblock/blk-iolatency.c:858:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_iolatency, \u0026ctx);\nblock/blk-iolatency.c-859-\tif (ret)\n--\nblock/blk-throttle.c=1352=static ssize_t tg_set_conf(struct kernfs_open_file *of,\n--\nblock/blk-throttle.c-1372-\nblock/blk-throttle.c:1373:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_throtl, \u0026ctx);\nblock/blk-throttle.c-1374-\tif (ret)\n--\nblock/blk-throttle.c=1540=static ssize_t tg_set_limit(struct kernfs_open_file *of,\n--\nblock/blk-throttle.c-1560-\nblock/blk-throttle.c:1561:\tret = blkg_conf_prep(blkcg, \u0026blkcg_policy_throtl, \u0026ctx);\nblock/blk-throttle.c-1562-\tif (ret)\n"
}

gemini-3.1-pro-preview

							input: 31119
							output: 26
							thoughts: 194

**Analyzing Block I/O Controller (blkcg) Initialization and Lifecycle**

Okay, so I'm thinking about the critical points for integrating new blkcg-related functionality. Let's break down where the key initialization, allocation, and destruction routines are called. `blkg_conf_prep` appears to be the go-to function for preparing things when configuring block I/O controller policies through sysfs or cgroupfs; this seems like a solid entry point for potential instrumentation or modifications. Similarly, `blkg_init_queue`, called from the core `blk_alloc_queue` function, will be important to observe how queue structures are initialized. `blkcg_init_disk`, initiated by `add_disk`, should be an equally important point of focus.

Now, I'm zeroing in on the blkcg lifecycle itself. `blkcg_css_alloc` handles the creation of a new blkcg, so this is where resource initialization happens. Then, `blkcg_css_online` is triggered when a blkcg comes online. Finally, `blkcg_css_free` is the critical point for freeing a blkcg. We have to consider how these calls interact and make sure all resources are properly managed.

Let's also not forget about `blkg_destroy`, which deals with the destruction of a blkg (block group?), and `blkg_release`, which handles refcount dropping to 0 for a blkg. These are all equally important to track.

For now, I'm going to focus my attention specifically on `blkcg_css_alloc`, `blkcg_css_online`, and `blkcg_css_free`. These are going to be the most critical for understanding the lifecycle. I need to understand what's happening during these allocation/online/free phases.

{
  "Expression": "blkcg_css_alloc",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=1390=static struct cgroup_subsys_state *\nblock/blk-cgroup.c:1391:blkcg_css_alloc(struct cgroup_subsys_state *parent_css)\nblock/blk-cgroup.c-1392-{\n--\nblock/blk-cgroup.c=1536=struct cgroup_subsys io_cgrp_subsys = {\nblock/blk-cgroup.c:1537:\t.css_alloc = blkcg_css_alloc,\nblock/blk-cgroup.c-1538-\t.css_online = blkcg_css_online,\n"
}

gemini-3.1-pro-preview

							input: 31903
							output: 28
							thoughts: 232

**Analyzing Cgroup Subsystem Callbacks**

Okay, so I'm looking at these cgroup subsystem callbacks. I'm pretty sure they are definitely reachable but not in the hot-path. My first instinct is to check out `bio_blkg` and `bio_blkg_lookup`. Hmm, but `bio_blkg` is called from `bfq_bio_merge`, `bfq_insert_request`, `ioc_rqos_throttle`, `blkcg_iolatency_throttle`, and `__blk_throtl_bio`, which are all hot-path functions. Wait, though, this `bio_blkg` is new, replacing the direct `bio->bi_blkg` access, which is interesting. Maybe there's a non-hot-path function we should focus on? Let me see. I've got `blkg_conf_prep`, `blkcg_css_alloc`, `blkcg_css_free`, `blkg_init_queue`, `blkcg_init_disk`, `bio_associate_blkcg_from_css`, `bio_clone_blkcg_association`, and `bio_put_blkg_ref`. Right, I'll go with `bio_put_blkg_ref` for now.

{
  "Expression": "bio_put_blkg_ref",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=2119=EXPORT_SYMBOL_GPL(bio_blkg_lookup);\n--\nblock/blk-cgroup.c-2121-/**\nblock/blk-cgroup.c:2122: * bio_put_blkg_ref - drop the blkg reference pinned by a bio\nblock/blk-cgroup.c-2123- * @bio: target bio\n--\nblock/blk-cgroup.c-2126- */\nblock/blk-cgroup.c:2127:void bio_put_blkg_ref(struct bio *bio)\nblock/blk-cgroup.c-2128-{\n--\nblock/blk-cgroup.c-2136-}\nblock/blk-cgroup.c:2137:EXPORT_SYMBOL_GPL(bio_put_blkg_ref);\nblock/blk-cgroup.c-2138-\n--\nblock/blk-cgroup.h=373=static inline void bio_clear_blkcg(struct bio *bio)\n--\nblock/blk-cgroup.h-376-\nblock/blk-cgroup.h:377:\tbio_put_blkg_ref(bio);\nblock/blk-cgroup.h-378-\n--\ninclude/linux/bio.h=548=void bio_clone_blkcg_association(struct bio *dst, struct bio *src);\ninclude/linux/bio.h:549:void bio_put_blkg_ref(struct bio *bio);\ninclude/linux/bio.h-550-void blkcg_punt_bio_submit(struct bio *bio);\n--\ninclude/linux/bio.h=556=static inline void bio_clone_blkcg_association(struct bio *dst,\ninclude/linux/bio.h-557-\t\t\t\t\t       struct bio *src) { }\ninclude/linux/bio.h:558:static inline void bio_put_blkg_ref(struct bio *bio) { }\ninclude/linux/bio.h-559-static inline void blkcg_punt_bio_submit(struct bio *bio)\n--\ninclude/linux/bio.h=565=static inline void bio_set_dev(struct bio *bio, struct block_device *bdev)\n--\ninclude/linux/bio.h-568-\tif (bio-\u003ebi_bdev != bdev) {\ninclude/linux/bio.h:569:\t\tbio_put_blkg_ref(bio);\ninclude/linux/bio.h-570-\t\tbio_clear_flag(bio, BIO_BPS_THROTTLED);\n"
}

gemini-3.1-pro-preview

							input: 33098
							output: 30
							thoughts: 76

Here's my summary, as I see it:

**Thinking Through Block I/O Context Management**

Okay, so I'm looking at how block I/O context is being managed. `bio_put_blkg_ref` is called from two key places: `bio_clear_blkcg` and `bio_set_dev`.  Both of these are important, but `bio_clear_blkcg` gets called from `bio_uninit` and `bio_endio`! Hmm, now *that* definitely raises a flag for performance sensitivity. Those are *very* hot paths.

I'm starting to think about where the performance is.

The real question I should be asking is: what about `bio_associate_blkcg_from_css`? Where does that fit into the picture? What's the relative frequency of that call versus the ones I've just highlighted? Is there something in the association and deassociation of the `blkcg` that is creating a performance bottleneck? I'll need to dig into the call graphs and analyze the code around `bio_associate_blkcg_from_css` carefully. That's the likely weak spot.

{
  "Expression": "bio_associate_blkcg_from_css",
  "PathPrefix": ""
}

{
  "Output": "block/bfq-cgroup.c=609=struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio)\n--\nblock/bfq-cgroup.c-621-\t\tif (bfqg-\u003epd.online) {\nblock/bfq-cgroup.c:622:\t\t\tbio_associate_blkcg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\nblock/bfq-cgroup.c-623-\t\t\treturn bfqg;\n--\nblock/bfq-cgroup.c-628-\tblkg = bfqg_to_blkg(bfqd-\u003eroot_group);\nblock/bfq-cgroup.c:629:\tbio_associate_blkcg_from_css(bio, \u0026blkg-\u003eblkcg-\u003ecss);\nblock/bfq-cgroup.c-630-\treturn bfqd-\u003eroot_group;\n--\nblock/blk-cgroup.c=2171=EXPORT_SYMBOL_GPL(bio_blkg);\n--\nblock/blk-cgroup.c-2173-/**\nblock/blk-cgroup.c:2174: * bio_associate_blkcg_from_css - associate a bio with a specified css\nblock/blk-cgroup.c-2175- * @bio: target bio\n--\nblock/blk-cgroup.c-2184- */\nblock/blk-cgroup.c:2185:void bio_associate_blkcg_from_css(struct bio *bio,\nblock/blk-cgroup.c-2186-\t\t\t\t  struct cgroup_subsys_state *css)\n--\nblock/blk-cgroup.c-2203-}\nblock/blk-cgroup.c:2204:EXPORT_SYMBOL_GPL(bio_associate_blkcg_from_css);\nblock/blk-cgroup.c-2205-\n--\nblock/blk-cgroup.c=2213=void bio_associate_blkcg(struct bio *bio)\n--\nblock/blk-cgroup.c-2221-\trcu_read_lock();\nblock/blk-cgroup.c:2222:\tbio_associate_blkcg_from_css(bio, blkcg_css());\nblock/blk-cgroup.c-2223-\trcu_read_unlock();\n--\nblock/blk-cgroup.c=2232=void bio_clone_blkcg_association(struct bio *dst, struct bio *src)\n--\nblock/blk-cgroup.c-2234-\tif (bio_blkcg(src))\nblock/blk-cgroup.c:2235:\t\tbio_associate_blkcg_from_css(dst, bio_blkcg_css(src));\nblock/blk-cgroup.c-2236-}\n--\ninclude/linux/bio.h=545=void bio_associate_blkcg(struct bio *bio);\ninclude/linux/bio.h:546:void bio_associate_blkcg_from_css(struct bio *bio,\ninclude/linux/bio.h-547-\t\t\t\t  struct cgroup_subsys_state *css);\n--\ninclude/linux/bio.h=552=static inline void bio_associate_blkcg(struct bio *bio) { }\ninclude/linux/bio.h:553:static inline void bio_associate_blkcg_from_css(struct bio *bio,\ninclude/linux/bio.h-554-\t\t\t\t\t\tstruct cgroup_subsys_state *css)\n--\ninclude/linux/writeback.h=256=static inline void wbc_init_bio(struct writeback_control *wbc, struct bio *bio)\n--\ninclude/linux/writeback.h-264-\tif (wbc-\u003ewb)\ninclude/linux/writeback.h:265:\t\tbio_associate_blkcg_from_css(bio, wbc-\u003ewb-\u003eblkcg_css);\ninclude/linux/writeback.h-266-}\n--\nmm/page_io.c=319=static void bio_associate_blkcg_from_page(struct bio *bio, struct folio *folio)\n--\nmm/page_io.c-333-\nmm/page_io.c:334:\tbio_associate_blkcg_from_css(bio, css);\nmm/page_io.c-335-\tif (css)\n"
}

gemini-3.1-pro-preview

							input: 34564
							output: 26
							thoughts: 265

**Analyzing Block I/O Cgroup Hotspots**

Okay, so I'm digging into the block I/O cgroup code and need to identify potential performance bottlenecks. `bio_associate_blkcg_from_css` is clearly in the hot path, given its calls from `bio_associate_blkcg`, `bio_clone_blkcg_association`, `wbc_init_bio`, `bio_associate_blkcg_from_page`, and `bfq_bio_bfqg`. That's a lot of important places. I should definitely keep an eye on that.

Now, what about the other functions? Let's see... `blkg_conf_prep` is called from sysfs and cgroupfs writes. That could be interesting, perhaps something we could optimize via `FocusSymbols`. `blkg_init_queue` is part of `blk_alloc_queue`, which could also be a performance critical piece, but might not be as hot as the I/O path. `blkcg_init_disk` is triggered by `add_disk`, and `blkcg_css_alloc`, `blkcg_css_online`, and `blkcg_css_free` are involved in the lifecycle of block I/O cgroups, so that's something to keep in mind, and the `blkg_destroy` and `blkg_release` functions will be important when cleaning up.

For now, `blkg_conf_prep` looks like a prime candidate for further investigation. Sysfs/cgroupfs interaction is often a point of contention. I should also take a look at `blkg_lookup_create`.

{
  "Expression": "blkg_lookup_create",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=474=static struct blkcg_gq *blkg_lookup_tryget(struct blkcg_gq *blkg)\n--\nblock/blk-cgroup.c-481-/**\nblock/blk-cgroup.c:482: * blkg_lookup_create - lookup blkg, try to create one if not there\nblock/blk-cgroup.c-483- * @blkcg: blkcg of interest\n--\nblock/blk-cgroup.c-495- */\nblock/blk-cgroup.c:496:static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,\nblock/blk-cgroup.c-497-\t\tstruct gendisk *disk)\n--\nblock/blk-cgroup.c=2148=struct blkcg_gq *bio_blkg(struct bio *bio)\n--\nblock/blk-cgroup.c-2164-\tspin_lock_irq(\u0026q-\u003equeue_lock);\nblock/blk-cgroup.c:2165:\tblkg = blkg_lookup_create(blkcg, disk);\nblock/blk-cgroup.c-2166-\tspin_unlock_irq(\u0026q-\u003equeue_lock);\n"
}

gemini-3.1-pro-preview

							input: 35538
							output: 26
							thoughts: 263

**Analyzing the Block Group Lookup**

Okay, so `blkg_lookup_create` is called from `bio_blkg`. And I know `bio_blkg` is in the hot-path, which immediately flags it as something to pay attention to. Wait a minute, I have to remember, `bio_blkg` handles the *creation* of a blkg for a bio, so `bio_blkg` is called for every bio that *needs* a blkg. I have to make sure I am thinking this through correctly because it's a very fast execution path.

So, `blkg_lookup_create` is only called if `bio_flagged(bio, BIO_BLKG_REF)` is false. Ah, right! That means it's only called the *first* time a bio needs a blkg. The subsequent calls for that bio shouldn't be calling this function. Okay, now I have it. `bio_blkg` is called when `bio_flagged(bio, BIO_BLKG_REF)` is false, meaning it is called for the first IO of a bio that needs a blkg.

But, is `bio_blkg` hot-path? I have to confirm. Let me see... yes, it's called in several key places: `bfq_bio_merge`, `bfq_insert_request`, `ioc_rqos_throttle`, `blkcg_iolatency_throttle`, and `__blk_throtl_bio`, *if* the bio doesn't already have `BIO_BLKG_REF` set. That reinforces that is definitely in the hot-path!

Wait, `bio_blkg` sets `BIO_BLKG_REF` and pins the blkg, so that's why it is only called *once per bio*. That is a crucial optimization. Okay, it's still hot-path, but at least the number of lookups is reduced.

Now, I think I need to look at `blkg_init_queue` and `blkcg_init_disk` to understand the initialization side better.

{
  "Expression": "blkg_init_queue",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=1460=static int blkcg_css_online(struct cgroup_subsys_state *css)\n--\nblock/blk-cgroup.c-1473-\nblock/blk-cgroup.c:1474:int blkg_init_queue(struct request_queue *q)\nblock/blk-cgroup.c-1475-{\n--\nblock/blk-cgroup.h=200=extern const struct rhashtable_params blkg_hash_params;\nblock/blk-cgroup.h-201-\nblock/blk-cgroup.h:202:int blkg_init_queue(struct request_queue *q);\nblock/blk-cgroup.h-203-void blkg_exit_queue(struct request_queue *q);\n--\nblock/blk-cgroup.h=544=static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, void *key) { return NULL; }\nblock/blk-cgroup.h:545:static inline int blkg_init_queue(struct request_queue *q) { return 0; }\nblock/blk-cgroup.h-546-static inline void blkg_exit_queue(struct request_queue *q) { }\n--\nblock/blk-core.c=437=struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)\n--\nblock/blk-core.c-484-\nblock/blk-core.c:485:\terror = blkg_init_queue(q);\nblock/blk-core.c-486-\tif (error)\n"
}

gemini-3.1-pro-preview

							input: 36708
							output: 26
							thoughts: 38

Here's my understanding, as if these were my internal thoughts:

**Analyzing Block Device Initialization**

Okay, so I'm looking at the block device initialization flow. It seems pretty clear that the `blkg_init_queue` function is called directly from `blk_alloc_queue`. That makes sense; we're initializing the queue structure, likely related to I/O accounting at the group level, right as the queue is being allocated. I'd need to dig deeper into the code of `blk_alloc_queue` to understand the conditions triggering the call.

Now, the connection between `blkcg_init_disk` and `add_disk` is the more interesting piece. The comments state that `blkcg_init_disk` gets called from `add_disk`. This strongly suggests that we're dealing with initialization related to block cgroups whenever a new disk is *added* to the system. This seems correct. It likely sets up the cgroup integration points within the disk structure itself, so that subsequent I/O on that disk can be properly attributed to the right control groups. It’s also entirely possible that disk addition is what triggers the initialization of the disk-specific control group data, which would make sense. I'll need to confirm the conditions for the call.

{
  "Expression": "blkcg_init_disk",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=1481=void blkg_exit_queue(struct request_queue *q)\n--\nblock/blk-cgroup.c-1485-\nblock/blk-cgroup.c:1486:int blkcg_init_disk(struct gendisk *disk)\nblock/blk-cgroup.c-1487-{\n--\nblock/blk-cgroup.h=203=void blkg_exit_queue(struct request_queue *q);\nblock/blk-cgroup.h:204:int blkcg_init_disk(struct gendisk *disk);\nblock/blk-cgroup.h-205-void blkcg_exit_disk(struct gendisk *disk);\n--\nblock/blk-cgroup.h=546=static inline void blkg_exit_queue(struct request_queue *q) { }\nblock/blk-cgroup.h:547:static inline int blkcg_init_disk(struct gendisk *disk) { return 0; }\nblock/blk-cgroup.h-548-static inline void blkcg_exit_disk(struct gendisk *disk) { }\n--\nblock/genhd.c=1447=struct gendisk *__alloc_disk_node(struct request_queue *q, int node_id,\n--\nblock/genhd.c-1475-\nblock/genhd.c:1476:\tif (blkcg_init_disk(disk))\nblock/genhd.c-1477-\t\tgoto out_erase_part0;\n"
}

gemini-3.1-pro-preview

							input: 37442
							output: 24
							thoughts: 260

**Analyzing Block IO CGroups Initialization and Destruction**

Okay, so I'm looking at these block I/O cgroup initialization and destruction paths. `blkcg_init_disk` gets called from `__alloc_disk_node`, which is definitely not in a performance-critical area, so that's good.

Now, let's see, `blkg_conf_prep` is the next one to investigate, which is called from sysfs and cgroupfs writes. Then we have `blkcg_css_alloc`, that's happening whenever a new blkcg is created, `blkcg_css_online` onlining and `blkcg_css_free` when freeing. Finally, `blkg_destroy` is called upon destroy and `blkg_release` when the refcount drops to zero.

Based on that analysis, the non-hot-path functions that warrant further examination are: `blkg_conf_prep`, `blkcg_css_alloc`, `blkcg_css_online`, `blkcg_css_free`, `blkg_init_queue`, `blkcg_init_disk`, `blkg_destroy`, and `blkg_release`.

Hang on a second… weren't `blkg_destroy` and `blkg_release` static? I should probably double-check that. Static means, potentially, they are internal to a particular module, and we might need to think about that implication. I'll make sure to verify that.

{
  "Expression": "blkg_destroy",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=124=static void blkg_free_workfn(struct work_struct *work)\n--\nblock/blk-cgroup.c-133-\t * in order to make sure pd_free_fn() is called in order, the deletion\nblock/blk-cgroup.c:134:\t * of the list blkg-\u003eq_node is delayed to here from blkg_destroy(), and\nblock/blk-cgroup.c-135-\t * blkcg_mutex is used to synchronize blkg_free_workfn() and\n--\nblock/blk-cgroup.c=496=static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,\n--\nblock/blk-cgroup.c-545-\nblock/blk-cgroup.c:546:static void blkg_destroy(struct blkcg_gq *blkg)\nblock/blk-cgroup.c-547-{\n--\nblock/blk-cgroup.c-556-\t * blkg_free_workfn(), hence this function can be called from\nblock/blk-cgroup.c:557:\t * blkcg_destroy_blkgs() first and again from blkg_destroy_all() before\nblock/blk-cgroup.c-558-\t * blkg_free_workfn().\n--\nblock/blk-cgroup.c-583-\nblock/blk-cgroup.c:584:static void blkg_destroy_all(struct gendisk *disk)\nblock/blk-cgroup.c-585-{\n--\nblock/blk-cgroup.c-600-\t\tspin_lock(\u0026blkcg-\u003elock);\nblock/blk-cgroup.c:601:\t\tblkg_destroy(blkg);\nblock/blk-cgroup.c-602-\t\tspin_unlock(\u0026blkcg-\u003elock);\n--\nblock/blk-cgroup.c=1281=static void blkcg_destroy_blkgs(struct blkcg *blkcg)\n--\nblock/blk-cgroup.c-1292-\nblock/blk-cgroup.c:1293:\t\tblkg_destroy(blkg);\nblock/blk-cgroup.c-1294-\n--\nblock/blk-cgroup.c=1523=void blkcg_exit_disk(struct gendisk *disk)\nblock/blk-cgroup.c-1524-{\nblock/blk-cgroup.c:1525:\tblkg_destroy_all(disk);\nblock/blk-cgroup.c-1526-\tblk_throtl_exit(disk);\n--\nblock/blk-throttle.c=1886=void blk_throtl_exit(struct gendisk *disk)\n--\nblock/blk-throttle.c-1890-\t/*\nblock/blk-throttle.c:1891:\t * blkg_destroy_all() already deactivate throtl policy, just check and\nblock/blk-throttle.c-1892-\t * free throtl data.\n"
}

gemini-3.1-pro-preview

							input: 38814
							output: 26
							thoughts: 64

Okay, here's my take on those functions.

**Static Functions in the Block I/O Control Group Implementation**

Alright, let's see. So, `blkg_destroy` is definitely static, yep. And `blkg_release`, that's static as well, no question. Okay, moving on... `blkcg_css_alloc`, `blkcg_css_online`, and `blkcg_css_free` are all static too. Hmm, wait a minute... I have to double-check that `blkcg_css_alloc` is static. I could have sworn it was static. Just want to make absolutely sure before I commit that to memory. This is foundational to the concurrency design, so I need to get this right.

{
  "Expression": "blkcg_css_alloc",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=1390=static struct cgroup_subsys_state *\nblock/blk-cgroup.c:1391:blkcg_css_alloc(struct cgroup_subsys_state *parent_css)\nblock/blk-cgroup.c-1392-{\n--\nblock/blk-cgroup.c=1536=struct cgroup_subsys io_cgrp_subsys = {\nblock/blk-cgroup.c:1537:\t.css_alloc = blkcg_css_alloc,\nblock/blk-cgroup.c-1538-\t.css_online = blkcg_css_online,\n"
}

gemini-3.1-pro-preview

							input: 39242
							output: 34
							thoughts: 340

**Analyzing Block I/O Control Group Function Calls**

Okay, so I'm looking at these functions, and it's clear the static ones aren't the focus here. The non-static ones are what matter. Right now, I'm thinking about `blkg_conf_prep`, `blkg_init_queue`, `blkcg_init_disk`, `bio_associate_blkcg_from_css`, `bio_associate_blkcg`, `bio_clone_blkcg_association`, `bio_put_blkg_ref`, `bio_blkg`, and `bio_blkg_lookup`.

Hang on, `bio_blkg` and `bio_blkg_lookup` are exported, which means they're used outside this module and therefore probably critical. The question now is whether they're in the hot-path. Since `bio_blkg` and `bio_blkg_lookup` are called for *every* bio that needs a blkg, yeah, they're definitely hot-path functions.

Now, `bio_associate_blkcg_from_css` is called for every bio that needs a blkcg, so, yes, hot-path. `bio_associate_blkcg` is called for *every* bio... that's also hot-path. And `bio_clone_blkcg_association`, since it's called for every cloned bio? Yep, that's hot-path too. And `bio_put_blkg_ref` is called for *every* bio that is freed or remapped. So, again, it's hot-path.

That leaves the non-hot-path functions: `blkg_conf_prep`, `blkg_init_queue`, and `blkcg_init_disk`. Now, let me just double check. Is `blkg_conf_prep` exported? It's important to be thorough.

{
  "Expression": "EXPORT_SYMBOL_GPL\\(blkg_conf_prep\\)",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=851=int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,\n--\nblock/blk-cgroup.c-944-}\nblock/blk-cgroup.c:945:EXPORT_SYMBOL_GPL(blkg_conf_prep);\nblock/blk-cgroup.c-946-\n"
}

gemini-3.1-pro-preview

							input: 40140
							output: 34
							thoughts: 18

Okay, here's my summary, in the first person, as requested, focusing on my internal thought process:

**Exported Functions: Quick Check**

Right, so the code indicates that something is definitely exported. My immediate thought is, "Yes, it is exported." That much is clear. Now, to the more interesting point, a critical function for managing a block queue, `blkg_init_queue`, is that also exported? I need to know for sure. It's a key piece in many designs. This requires a quick check to confirm access.

{
  "Expression": "EXPORT_SYMBOL_GPL\\(blkg_init_queue\\)",
  "PathPrefix": ""
}

{
  "error": "no matches"
}

gemini-3.1-pro-preview

							input: 40325
							output: 34
							thoughts: 27

Okay, here's my summary, presented as a stream of thought:

**Checking the Export Status and Function Availability**

Wait a second, the item in question isn't explicitly *exported*, but that doesn't necessarily mean it's entirely static either. Okay, so that means I need to dig deeper. Let's see... the core question I really need to answer now is: is `blkcg_init_disk` actually exposed? Is it even exported for use elsewhere? That’s the key. Knowing that will tell me a lot about how I should be handling this.

{
  "Expression": "EXPORT_SYMBOL_GPL\\(blkcg_init_disk\\)",
  "PathPrefix": ""
}

{
  "error": "no matches"
}

gemini-3.1-pro-preview

							input: 40524
							output: 36
							thoughts: 29

Okay, here's my interpretation of those thoughts, tailored for an expert audience:

**Analyzing the Dynamic Nature of the System**

No, this isn't a simple, fixed situation. My mind immediately jumps to the dynamic aspects of the problem. I need to understand what's *changing* here. Is it an issue with data that *can* change, or the way the data changes?

Let's dig into a particular function to see what we can find, and to build context around this problem. I'll start with `blkcg_punt_bio_submit`. I seem to recall that one being exported. It's a key entry point, and that usually means it's pretty well-understood and documented, making it a good place to start my analysis.

{
  "Expression": "EXPORT_SYMBOL_GPL\\(blkcg_punt_bio_submit\\)",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=255=void blkcg_punt_bio_submit(struct bio *bio)\n--\nblock/blk-cgroup.c-268-}\nblock/blk-cgroup.c:269:EXPORT_SYMBOL_GPL(blkcg_punt_bio_submit);\nblock/blk-cgroup.c-270-\n"
}

gemini-3.1-pro-preview

							input: 40855
							output: 28
							thoughts: 64

Here's my take on those thoughts:

**Analyzing a Specific Block CGROUP Call**

Okay, so I'm looking at this call, and my initial thought is, "Is this a performance-critical path? Is it truly a 'hot-path' operation?" The context is a bit unusual; it's being called from btrfs specifically for certain BIOS interactions (`REQ_BTRFS_CGROUP_PUNT`). That's not your typical, widely-used, performance-sensitive code branch. That's good to keep in mind, and that the function itself, is not generic.

To be sure, I'll need to dig deeper. I need to understand what this code is actually *doing*. So, next up I should really look at `blkcg_set_fc_appid` and `blkcg_get_fc_appid`. Those will show me what parameters are set and used. That will allow me to check the data flow, and determine if my initial assumption is correct, or if there is something else going on here.

{
  "Expression": "blkcg_get_fc_appid",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup-fc-appid.c=43=EXPORT_SYMBOL_GPL(blkcg_set_fc_appid);\n--\nblock/blk-cgroup-fc-appid.c-45-/**\nblock/blk-cgroup-fc-appid.c:46: * blkcg_get_fc_appid - get the fc app identifier associated with a bio\nblock/blk-cgroup-fc-appid.c-47- * @bio: target bio\n--\nblock/blk-cgroup-fc-appid.c-50- */\nblock/blk-cgroup-fc-appid.c:51:char *blkcg_get_fc_appid(struct bio *bio)\nblock/blk-cgroup-fc-appid.c-52-{\n--\nblock/blk-cgroup-fc-appid.c-62-}\nblock/blk-cgroup-fc-appid.c:63:EXPORT_SYMBOL_GPL(blkcg_get_fc_appid);\n--\ndrivers/nvme/host/fc.c=1884=char *nvme_fc_io_getuuid(struct nvmefc_fcp_req *req)\n--\ndrivers/nvme/host/fc.c-1890-\t\treturn NULL;\ndrivers/nvme/host/fc.c:1891:\treturn blkcg_get_fc_appid(rq-\u003ebio);\ndrivers/nvme/host/fc.c-1892-}\n--\ndrivers/scsi/lpfc/lpfc_scsi.c=5213=static char *lpfc_is_command_vm_io(struct scsi_cmnd *cmd)\n--\ndrivers/scsi/lpfc/lpfc_scsi.c-5218-\t\treturn NULL;\ndrivers/scsi/lpfc/lpfc_scsi.c:5219:\treturn blkcg_get_fc_appid(bio);\ndrivers/scsi/lpfc/lpfc_scsi.c-5220-}\n--\ninclude/linux/blk-cgroup.h=69=int blkcg_set_fc_appid(char *app_id, u64 cgrp_id, size_t app_id_len);\ninclude/linux/blk-cgroup.h:70:char *blkcg_get_fc_appid(struct bio *bio);\ninclude/linux/blk-cgroup.h-71-\n"
}

gemini-3.1-pro-preview

							input: 41766
							output: 26
							thoughts: 216

**Analyzing the `blkcg_get_fc_appid` Modification**

Okay, so I'm looking at this patch and the first thing I notice is the change to `blkcg_get_fc_appid`.  It's a straightforward tweak, but the implications are important.  The original code had a rather verbose check that combined NULL checks of `bio->bi_blkg` and a null-string check of `fc_app_id`. This has been streamlined by fetching `blkcg` first with `bio_blkcg(bio)` and then proceeding to check its `fc_app_id` directly. The change is conceptually simple: make a local variable and simplify the if statement.

I see that this function, based on the context provided, is called from the nvme/fc and lpfc drivers.  Now, while this isn't a *generic* hot-path, it *is* a hot-path for those specific Fibre Channel NVMe drivers, so even minor optimizations here can matter. Since this is in the storage layers it is possible this could be on the hot-path and therefore it is good to have more efficient code.

Let's check the code for `bio_blkcg_css`. I want to make sure the blkcg fetch is still correct and performs as expected.

{
  "Expression": "bio_blkcg_css",
  "PathPrefix": ""
}

{
  "Output": "block/blk-cgroup.c=280=subsys_initcall(blkcg_punt_bio_init);\n--\nblock/blk-cgroup.c-283-/**\nblock/blk-cgroup.c:284: * bio_blkcg_css - return the blkcg CSS associated with a bio\nblock/blk-cgroup.c-285- * @bio: target bio\n--\nblock/blk-cgroup.c-290- */\nblock/blk-cgroup.c:291:struct cgroup_subsys_state *bio_blkcg_css(struct bio *bio)\nblock/blk-cgroup.c-292-{\n--\nblock/blk-cgroup.c-300-}\nblock/blk-cgroup.c:301:EXPORT_SYMBOL_GPL(bio_blkcg_css);\nblock/blk-cgroup.c-302-\n--\nblock/blk-cgroup.c=2232=void bio_clone_blkcg_association(struct bio *dst, struct bio *src)\n--\nblock/blk-cgroup.c-2234-\tif (bio_blkcg(src))\nblock/blk-cgroup.c:2235:\t\tbio_associate_blkcg_from_css(dst, bio_blkcg_css(src));\nblock/blk-cgroup.c-2236-}\n--\nblock/blk-cgroup.h=237=void blkg_conf_close_bdev(struct blkg_conf_ctx *ctx)\n--\nblock/blk-cgroup.h-247- * it were attached to the root blkg, and then backcharge to the actual owning\nblock/blk-cgroup.h:248: * blkg.  The idea is we do bio_blkcg_css() to look up the actual context for\nblock/blk-cgroup.h-249- * the bio and attach the appropriate blkg to the bio.  Then we call this helper\n--\ndrivers/block/loop.c=1873=static blk_status_t loop_queue_rq(struct blk_mq_hw_ctx *hctx,\n--\ndrivers/block/loop.c-1900-\tif (rq-\u003ebio) {\ndrivers/block/loop.c:1901:\t\tcmd-\u003eblkcg_css = bio_blkcg_css(rq-\u003ebio);\ndrivers/block/loop.c-1902-#ifdef CONFIG_MEMCG\n--\ninclude/linux/blk-cgroup.h=54=struct list_head *blkcg_get_cgwb_list(struct cgroup_subsys_state *css);\ninclude/linux/blk-cgroup.h:55:struct cgroup_subsys_state *bio_blkcg_css(struct bio *bio);\ninclude/linux/blk-cgroup.h-56-\n--\ninclude/linux/blk-cgroup.h=62=static inline bool blk_cgroup_congested(void) { return false; }\ninclude/linux/blk-cgroup.h:63:static inline struct cgroup_subsys_state *bio_blkcg_css(struct bio *bio)\ninclude/linux/blk-cgroup.h-64-{\n--\nkernel/trace/blktrace.c=974=static u64 blk_trace_bio_get_cgid(struct request_queue *q, struct bio *bio)\n--\nkernel/trace/blktrace.c-983-\nkernel/trace/blktrace.c:984:\tblkcg_css = bio_blkcg_css(bio);\nkernel/trace/blktrace.c-985-\tif (!blkcg_css)\n"
}

gemini-3.1-pro-preview

							input: 43196
							output: 32
							thoughts: 1317

**Focusing on Block I/O Cgroup Functions**

Okay, let's see, my goal here is to identify good candidates for further investigation within the block I/O cgroup system. I'm looking for functions that are not in the hot path, but are still significant and preferably non-static, non-inlined API entrypoints. The initial list includes several interesting functions: `blkg_conf_prep`, `blkcg_punt_bio_submit`, `bio_associate_blkcg_from_css`, `bio_clone_blkcg_association`, `bio_blkg`, and `bio_blkg_lookup`.

Let me think. `bio_blkg` and `bio_blkg_lookup` are definitely hot-path functions, so those are out, based on the requirements. Now, the instructions I've been given are pretty specific about avoiding generic hot-path stuff to prevent test bias. 

Okay, let's look at the remaining candidates, digging in further. `bio_associate_blkcg_from_css` is called from `bio_associate_blkcg` which itself is called from `bio_init`, which is hot-path.  So, no. `bio_clone_blkcg_association` is called from `__bio_clone`, another hot path.  So, also no. I should probably avoid `bio_put_blkg_ref` as well if it's hot path, and let's assume it is.

Now, considering the remaining functions, I'm thinking about `blkcg_punt_bio_submit`. This one seems promising because it's called from `btrfs endio` which is a hot-path for btrfs. However, `blkcg_punt_bio_submit` queues a work item, so it's likely not hot-path on its own, given that. I need to make sure I'm getting the right level of abstraction here. So, yes, that one is a strong possibility. I'll need to check the code.

Alright, what about `blkg_conf_prep`? It's called from sysfs writes, so it's not in the hot-path and is exported. That's a good sign. Let's keep that on the list.

Now, what about `blkg_init_queue`? It's called from `blk_alloc_queue`, which I think is not hot-path. And `blkcg_init_disk` gets called from `add_disk`, which is not hot-path either. Neither of those is static, so they're in the running. That's good. But are those exported? No, they're not. So I will have to be mindful of that limitation. But, it is okay since it's an internal function.

Let me just quickly check `blkcg_css_alloc`, `blkcg_css_online`, and `blkcg_css_free` – those are all static, so no. Not a candidate.

Okay, is anything else interesting? I should probably check any modifications, though. Hmm, `blkcg_set_ioprio` seems to just be using `bio_blkcg(bio)`, which is something I should keep in mind.  And, `blkcg_get_fc_appid` *is* modified, but that's hot-path for fc. So, no.

Back to `blkcg_punt_bio_submit`. Examining the code, it looks like it's called from `btrfs_submit_dev_bio` if `REQ_BTRFS_CGROUP_PUNT` is set. It queues a work item, so that's good in terms of avoiding the hot-path concern. That's solid.

Okay, let's re-evaluate. I think these are my best candidates so far: `blkcg_punt_bio_submit` and `blkg_conf_prep`. These are the non-hot-path, exported functions. The others I found are a bit too hot-path for this exercise.

{
  "Expression": "config BLK_CGROUP_PUNT_BIO",
  "PathPrefix": ""
}

{
  "Output": "block/Kconfig=42=config BLK_CGROUP_RWSTAT\n--\nblock/Kconfig-44-\nblock/Kconfig:45:config BLK_CGROUP_PUNT_BIO\nblock/Kconfig-46-\tbool\n"
}

gemini-3.1-pro-preview

							input: 45498
							output: 32
							thoughts: 18

Here's a summary of my thinking, presented as if it were my thought process:

**Decoding the Configuration Selection**

Okay, so I'm looking at this... and it's definitely a hidden configuration option. The key giveaway is that it's selected – meaning it's dependent on something else being true. The trigger, the parent setting, the thing that makes this specific configuration active, is BTRFS_FS. So, the presence of BTRFS_FS in the overall config file is what's telling the system to *include* this, whatever it is. I need to now figure out exactly what *this* hidden config entails and why BTRFS_FS is the key. Perhaps it's some sort of optimization or feature specific to that filesystem? The fact it's hidden implies it's either an internal detail, a rarely-used option, or something that shouldn't typically be tweaked directly by users.

{
  "Expression": "select BLK_CGROUP_PUNT_BIO",
  "PathPrefix": ""
}

{
  "Output": "fs/btrfs/Kconfig=3=config BTRFS_FS\nfs/btrfs/Kconfig-4-\ttristate \"Btrfs filesystem support\"\nfs/btrfs/Kconfig:5:\tselect BLK_CGROUP_PUNT_BIO\nfs/btrfs/Kconfig-6-\tselect CRC32\n"
}

gemini-3.1-pro-preview

							input: 45840
							output: 30
							thoughts: 258

**My Configuration Quandary**

Okay, so the initial thought was to explicitly enable `BLK_CGROUP_PUNT_BIO` since it seems relevant. But hang on, it appears that `BLK_CGROUP_PUNT_BIO` is actually *selected* by `BTRFS_FS`. So, if I enable `BTRFS_FS`, `BLK_CGROUP_PUNT_BIO` should automatically be included, right?

But then I thought, maybe I should just list `BLK_CGROUP_PUNT_BIO` directly? Hmm, no, the instructions are very clear: list it in the output array *without* the `CONFIG_` prefix. However, if `BLK_CGROUP_PUNT_BIO` is a hidden config, there's no way to enable it directly in the configuration menu. Listing both `BTRFS_FS` *and* `BLK_CGROUP_PUNT_BIO` seemed like the safer bet.

Then I realized something crucial: `BLK_CGROUP_PUNT_BIO` is a boolean without a prompt. This means it can't be enabled directly in a menuconfig session. But I remember that syzkaller *can* enable it by simply including it in the configuration file, right? Actually, hold on. I remember that syzkaller uses `make olddefconfig`. With `make olddefconfig`, hidden configs cannot be directly enabled unless their dependencies or selects are already met.

Therefore, the most reliable and correct approach is to simply include `BTRFS_FS`. It's the dependency, and that should be enough.

Now, let me check the status of `BLK_CGROUP_IOCOST`...

{
  "Expression": "config BLK_CGROUP_IOCOST",
  "PathPrefix": ""
}

{
  "Output": "block/Kconfig=145=config BLK_CGROUP_FC_APPID\n--\nblock/Kconfig-153-\nblock/Kconfig:154:config BLK_CGROUP_IOCOST\nblock/Kconfig-155-\tbool \"Enable support for cost model based cgroup IO controller\"\n"
}

gemini-3.1-pro-preview

							input: 46587
							output: 27
							thoughts: 210

**My Configuration Strategy**

Okay, so I have a set of configurations I need to enable based on prompts. I see that `BLK_CGROUP`, `BFQ_GROUP_IOSCHED`, `BLK_CGROUP_IOCOST`, `BLK_CGROUP_IOLATENCY`, `BLK_CGROUP_IOPRIO`, and `BLK_CGROUP_FC_APPID` all have prompts, so they're straightforward enables. And, of course, `BTRFS_FS` is on the list, too.

However, let me just quickly confirm something about `BFQ_GROUP_IOSCHED` before I finalize this. Ah, yes, it depends on `IOSCHED_BFQ`. That's important! So I'll definitely need to include `IOSCHED_BFQ` in my final configuration as well to make sure `BFQ_GROUP_IOSCHED` functions correctly.

{
  "Expression": "config IOSCHED_BFQ",
  "PathPrefix": ""
}

{
  "Output": "block/Kconfig.iosched=10=config MQ_IOSCHED_KYBER\n--\nblock/Kconfig.iosched-18-\nblock/Kconfig.iosched:19:config IOSCHED_BFQ\nblock/Kconfig.iosched-20-\ttristate \"BFQ I/O scheduler\"\n"
}

gemini-3.1-pro-preview

							input: 47117
							output: 32
							thoughts: 58

Okay, here's my take on those thoughts, tailored for an expert audience:

**Analyzing IO Schedulers and FC Support**

Alright, so `IOSCHED_BFQ` definitely has a prompt, got it. Now, let's pivot to `BLK_CGROUP_FC_APPID`. My initial thought is to quickly verify its dependencies. Does it rely on `NVME_FC` or `SCSI_LPFC`, like I'd expect for related functionalities? Nope, it's just a straight boolean. So, it’s a pretty simple flag, probably enabling or disabling a core feature. Good to know.

{
  "File": "block/Kconfig",
  "FirstLine": 140,
  "LineCount": 15
}

{
  "Contents": " 140:\t\tthe configured latency target, throttling anybody with a higher latency\n 141:\t\ttarget than the victimized group.\n 142:\t\n 143:\t\tNote, this is an experimental interface and could be changed someday.\n 144:\t\n 145:\tconfig BLK_CGROUP_FC_APPID\n 146:\t\tbool \"Enable support to track FC I/O Traffic across cgroup applications\"\n 147:\t\tdepends on BLK_CGROUP \u0026\u0026 NVME_FC\n 148:\t\thelp\n 149:\t\t  Enabling this option enables the support to track FC I/O traffic across\n 150:\t\t  cgroup applications. It enables the Fabric and the storage targets to\n 151:\t\t  identify, monitor, and handle FC traffic based on VM tags by inserting\n 152:\t\t  application specific identification into the FC frame.\n 153:\t\n 154:\tconfig BLK_CGROUP_IOCOST\n"
}

gemini-3.1-pro-preview

							input: 47599
							output: 25
							thoughts: 27

Okay, here's my take on those thoughts, tailored for an expert audience, presented in the first person:

**Navigating the NVMe-FC Dependency**

Alright, looking at this, it seems like the inclusion of `NVME_FC` hinges on the state of `BLK_CGROUP`. Given my understanding of the system's architecture, this is the crux of the matter. We need to be absolutely certain of the configuration surrounding `BLK_CGROUP` before we make any decisions about `NVME_FC`. Ultimately though, I'm thinking we should definitely be including `NVME_FC`. Given the functionality it enables, it's something that we should always make available in our configuration.

{
  "Expression": "config NVME_FC",
  "PathPrefix": ""
}

{
  "Output": "drivers/nvme/host/Kconfig=52=config NVME_RDMA\n--\ndrivers/nvme/host/Kconfig-66-\ndrivers/nvme/host/Kconfig:67:config NVME_FC\ndrivers/nvme/host/Kconfig-68-\ttristate \"NVM Express over Fabrics FC host driver\"\n"
}

gemini-3.1-pro-preview

							input: 47897
							output: 32
							thoughts: 27

Here's my take on the thought process:

**NVMe over Fibre Channel Investigation**

Okay, so I've been diving into this `NVMe_FC` thing, and it does seem like there's a specific prompt associated with it. That's good to know. Now, before I get too deep into this particular aspect, I think it's worth taking a quick look at `BLK_CGROUP_IOCOST`. It's connected, and understanding the interaction with I/O costs and cgroups is often critical in performance analysis and optimization. Let's just make sure I have the full picture before moving on.

{
  "File": "block/Kconfig",
  "FirstLine": 150,
  "LineCount": 15
}

{
  "Contents": " 150:\t\t  cgroup applications. It enables the Fabric and the storage targets to\n 151:\t\t  identify, monitor, and handle FC traffic based on VM tags by inserting\n 152:\t\t  application specific identification into the FC frame.\n 153:\t\n 154:\tconfig BLK_CGROUP_IOCOST\n 155:\t\tbool \"Enable support for cost model based cgroup IO controller\"\n 156:\t\tdepends on BLK_CGROUP\n 157:\t\tselect BLK_RQ_ALLOC_TIME\n 158:\t\thelp\n 159:\t\tEnabling this option enables the .weight interface for cost\n 160:\t\tmodel based proportional IO control.  The IO controller\n 161:\t\tdistributes IO capacity between different groups based on\n 162:\t\ttheir share of the overall weight distribution.\n 163:\t\n 164:\tconfig BLK_CGROUP_IOPRIO\n"
}

gemini-3.1-pro-preview

							input: 48337
							output: 32
							thoughts: 27

Okay, here's my take on those thoughts, tailored for an expert audience:

**Analyzing Block I/O Control**

Right, so the behavior here hinges on `BLK_CGROUP`. That's the baseline. Whether this specific aspect of block I/O control even comes into play is the critical first question. We need to establish the foundational infrastructure before diving deeper.

Now, presuming that `BLK_CGROUP` is indeed enabled and we are, therefore, operating within the cgroup ecosystem, the logical next step is to examine `BLK_CGROUP_IOLATENCY`. That's where the interesting stuff likely resides. Is I/O latency accounting enabled at the cgroup level? That setting dictates whether we can even *see* and *control* latency at this level. Understanding its configuration will determine our potential actions and insights going forward.

{
  "File": "block/Kconfig",
  "FirstLine": 130,
  "LineCount": 15
}

{
  "Contents": " 130:\t\tdepends on BLK_WBT\n 131:\t\thelp\n 132:\t\tEnable writeback throttling by default for request-based block devices.\n 133:\t\n 134:\tconfig BLK_CGROUP_IOLATENCY\n 135:\t\tbool \"Enable support for latency based cgroup IO protection\"\n 136:\t\tdepends on BLK_CGROUP\n 137:\t\thelp\n 138:\t\tEnabling this option enables the .latency interface for IO throttling.\n 139:\t\tThe IO controller will attempt to maintain average IO latencies below\n 140:\t\tthe configured latency target, throttling anybody with a higher latency\n 141:\t\ttarget than the victimized group.\n 142:\t\n 143:\t\tNote, this is an experimental interface and could be changed someday.\n 144:\t\n"
}

gemini-3.1-pro-preview

							input: 48807
							output: 32
							thoughts: 26

Here's my summary, as I see it:

**Navigating the Block I/O Landscape**

Okay, so we're talking about block I/O, and the crucial piece here is `BLK_CGROUP`. Everything hinges on that. Before I go further down the rabbit hole, I need to verify that we're dealing with cgroups. I think `BLK_CGROUP_IOPRIO` is the key. Let's start there and see where it takes us. We'll verify this configuration and then proceed with further analysis.

{
  "File": "block/Kconfig",
  "FirstLine": 160,
  "LineCount": 15
}

{
  "Contents": " 160:\t\tmodel based proportional IO control.  The IO controller\n 161:\t\tdistributes IO capacity between different groups based on\n 162:\t\ttheir share of the overall weight distribution.\n 163:\t\n 164:\tconfig BLK_CGROUP_IOPRIO\n 165:\t\tbool \"Cgroup I/O controller for assigning an I/O priority class\"\n 166:\t\tdepends on BLK_CGROUP\n 167:\t\thelp\n 168:\t\tEnable the .prio interface for assigning an I/O priority class to\n 169:\t\trequests. The I/O priority class affects the order in which an I/O\n 170:\t\tscheduler and block devices process requests. Only some I/O schedulers\n 171:\t\tand some block devices support I/O priorities.\n 172:\t\n 173:\tconfig BLK_DEBUG_FS\n 174:\t\tbool \"Block layer debugging information in debugfs\"\n"
}

gemini-3.1-pro-preview

							input: 49239
							output: 32
							thoughts: 2512

**Analysis of Block Cgroup Refactoring Patch for Fuzzing**

Okay, so I need to assess this patch for fuzzing potential. It seems like a significant refactoring of the block cgroup subsystem, which immediately piques my interest. It looks like the core of the change revolves around how `bio` structures are associated with block cgroups. Instead of a direct pointer to a `blkcg_gq` (blkg) in the `bio`, the patch now uses a more generic `blkcg` and dynamically looks up or pins the queue-specific `blkg` when needed, or when `BIO_BLKG_REF` flag is set. That sounds like a fundamental shift in how the system handles IO scheduling and resource allocation at the block layer. The `radix_tree` has been replaced with an `rhashtable` for `blkg` lookups, which also suggests performance implications. These are definitely functional changes worth investigating, since these kinds of core changes can easily introduce regressions or new corner cases.

First, I need to identify the relevant configurations. Based on the code, it looks like these are the important ones: `BLK_CGROUP`, `IOSCHED_BFQ`, `BFQ_GROUP_IOSCHED`, `BLK_CGROUP_IOCOST`, `BLK_CGROUP_IOLATENCY`, `BLK_CGROUP_IOPRIO`, `NVME_FC`, `BLK_CGROUP_FC_APPID`, and `BTRFS_FS`. I'll need to remember those for building a fuzzer configuration. And because `BLK_CGROUP_PUNT_BIO` is selected by `BTRFS_FS`, I can just list `BTRFS_FS`.

Next, I need to pinpoint the key functions (FocusSymbols) that are most impacted by this change. I've already identified several candidates, going through the code. These are: `blkg_conf_prep`, `blkcg_punt_bio_submit`, `blkg_init_queue`, `blkg_exit_queue`, `blkcg_init_disk`, and `blkcg_exit_disk`.

I need to confirm that these functions are actually reachable and how they are invoked. `blkg_conf_prep` is a good candidate since it's called from sysfs write operations, which are readily accessible. `blkcg_punt_bio_submit` looks interesting too. It's called from the `btrfs_submit_dev_bio` function if `REQ_BTRFS_CGROUP_PUNT` is set, so BTRFS IO is a good target. I can potentially craft IO scenarios with BTRFS to trigger it. The device creation/destruction functions `blkg_init_queue`, `blkg_exit_queue`, `blkcg_init_disk`, `blkcg_exit_disk` also look promising, since syzkaller can create/destroy loop devices, null_blk, etc. to trigger these.

I double-checked to confirm the dependencies for some of the related configurations: `BTRFS_FS`, `NVME_FC`, `IOSCHED_BFQ`. Good to know.

Therefore, my strategy is to focus on these functions and configurations when constructing fuzz test cases. The refactoring itself looks pretty significant. I'm seeing a deep change that's fundamental to block cgroup IO control, the block layer, and btrfs.

{
  "File": "block/Kconfig",
  "FirstLine": 130,
  "LineCount": 10
}

{
  "Contents": " 130:\t\tdepends on BLK_WBT\n 131:\t\thelp\n 132:\t\tEnable writeback throttling by default for request-based block devices.\n 133:\t\n 134:\tconfig BLK_CGROUP_IOLATENCY\n 135:\t\tbool \"Enable support for latency based cgroup IO protection\"\n 136:\t\tdepends on BLK_CGROUP\n 137:\t\thelp\n 138:\t\tEnabling this option enables the .latency interface for IO throttling.\n 139:\t\tThe IO controller will attempt to maintain average IO latencies below\n"
}

gemini-3.1-pro-preview

							input: 52683
							output: 32
							thoughts: 28

Okay, here's my take on those thoughts, framed as an expert's internal monologue:

**Investigating Block Device I/O Control**

Alright, this situation hinges on `BLK_CGROUP`. That's the core, isn't it? It dictates how much of the relevant functionality is even *available* in the first place. Without it, we're likely dealing with a severely limited scenario. I need to keep that in mind as the fundamental building block.

Now, before I dive too deep, I need to revisit the dependency relationships for `BLK_CGROUP_IOCOST`. This is crucial. It's easy to get lost in a tangled web of kernel configuration options, and I want to be *absolutely* sure that I understand which options are enabled. I'll re-examine the direct dependencies, any indirect ones, and the conditional dependencies, to make certain that `BLK_CGROUP_IOCOST` is being pulled in as expected under the relevant conditions. There's a risk of introducing the wrong behaviour.

{
  "File": "block/Kconfig",
  "FirstLine": 150,
  "LineCount": 10
}

{
  "Contents": " 150:\t\t  cgroup applications. It enables the Fabric and the storage targets to\n 151:\t\t  identify, monitor, and handle FC traffic based on VM tags by inserting\n 152:\t\t  application specific identification into the FC frame.\n 153:\t\n 154:\tconfig BLK_CGROUP_IOCOST\n 155:\t\tbool \"Enable support for cost model based cgroup IO controller\"\n 156:\t\tdepends on BLK_CGROUP\n 157:\t\tselect BLK_RQ_ALLOC_TIME\n 158:\t\thelp\n 159:\t\tEnabling this option enables the .weight interface for cost\n"
}

gemini-3.1-pro-preview

							input: 53137
							output: 32
							thoughts: 29

Here's my take on those thoughts:

**Analyzing Block I/O Cgroup Configuration**

Okay, so the architecture I'm working with hinges on `BLK_CGROUP`. That's the baseline. I need to understand how block I/O cgroups are actually implemented in this system before I go any further. It's the foundation upon which everything else is built.

Now, let me double-check the dependencies for `BLK_CGROUP_IOPRIO`. I *thought* I had the relationship mapped, but I need to make sure I haven't missed anything. I want to be *absolutely* certain that I haven't overlooked any crucial interactions or potential conflicts. I need to make sure that its configuration is solid, to avoid issues downstream. Gotta get this right. This is important. I’ll make sure it plays nicely with other relevant features.

{
  "File": "block/Kconfig",
  "FirstLine": 160,
  "LineCount": 10
}

{
  "Contents": " 160:\t\tmodel based proportional IO control.  The IO controller\n 161:\t\tdistributes IO capacity between different groups based on\n 162:\t\ttheir share of the overall weight distribution.\n 163:\t\n 164:\tconfig BLK_CGROUP_IOPRIO\n 165:\t\tbool \"Cgroup I/O controller for assigning an I/O priority class\"\n 166:\t\tdepends on BLK_CGROUP\n 167:\t\thelp\n 168:\t\tEnable the .prio interface for assigning an I/O priority class to\n 169:\t\trequests. The I/O priority class affects the order in which an I/O\n"
}

gemini-3.1-pro-preview

							input: 53563
							output: 327
							thoughts: 121

Okay, here's my interpretation of this configuration landscape, presented as a summary of my thought process:

**Analyzing Block I/O Configuration Dependencies**

Alright, let's break this down. My initial observation hinges directly on the `BLK_CGROUP` setting. That's the linchpin, the foundation of this whole block I/O configuration space. The presence (or absence) of that macro will significantly influence everything else.

Therefore, the relevant configurations here are: `BLK_CGROUP` itself, of course. Then we have `IOSCHED_BFQ`, `BFQ_GROUP_IOSCHED`, `BLK_CGROUP_IOCOST`, `BLK_CGROUP_IOLATENCY`, `BLK_CGROUP_IOPRIO`, `NVME_FC`, `BLK_CGROUP_FC_APPID` and `BTRFS_FS`. Each of these are likely conditional on that initial `BLK_CGROUP` being defined, or at least they are interconnected. The specific behavior of each depends on which features are enabled.

At this point, I'll switch over to the `set-results` tool to analyze the specific combinations of these configurations and see their impact on the system. Time to execute.

{
  "EnableConfigs": [
    "BLK_CGROUP",
    "IOSCHED_BFQ",
    "BFQ_GROUP_IOSCHED",
    "BLK_CGROUP_IOCOST",
    "BLK_CGROUP_IOLATENCY",
    "BLK_CGROUP_IOPRIO",
    "NVME_FC",
    "BLK_CGROUP_FC_APPID",
    "BTRFS_FS"
  ],
  "FocusSymbols": [
    "blkg_conf_prep",
    "blkcg_punt_bio_submit",
    "blkg_init_queue",
    "blkg_exit_queue",
    "blkcg_init_disk",
    "blkcg_exit_disk"
  ],
  "Reasoning": "The patch significantly refactors the block cgroup (blkcg) subsystem by changing how `bio`s are associated with block cgroups. Instead of directly storing a queue-specific `blkcg_gq` (blkg) in `bio-\u003ebi_blkg`, it now stores the generic `blkcg` in `bio-\u003ebi_blkcg` and dynamically looks up or pins the queue-specific `blkg` using a new `BIO_BLKG_REF` flag. It also replaces the `radix_tree` used for `blkg` lookups with an `rhashtable`. These are core functional changes to the block layer and cgroup IO controllers, affecting IO scheduling, throttling, and accounting.",
  "WorthFuzzing": true
}

{
  "EnableConfigs": [
    "BLK_CGROUP",
    "IOSCHED_BFQ",
    "BFQ_GROUP_IOSCHED",
    "BLK_CGROUP_IOCOST",
    "BLK_CGROUP_IOLATENCY",
    "BLK_CGROUP_IOPRIO",
    "NVME_FC",
    "BLK_CGROUP_FC_APPID",
    "BTRFS_FS"
  ],
  "FocusSymbols": [
    "blkg_conf_prep",
    "blkcg_punt_bio_submit",
    "blkg_init_queue",
    "blkg_exit_queue",
    "blkcg_init_disk",
    "blkcg_exit_disk"
  ],
  "Reasoning": "The patch significantly refactors the block cgroup (blkcg) subsystem by changing how `bio`s are associated with block cgroups. Instead of directly storing a queue-specific `blkcg_gq` (blkg) in `bio-\u003ebi_blkg`, it now stores the generic `blkcg` in `bio-\u003ebi_blkcg` and dynamically looks up or pins the queue-specific `blkg` using a new `BIO_BLKG_REF` flag. It also replaces the `radix_tree` used for `blkg` lookups with an `rhashtable`. These are core functional changes to the block layer and cgroup IO controllers, affecting IO scheduling, throttling, and accounting.",
  "WorthFuzzing": true
}

{
  "WorthFuzzing": true
}

{
  "KMSANReasoning": "The patch refactors the block cgroup (blkcg) association for `bio` structures, replacing the direct `bi_blkg` pointer with `bi_blkcg` and dynamically looking up or pinning the queue-local `blkcg_gq` (blkg) using a hash table instead of a radix tree.\n\nThese changes involve modifying internal kernel structures (`bio`, `request_queue`, `blkcg_gq`) and their initialization/cleanup routines (`bio_init`, `bio_uninit`, `bio_clear_blkcg`). No new structures are copied to user space, nor are there any new buffers or complex data structures allocated without proper initialization. The modified fields in `bio` are explicitly initialized to `NULL` or appropriate values in `bio_init()`.\n\nAny bugs introduced by these changes would likely manifest as NULL pointer dereferences, use-after-free (e.g., incorrect reference counting of `blkcg` or `blkg`), or memory leaks, all of which are effectively caught by standard KASAN and other default debugging tools. There is no risk of uninitialized memory reads or kernel-to-user info-leaks that would necessitate a dedicated KMSAN fuzzing session.",
  "NeedsKMSAN": false
}

You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.

CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
  a comprehensive suite of debugging tools and sanitizers, including KASAN
  (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
  (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
  or page allocations) and kernel-to-user memory info-leaks.

Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.

Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.

Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
   attributes, ioctl output arguments, socket options, or BPF buffers) where fields
   or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
   or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
   could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
   uninitialized bytes of existing buffers.

Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
  dereferences, locking deadlocks, or use-after-free bugs (these are already caught
  by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
  or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.

Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit abc0b1a65e87d979eec053ca54836b90fdec9729
Author: syz-cluster <triage@syzkaller.com>
Date:   Tue Aug 18 08:04:15 2026 +0000

    syz-cluster: applied patch under review

diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 14b8c571c0d14..bbd79931d5ab5 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -3235,7 +3235,7 @@ the configuration, the bio may be executed at a lower priority and if
 the writeback session is holding shared resources, e.g. a journal
 entry, may lead to priority inversion.  There is no one easy solution
 for the problem.  Filesystems can try to work around specific problem
-cases by skipping wbc_init_bio() and using bio_associate_blkg()
+cases by skipping wbc_init_bio() and using bio_associate_blkcg()
 directly.
 
 
diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
index ef9d63e8542ed..4a3975f9ff744 100644
--- a/block/bfq-cgroup.c
+++ b/block/bfq-cgroup.c
@@ -363,11 +363,13 @@ void bfqg_and_blkg_put(struct bfq_group *bfqg)
 
 void bfqg_stats_update_legacy_io(struct request_queue *q, struct request *rq)
 {
-	struct bfq_group *bfqg = blkg_to_bfqg(rq->bio->bi_blkg);
+	struct blkcg_gq *blkg = bio_blkg_lookup(rq->bio);
+	struct bfq_group *bfqg;
 
-	if (!bfqg)
+	if (!blkg)
 		return;
 
+	bfqg = blkg_to_bfqg(blkg);
 	blkg_rwstat_add(&bfqg->stats.bytes, rq->cmd_flags, blk_rq_bytes(rq));
 	blkg_rwstat_add(&bfqg->stats.ios, rq->cmd_flags, 1);
 }
@@ -606,7 +608,7 @@ static void bfq_link_bfqg(struct bfq_data *bfqd, struct bfq_group *bfqg)
 
 struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio)
 {
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg_gq *blkg = bio_blkg_lookup(bio);
 	struct bfq_group *bfqg;
 
 	while (blkg) {
@@ -614,15 +616,17 @@ struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio)
 			blkg = blkg->parent;
 			continue;
 		}
+
 		bfqg = blkg_to_bfqg(blkg);
 		if (bfqg->pd.online) {
-			bio_associate_blkg_from_css(bio, &blkg->blkcg->css);
+			bio_associate_blkcg_from_css(bio, &blkg->blkcg->css);
 			return bfqg;
 		}
 		blkg = blkg->parent;
 	}
-	bio_associate_blkg_from_css(bio,
-				&bfqg_to_blkg(bfqd->root_group)->blkcg->css);
+
+	blkg = bfqg_to_blkg(bfqd->root_group);
+	bio_associate_blkcg_from_css(bio, &blkg->blkcg->css);
 	return bfqd->root_group;
 }
 
diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 0f75301b3115f..d4f657bc7a4e5 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -128,6 +128,7 @@
 
 #include "elevator.h"
 #include "blk.h"
+#include "blk-cgroup.h"
 #include "blk-mq.h"
 #include "blk-mq-sched.h"
 #include "bfq-iosched.h"
@@ -2452,15 +2453,16 @@ static bool bfq_bio_merge(struct request_queue *q, struct bio *bio,
 	struct request *free = NULL;
 	bool ret;
 
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+	/* blkg creation takes q->queue_lock, so do it before bfqd->lock. */
+	if (bic)
+		bio_blkg(bio);
+#endif
+
 	spin_lock_irq(&bfqd->lock);
 
 	if (bic) {
-		/*
-		 * Make sure cgroup info is uptodate for current process before
-		 * considering the merge.
-		 */
 		bfq_bic_update_cgroup(bic, bio);
-
 		bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf),
 					     bfq_actuator_index(bfqd, bio));
 	} else {
@@ -6245,6 +6247,13 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,
 	LIST_HEAD(free);
 
 #ifdef CONFIG_BFQ_GROUP_IOSCHED
+	/*
+	 * Pin the blkg used to look up bfqg.  If this is the first IO for
+	 * the blkcg on this queue, create the bfqg before holding bfqd->lock.
+	 */
+	if (rq->bio && !bio_flagged(rq->bio, BIO_BLKG_REF))
+		bio_blkg(rq->bio);
+
 	if (!cgroup_subsys_on_dfl(io_cgrp_subsys) && rq->bio)
 		bfqg_stats_update_legacy_io(q, rq);
 #endif
diff --git a/block/bio.c b/block/bio.c
index 898b2f5ef8c81..6c84042adf5dd 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -179,12 +179,7 @@ static inline gfp_t try_alloc_gfp(gfp_t gfp)
 
 void bio_uninit(struct bio *bio)
 {
-#ifdef CONFIG_BLK_CGROUP
-	if (bio->bi_blkg) {
-		blkg_put(bio->bi_blkg);
-		bio->bi_blkg = NULL;
-	}
-#endif
+	bio_clear_blkcg(bio);
 	if (bio_integrity(bio))
 		bio_integrity_free(bio);
 
@@ -233,10 +228,10 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,
 	bio->bi_end_io = NULL;
 	bio->bi_private = NULL;
 #ifdef CONFIG_BLK_CGROUP
-	bio->bi_blkg = NULL;
+	bio->bi_blkcg = NULL;
 	bio->issue_time_ns = 0;
 	if (bdev)
-		bio_associate_blkg(bio);
+		bio_associate_blkcg(bio);
 #ifdef CONFIG_BLK_CGROUP_IOCOST
 	bio->bi_iocost_cost = 0;
 #endif
@@ -281,7 +276,7 @@ void bio_reset(struct bio *bio, struct block_device *bdev, blk_opf_t opf)
 	bio->bi_io_vec = bv;
 	bio->bi_bdev = bdev;
 	if (bio->bi_bdev)
-		bio_associate_blkg(bio);
+		bio_associate_blkcg(bio);
 	bio->bi_opf = opf;
 }
 EXPORT_SYMBOL(bio_reset);
@@ -866,7 +861,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
 		if (bio->bi_bdev == bio_src->bi_bdev &&
 		    bio_flagged(bio_src, BIO_REMAPPED))
 			bio_set_flag(bio, BIO_REMAPPED);
-		bio_clone_blkg_association(bio, bio_src);
+		bio_clone_blkcg_association(bio, bio_src);
 	}
 
 	if (bio_crypt_clone(bio, bio_src, gfp) < 0)
@@ -1910,17 +1905,12 @@ void bio_endio(struct bio *bio)
 		goto again;
 	}
 
-#ifdef CONFIG_BLK_CGROUP
 	/*
 	 * Release cgroup info.  We shouldn't have to do this here, but quite
 	 * a few callers of bio_init fail to call bio_uninit, so we cover up
 	 * for that here at least for now.
 	 */
-	if (bio->bi_blkg) {
-		blkg_put(bio->bi_blkg);
-		bio->bi_blkg = NULL;
-	}
-#endif
+	bio_clear_blkcg(bio);
 
 	if (bio_flagged(bio, BIO_COMPLETE_IN_TASK) && bio_in_atomic())
 		__bio_complete_in_task(bio);
diff --git a/block/blk-cgroup-fc-appid.c b/block/blk-cgroup-fc-appid.c
index 3ec21333f393b..7589c62099898 100644
--- a/block/blk-cgroup-fc-appid.c
+++ b/block/blk-cgroup-fc-appid.c
@@ -50,8 +50,14 @@ EXPORT_SYMBOL_GPL(blkcg_set_fc_appid);
  */
 char *blkcg_get_fc_appid(struct bio *bio)
 {
-	if (!bio->bi_blkg || bio->bi_blkg->blkcg->fc_app_id[0] == '\0')
+	struct blkcg *blkcg = bio_blkcg(bio);
+
+	if (!blkcg)
+		return NULL;
+
+	if (blkcg->fc_app_id[0] == '\0')
 		return NULL;
-	return bio->bi_blkg->blkcg->fc_app_id;
+
+	return blkcg->fc_app_id;
 }
 EXPORT_SYMBOL_GPL(blkcg_get_fc_appid);
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 2b5c29434e426..2c758c26bce60 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -66,6 +66,13 @@ static DEFINE_RAW_SPINLOCK(blkg_stat_lock);
 
 #define BLKG_DESTROY_BATCH_SIZE  64
 
+const struct rhashtable_params blkg_hash_params = {
+	.key_len		= sizeof_field(struct blkcg_gq, blkcg_id),
+	.key_offset		= offsetof(struct blkcg_gq, blkcg_id),
+	.head_offset		= offsetof(struct blkcg_gq, q_hash_node),
+	.automatic_shrinking	= true,
+};
+
 /*
  * Lockless lists for tracking IO stats update
  *
@@ -173,10 +180,6 @@ static void __blkg_release(struct rcu_head *rcu)
 {
 	struct blkcg_gq *blkg = container_of(rcu, struct blkcg_gq, rcu_head);
 
-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
-	WARN_ON(!bio_list_empty(&blkg->async_bios));
-#endif
-
 	blkg_free(blkg);
 }
 
@@ -194,6 +197,16 @@ static void blkg_release(struct percpu_ref *ref)
 	struct blkcg *blkcg = blkg->blkcg;
 	int cpu;
 
+	/*
+	 * A blkg that was never inserted into q->blkg_list has no hash
+	 * entry.  This happens when allocation or creation fails before
+	 * rhashtable_insert_fast() succeeds.
+	 */
+	if (!list_empty(&blkg->q_node))
+		WARN_ON_ONCE(rhashtable_remove_fast(&blkg->q->blkg_hash,
+						    &blkg->q_hash_node,
+						    blkg_hash_params));
+
 	/*
 	 * Flush all the non-empty percpu lockless lists before releasing
 	 * us, given these stat belongs to us.
@@ -209,19 +222,18 @@ static void blkg_release(struct percpu_ref *ref)
 #ifdef CONFIG_BLK_CGROUP_PUNT_BIO
 static struct workqueue_struct *blkcg_punt_bio_wq;
 
-static void blkg_async_bio_workfn(struct work_struct *work)
+static void blkcg_async_bio_workfn(struct work_struct *work)
 {
-	struct blkcg_gq *blkg = container_of(work, struct blkcg_gq,
-					     async_bio_work);
+	struct blkcg *blkcg = container_of(work, struct blkcg, async_bio_work);
 	struct bio_list bios = BIO_EMPTY_LIST;
 	struct bio *bio;
 	struct blk_plug plug;
 	bool need_plug = false;
 
-	/* as long as there are pending bios, @blkg can't go away */
-	spin_lock(&blkg->async_bio_lock);
-	bio_list_merge_init(&bios, &blkg->async_bios);
-	spin_unlock(&blkg->async_bio_lock);
+	/* as long as there are pending bios, @blkcg can't go away */
+	spin_lock(&blkcg->async_bio_lock);
+	bio_list_merge_init(&bios, &blkcg->async_bios);
+	spin_unlock(&blkcg->async_bio_lock);
 
 	/* start plug only when bio_list contains at least 2 bios */
 	if (bios.head && bios.head->bi_next) {
@@ -242,15 +254,15 @@ static void blkg_async_bio_workfn(struct work_struct *work)
  */
 void blkcg_punt_bio_submit(struct bio *bio)
 {
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg *blkcg = bio_blkcg(bio);
 
-	if (blkg->parent) {
-		spin_lock(&blkg->async_bio_lock);
-		bio_list_add(&blkg->async_bios, bio);
-		spin_unlock(&blkg->async_bio_lock);
-		queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work);
+	if (blkcg && cgroup_parent(blkcg->css.cgroup)) {
+		spin_lock(&blkcg->async_bio_lock);
+		bio_list_add(&blkcg->async_bios, bio);
+		spin_unlock(&blkcg->async_bio_lock);
+		queue_work(blkcg_punt_bio_wq, &blkcg->async_bio_work);
 	} else {
-		/* never bounce for the root cgroup */
+		/* Never bounce if there is no non-root blkcg to queue on. */
 		submit_bio(bio);
 	}
 }
@@ -278,9 +290,13 @@ subsys_initcall(blkcg_punt_bio_init);
  */
 struct cgroup_subsys_state *bio_blkcg_css(struct bio *bio)
 {
-	if (!bio || !bio->bi_blkg)
+	struct blkcg *blkcg;
+
+	if (!bio)
 		return NULL;
-	return &bio->bi_blkg->blkcg->css;
+
+	blkcg = bio_blkcg(bio);
+	return blkcg ? &blkcg->css : NULL;
 }
 EXPORT_SYMBOL_GPL(bio_blkcg_css);
 
@@ -327,12 +343,8 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct gendisk *disk,
 	blkg->q = disk->queue;
 	INIT_LIST_HEAD(&blkg->q_node);
 	blkg->blkcg = blkcg;
+	blkg->blkcg_id = blkcg->css.id;
 	blkg->iostat.blkg = blkg;
-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
-	spin_lock_init(&blkg->async_bio_lock);
-	bio_list_init(&blkg->async_bios);
-	INIT_WORK(&blkg->async_bio_work, blkg_async_bio_workfn);
-#endif
 
 	u64_stats_init(&blkg->iostat.sync);
 	for_each_possible_cpu(cpu) {
@@ -423,7 +435,8 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,
 
 	/* insert */
 	spin_lock(&blkcg->lock);
-	ret = radix_tree_insert(&blkcg->blkg_tree, disk->queue->id, blkg);
+	ret = rhashtable_insert_fast(&disk->queue->blkg_hash,
+				     &blkg->q_hash_node, blkg_hash_params);
 	if (likely(!ret)) {
 		hlist_add_head_rcu(&blkg->blkcg_node, &blkcg->blkg_list);
 		list_add(&blkg->q_node, &disk->queue->blkg_list);
@@ -454,6 +467,17 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,
 	return ERR_PTR(ret);
 }
 
+/*
+ * The root blkg holds a live reference while the disk is active, so walking
+ * the parent chain always finds a blkg which can be pinned.
+ */
+static struct blkcg_gq *blkg_lookup_tryget(struct blkcg_gq *blkg)
+{
+	while (!blkg_tryget(blkg))
+		blkg = blkg->parent;
+	return blkg;
+}
+
 /**
  * blkg_lookup_create - lookup blkg, try to create one if not there
  * @blkcg: blkcg of interest
@@ -461,11 +485,13 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,
  *
  * Lookup blkg for the @blkcg - @disk pair.  If it doesn't exist, try to
  * create one.  blkg creation is performed recursively from blkcg_root such
- * that all non-root blkg's have access to the parent blkg.  This function
- * should be called under RCU read lock and takes @disk->queue->queue_lock.
+ * that all non-root blkg's have access to the parent blkg.
  *
- * Returns the blkg or the closest blkg if blkg_create() fails as it walks
- * down from root.
+ * Must be called with @disk->queue->queue_lock held.
+ *
+ * Returns the closest blkg with an extra reference acquired.  If
+ * blkg_create() fails while walking down from root, the returned blkg may
+ * belong to an ancestor of @blkcg.  This function never returns %NULL.
  */
 static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,
 		struct gendisk *disk)
@@ -473,12 +499,12 @@ static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,
 	struct request_queue *q = disk->queue;
 	struct blkcg_gq *blkg;
 
+	lockdep_assert_held(&q->queue_lock);
+
 	rcu_read_lock();
 	blkg = blkg_lookup(blkcg, q);
 	if (blkg) {
-		if (blkcg != &blkcg_root &&
-		    blkg != rcu_dereference(blkcg->blkg_hint))
-			rcu_assign_pointer(blkcg->blkg_hint, blkg);
+		blkg = blkg_lookup_tryget(blkg);
 		rcu_read_unlock();
 		return blkg;
 	}
@@ -514,7 +540,7 @@ static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,
 			break;
 	}
 
-	return blkg;
+	return blkg_lookup_tryget(blkg);
 }
 
 static void blkg_destroy(struct blkcg_gq *blkg)
@@ -546,17 +572,8 @@ static void blkg_destroy(struct blkcg_gq *blkg)
 
 	blkg->online = false;
 
-	radix_tree_delete(&blkcg->blkg_tree, blkg->q->id);
 	hlist_del_init_rcu(&blkg->blkcg_node);
 
-	/*
-	 * Both setting lookup hint to and clearing it from @blkg are done
-	 * under queue_lock.  If it's not pointing to @blkg now, it never
-	 * will.  Hint assignment itself can race safely.
-	 */
-	if (rcu_access_pointer(blkcg->blkg_hint) == blkg)
-		rcu_assign_pointer(blkcg->blkg_hint, NULL);
-
 	/*
 	 * Put the reference taken at the time of creation so that when all
 	 * queues are gone, group can be destroyed.
@@ -882,18 +899,12 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 			goto fail_exit;
 		}
 
-		if (radix_tree_preload(GFP_KERNEL)) {
-			blkg_free(new_blkg);
-			ret = -ENOMEM;
-			goto fail_exit;
-		}
-
 		spin_lock_irq(&q->queue_lock);
 
 		if (!blkcg_policy_enabled(q, pol)) {
 			blkg_free(new_blkg);
 			ret = -EOPNOTSUPP;
-			goto fail_preloaded;
+			goto fail_unlock;
 		}
 
 		blkg = blkg_lookup(pos, q);
@@ -903,12 +914,10 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 			blkg = blkg_create(pos, disk, new_blkg);
 			if (IS_ERR(blkg)) {
 				ret = PTR_ERR(blkg);
-				goto fail_preloaded;
+				goto fail_unlock;
 			}
 		}
 
-		radix_tree_preload_end();
-
 		if (pos == blkcg)
 			goto success;
 	}
@@ -917,8 +926,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 	ctx->blkg = blkg;
 	return 0;
 
-fail_preloaded:
-	radix_tree_preload_end();
 fail_unlock:
 	spin_unlock_irq(&q->queue_lock);
 fail_exit:
@@ -1373,6 +1380,9 @@ static void blkcg_css_free(struct cgroup_subsys_state *css)
 
 	mutex_unlock(&blkcg_pol_mutex);
 
+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
+	WARN_ON(!bio_list_empty(&blkcg->async_bios));
+#endif
 	free_percpu(blkcg->lhead);
 	kfree(blkcg);
 }
@@ -1420,8 +1430,12 @@ blkcg_css_alloc(struct cgroup_subsys_state *parent_css)
 
 	spin_lock_init(&blkcg->lock);
 	refcount_set(&blkcg->online_pin, 1);
-	INIT_RADIX_TREE(&blkcg->blkg_tree, GFP_NOWAIT);
 	INIT_HLIST_HEAD(&blkcg->blkg_list);
+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
+	spin_lock_init(&blkcg->async_bio_lock);
+	bio_list_init(&blkcg->async_bios);
+	INIT_WORK(&blkcg->async_bio_work, blkcg_async_bio_workfn);
+#endif
 #ifdef CONFIG_CGROUP_WRITEBACK
 	INIT_LIST_HEAD(&blkcg->cgwb_list);
 #endif
@@ -1457,17 +1471,22 @@ static int blkcg_css_online(struct cgroup_subsys_state *css)
 	return 0;
 }
 
-void blkg_init_queue(struct request_queue *q)
+int blkg_init_queue(struct request_queue *q)
 {
 	INIT_LIST_HEAD(&q->blkg_list);
 	mutex_init(&q->blkcg_mutex);
+	return rhashtable_init(&q->blkg_hash, &blkg_hash_params);
+}
+
+void blkg_exit_queue(struct request_queue *q)
+{
+	rhashtable_destroy(&q->blkg_hash);
 }
 
 int blkcg_init_disk(struct gendisk *disk)
 {
 	struct request_queue *q = disk->queue;
 	struct blkcg_gq *new_blkg, *blkg;
-	bool preloaded;
 
 	/*
 	 * If the queue is shared across disk rebind (e.g., SCSI), the
@@ -1485,8 +1504,6 @@ int blkcg_init_disk(struct gendisk *disk)
 	if (!new_blkg)
 		return -ENOMEM;
 
-	preloaded = !radix_tree_preload(GFP_KERNEL);
-
 	/* Make sure the root blkg exists. */
 	/* spin_lock_irq can serve as RCU read-side critical section. */
 	spin_lock_irq(&q->queue_lock);
@@ -1496,15 +1513,10 @@ int blkcg_init_disk(struct gendisk *disk)
 	q->root_blkg = blkg;
 	spin_unlock_irq(&q->queue_lock);
 
-	if (preloaded)
-		radix_tree_preload_end();
-
 	return 0;
 
 err_unlock:
 	spin_unlock_irq(&q->queue_lock);
-	if (preloaded)
-		radix_tree_preload_end();
 	return PTR_ERR(blkg);
 }
 
@@ -2064,129 +2076,165 @@ void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta)
 	atomic64_add(delta, &blkg->delay_nsec);
 }
 
-static inline struct blkcg_gq *blkg_lookup_tryget(struct blkcg_gq *blkg)
+/*
+ * Return the blkg pinned by @bio through BIO_BLKG_REF.  The returned blkg is
+ * already owned by @bio and no extra reference is acquired.
+ */
+static struct blkcg_gq *bio_pinned_blkg(struct bio *bio)
 {
-retry:
-	if (blkg_tryget(blkg))
-		return blkg;
+	struct request_queue *q = bdev_get_queue(bio->bi_bdev);
+	struct blkcg_gq *blkg;
 
-	blkg = blkg->parent;
-	if (blkg)
-		goto retry;
+	rcu_read_lock();
+	blkg = blkg_lookup_any(bio_blkcg(bio), q);
+	rcu_read_unlock();
 
-	return NULL;
+	WARN_ON_ONCE(!blkg);
+	return blkg;
+}
+
+static void bio_set_blkg_ref(struct bio *bio, struct blkcg_gq *blkg)
+{
+	if (bio_blkcg(bio) != blkg->blkcg) {
+		css_get(&blkg->blkcg->css);
+		bio_clear_blkcg(bio);
+		bio->bi_blkcg = blkg->blkcg;
+	}
+	bio_set_flag(bio, BIO_BLKG_REF);
 }
+
 /**
- * blkg_tryget_closest - try and get a blkg ref on the closet blkg
+ * bio_blkg_lookup - look up a blkg associated with a bio
  * @bio: target bio
- * @css: target css
  *
- * As the failure mode here is to walk up the blkg tree, this ensure that the
- * blkg->parent pointers are always valid.  This returns the blkg that it ended
- * up taking a reference on or %NULL if no reference was taken.
+ * Return the blkg already pinned by @bio, or %NULL if @bio doesn't own a blkg
+ * reference.  Call bio_blkg() instead when a missing blkg should be created.
  */
-static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio,
-		struct cgroup_subsys_state *css)
+struct blkcg_gq *bio_blkg_lookup(struct bio *bio)
 {
-	struct request_queue *q = bio->bi_bdev->bd_queue;
-	struct blkcg *blkcg = css_to_blkcg(css);
+	if (!bio_flagged(bio, BIO_BLKG_REF))
+		return NULL;
+	return bio_pinned_blkg(bio);
+}
+EXPORT_SYMBOL_GPL(bio_blkg_lookup);
+
+/**
+ * bio_put_blkg_ref - drop the blkg reference pinned by a bio
+ * @bio: target bio
+ *
+ * Drop the bio-owned blkg reference acquired by bio_blkg(), if any.
+ */
+void bio_put_blkg_ref(struct bio *bio)
+{
+	if (bio_flagged(bio, BIO_BLKG_REF)) {
+		struct blkcg_gq *blkg = bio_pinned_blkg(bio);
+
+		if (blkg)
+			blkg_put(blkg);
+		bio_clear_flag(bio, BIO_BLKG_REF);
+	}
+}
+EXPORT_SYMBOL_GPL(bio_put_blkg_ref);
+
+/**
+ * bio_blkg - look up the blkg associated with a bio
+ * @bio: target bio
+ *
+ * Look up the queue-local blkg for @bio's current device and blkcg.  If this
+ * is the first policy use of @bio, create the missing blkg hierarchy if
+ * necessary, pin the closest available blkg, and update @bio's blkcg
+ * association if allocation falls back to an ancestor.
+ */
+struct blkcg_gq *bio_blkg(struct bio *bio)
+{
+	struct blkcg *blkcg = bio_blkcg(bio);
+	struct gendisk *disk;
+	struct request_queue *q;
 	struct blkcg_gq *blkg;
 
-	rcu_read_lock();
-	blkg = blkg_lookup(blkcg, q);
-	if (likely(blkg))
-		blkg = blkg_lookup_tryget(blkg);
-	rcu_read_unlock();
+	if (!blkcg || !bio->bi_bdev)
+		return NULL;
 
-	if (blkg)
-		return blkg;
+	if (bio_flagged(bio, BIO_BLKG_REF))
+		return bio_pinned_blkg(bio);
+
+	disk = bio->bi_bdev->bd_disk;
+	q = disk->queue;
 
-	/*
-	 * Fast path failed, we're probably issuing IO in this cgroup the first
-	 * time, hold lock to create new blkg.
-	 */
 	spin_lock_irq(&q->queue_lock);
-	blkg = blkg_lookup_create(blkcg, bio->bi_bdev->bd_disk);
-	if (blkg)
-		blkg = blkg_lookup_tryget(blkg);
+	blkg = blkg_lookup_create(blkcg, disk);
 	spin_unlock_irq(&q->queue_lock);
 
+	bio_set_blkg_ref(bio, blkg);
 	return blkg;
 }
+EXPORT_SYMBOL_GPL(bio_blkg);
 
 /**
- * bio_associate_blkg_from_css - associate a bio with a specified css
+ * bio_associate_blkcg_from_css - associate a bio with a specified css
  * @bio: target bio
  * @css: target css
  *
- * Associate @bio with the blkg found by combining the css's blkg and the
- * request_queue of the @bio.  An association failure is handled by walking up
- * the blkg tree.  Therefore, the blkg associated can be anything between @blkg
- * and q->root_blkg.  This situation only happens when a cgroup is dying and
- * then the remaining bios will spill to the closest alive blkg.
+ * Associate @bio with the blkcg found from @css.  The queue-local blkg is
+ * created and pinned by bio_blkg() when blkcg policies need it.  If @css
+ * can't be referenced online, associate @bio with the root blkcg instead.
  *
- * A reference will be taken on the blkg and will be released when @bio is
+ * A reference will be taken on the blkcg and will be released when @bio is
  * freed.
  */
-void bio_associate_blkg_from_css(struct bio *bio,
-				 struct cgroup_subsys_state *css)
+void bio_associate_blkcg_from_css(struct bio *bio,
+				  struct cgroup_subsys_state *css)
 {
-	if (bio->bi_blkg)
-		blkg_put(bio->bi_blkg);
+	struct blkcg *blkcg;
 
-	if (css && css->parent) {
-		bio->bi_blkg = blkg_tryget_closest(bio, css);
-	} else {
-		blkg_get(bdev_get_queue(bio->bi_bdev)->root_blkg);
-		bio->bi_blkg = bdev_get_queue(bio->bi_bdev)->root_blkg;
+	if (!css || !css_tryget_online(css)) {
+		css = &blkcg_root.css;
+		css_get(css);
 	}
+
+	blkcg = css_to_blkcg(css);
+	if (bio_blkcg(bio) == blkcg) {
+		css_put(css);
+		return;
+	}
+
+	bio_clear_blkcg(bio);
+	bio->bi_blkcg = blkcg;
 }
-EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css);
+EXPORT_SYMBOL_GPL(bio_associate_blkcg_from_css);
 
 /**
- * bio_associate_blkg - associate a bio with a blkg
+ * bio_associate_blkcg - associate a bio with a blkcg
  * @bio: target bio
  *
- * Associate @bio with the blkg found from the bio's css and request_queue.
- * If one is not found, bio_lookup_blkg() creates the blkg.  If a blkg is
- * already associated, the css is reused and association redone as the
- * request_queue may have changed.
+ * Associate @bio with the blkcg found from the bio's css.  If a blkcg is
+ * already associated, keep it as blkcg association is not queue-local.
  */
-void bio_associate_blkg(struct bio *bio)
+void bio_associate_blkcg(struct bio *bio)
 {
-	struct cgroup_subsys_state *css;
-
 	if (blk_op_is_passthrough(bio->bi_opf))
 		return;
 
-	if (bio->bi_blkg) {
-		css = bio_blkcg_css(bio);
-		bio_associate_blkg_from_css(bio, css);
-	} else {
-		rcu_read_lock();
-		css = blkcg_css();
-		if (!css_tryget_online(css))
-			css = NULL;
-		rcu_read_unlock();
+	if (bio_blkcg(bio))
+		return;
 
-		bio_associate_blkg_from_css(bio, css);
-		if (css)
-			css_put(css);
-	}
+	rcu_read_lock();
+	bio_associate_blkcg_from_css(bio, blkcg_css());
+	rcu_read_unlock();
 }
-EXPORT_SYMBOL_GPL(bio_associate_blkg);
+EXPORT_SYMBOL_GPL(bio_associate_blkcg);
 
 /**
- * bio_clone_blkg_association - clone blkg association from src to dst bio
+ * bio_clone_blkcg_association - clone blkcg association from src to dst bio
  * @dst: destination bio
  * @src: source bio
  */
-void bio_clone_blkg_association(struct bio *dst, struct bio *src)
+void bio_clone_blkcg_association(struct bio *dst, struct bio *src)
 {
-	if (src->bi_blkg)
-		bio_associate_blkg_from_css(dst, bio_blkcg_css(src));
+	if (bio_blkcg(src))
+		bio_associate_blkcg_from_css(dst, bio_blkcg_css(src));
 }
-EXPORT_SYMBOL_GPL(bio_clone_blkg_association);
+EXPORT_SYMBOL_GPL(bio_clone_blkcg_association);
 
 static int blk_cgroup_io_type(struct bio *bio)
 {
@@ -2199,20 +2247,27 @@ static int blk_cgroup_io_type(struct bio *bio)
 
 void blk_cgroup_bio_start(struct bio *bio)
 {
-	struct blkcg *blkcg = bio->bi_blkg->blkcg;
+	struct blkcg *blkcg = bio_blkcg(bio);
+	struct blkcg_gq *blkg;
 	int rwd = blk_cgroup_io_type(bio), cpu;
 	struct blkg_iostat_set *bis;
 	unsigned long flags;
 
 	if (!cgroup_subsys_on_dfl(io_cgrp_subsys))
 		return;
+	if (!blkcg)
+		return;
 
 	/* Root-level stats are sourced from system-wide IO stats */
 	if (!cgroup_parent(blkcg->css.cgroup))
 		return;
 
+	blkg = bio_blkg_lookup(bio);
+	if (!blkg)
+		return;
+
 	cpu = get_cpu();
-	bis = per_cpu_ptr(bio->bi_blkg->iostat_cpu, cpu);
+	bis = per_cpu_ptr(blkg->iostat_cpu, cpu);
 	flags = u64_stats_update_begin_irqsave(&bis->sync);
 
 	/*
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index e67c698391294..eb76cc38b41ba 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -19,6 +19,8 @@
 #include <linux/kthread.h>
 #include <linux/blk-mq.h>
 #include <linux/llist.h>
+#include <linux/rhashtable.h>
+#include <linux/rcupdate.h>
 #include "blk.h"
 
 struct blkcg_gq;
@@ -56,9 +58,11 @@ struct blkg_iostat_set {
 struct blkcg_gq {
 	/* Pointer to the associated request_queue */
 	struct request_queue		*q;
+	struct rhash_head		q_hash_node;
 	struct list_head		q_node;
 	struct hlist_node		blkcg_node;
 	struct blkcg			*blkcg;
+	int				blkcg_id;
 
 	/* all non-root blkcg_gq's are guaranteed to have access to parent */
 	struct blkcg_gq			*parent;
@@ -73,14 +77,7 @@ struct blkcg_gq {
 	struct blkg_iostat_set		iostat;
 
 	struct blkg_policy_data		*pd[BLKCG_MAX_POLS];
-#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
-	spinlock_t			async_bio_lock;
-	struct bio_list			async_bios;
-#endif
-	union {
-		struct work_struct	async_bio_work;
-		struct work_struct	free_work;
-	};
+	struct work_struct		free_work;
 
 	atomic_t			use_delay;
 	atomic64_t			delay_nsec;
@@ -98,8 +95,6 @@ struct blkcg {
 	/* If there is block congestion on this cgroup. */
 	atomic_t			congestion_count;
 
-	struct radix_tree_root		blkg_tree;
-	struct blkcg_gq	__rcu		*blkg_hint;
 	struct hlist_head		blkg_list;
 
 	struct blkcg_policy_data	*cpd[BLKCG_MAX_POLS];
@@ -111,6 +106,11 @@ struct blkcg {
 	 */
 	struct llist_head __percpu	*lhead;
 
+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
+	spinlock_t			async_bio_lock; /* protects async_bios */
+	struct bio_list			async_bios;
+	struct work_struct		async_bio_work;
+#endif
 #ifdef CONFIG_BLK_CGROUP_FC_APPID
 	char                            fc_app_id[FC_APPID_LEN];
 #endif
@@ -124,6 +124,11 @@ static inline struct blkcg *css_to_blkcg(struct cgroup_subsys_state *css)
 	return css ? container_of(css, struct blkcg, css) : NULL;
 }
 
+static inline struct blkcg *bio_blkcg(struct bio *bio)
+{
+	return bio->bi_blkcg;
+}
+
 /*
  * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a
  * request_queue (q).  This is used by blkcg policies which need to track
@@ -192,8 +197,10 @@ struct blkcg_policy {
 
 extern struct blkcg blkcg_root;
 extern bool blkcg_debug_stats;
+extern const struct rhashtable_params blkg_hash_params;
 
-void blkg_init_queue(struct request_queue *q);
+int blkg_init_queue(struct request_queue *q);
+void blkg_exit_queue(struct request_queue *q);
 int blkcg_init_disk(struct gendisk *disk);
 void blkcg_exit_disk(struct gendisk *disk);
 
@@ -249,11 +256,33 @@ static inline bool bio_issue_as_root_blkg(struct bio *bio)
 }
 
 /**
- * blkg_lookup - lookup blkg for the specified blkcg - q pair
+ * blkg_lookup_any - lookup any blkg for the specified blkcg - q pair
+ * @blkcg: blkcg of interest
+ * @q: request_queue of interest
+ *
+ * Lookup a blkg for the @blkcg - @q pair, whether it is online or dying.
+ *
+ * Must be called in a RCU critical section.
+ *
+ * This does not acquire a reference.  The caller must already hold one, or
+ * have the blkg pinned by I/O.
+ */
+static inline struct blkcg_gq *blkg_lookup_any(struct blkcg *blkcg,
+					       struct request_queue *q)
+{
+	RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
+			 "blkg_lookup_any() requires an RCU read lock");
+
+	return rhashtable_lookup(&q->blkg_hash, &blkcg->css.id,
+				 blkg_hash_params);
+}
+
+/**
+ * blkg_lookup - lookup an online blkg for the specified blkcg - q pair
  * @blkcg: blkcg of interest
  * @q: request_queue of interest
  *
- * Lookup blkg for the @blkcg - @q pair.
+ * Lookup an online blkg for the @blkcg - @q pair.
  *
  * Must be called in a RCU critical section.
  */
@@ -265,17 +294,15 @@ static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg,
 	if (blkcg == &blkcg_root)
 		return q->root_blkg;
 
-	blkg = rcu_dereference_check(blkcg->blkg_hint,
-			lockdep_is_held(&q->queue_lock));
-	if (blkg && blkg->q == q)
-		return blkg;
-
-	blkg = radix_tree_lookup(&blkcg->blkg_tree, q->id);
-	if (blkg && blkg->q != q)
+	blkg = blkg_lookup_any(blkcg, q);
+	if (blkg && !READ_ONCE(blkg->online))
 		blkg = NULL;
 	return blkg;
 }
 
+struct blkcg_gq *bio_blkg_lookup(struct bio *bio);
+struct blkcg_gq *bio_blkg(struct bio *bio);
+
 /**
  * blkg_to_pd - get policy private data
  * @blkg: blkg of interest
@@ -343,6 +370,18 @@ static inline void blkg_put(struct blkcg_gq *blkg)
 	percpu_ref_put(&blkg->refcnt);
 }
 
+static inline void bio_clear_blkcg(struct bio *bio)
+{
+	struct blkcg *blkcg = bio_blkcg(bio);
+
+	bio_put_blkg_ref(bio);
+
+	if (blkcg) {
+		css_put(&blkcg->css);
+		bio->bi_blkcg = NULL;
+	}
+}
+
 /**
  * blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants
  * @d_blkg: loop cursor pointing to the current descendant
@@ -472,7 +511,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq *blkg)
  */
 static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bio)
 {
-	return rq->bio->bi_blkg == bio->bi_blkg &&
+	return bio_blkcg(rq->bio) == bio_blkcg(bio) &&
 		bio_issue_as_root_blkg(rq->bio) == bio_issue_as_root_blkg(bio);
 }
 
@@ -498,8 +537,13 @@ struct blkcg_policy {
 struct blkcg {
 };
 
+static inline struct blkcg *bio_blkcg(struct bio *bio) { return NULL; }
+static inline struct blkcg_gq *bio_blkg_lookup(struct bio *bio) { return NULL; }
+static inline struct blkcg_gq *bio_blkg(struct bio *bio) { return NULL; }
+static inline struct blkcg_gq *blkg_lookup_any(struct blkcg *blkcg, void *key) { return NULL; }
 static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, void *key) { return NULL; }
-static inline void blkg_init_queue(struct request_queue *q) { }
+static inline int blkg_init_queue(struct request_queue *q) { return 0; }
+static inline void blkg_exit_queue(struct request_queue *q) { }
 static inline int blkcg_init_disk(struct gendisk *disk) { return 0; }
 static inline void blkcg_exit_disk(struct gendisk *disk) { }
 static inline int blkcg_policy_register(struct blkcg_policy *pol) { return 0; }
@@ -514,6 +558,7 @@ static inline struct blkg_policy_data *blkg_to_pd(struct blkcg_gq *blkg,
 static inline struct blkcg_gq *pd_to_blkg(struct blkg_policy_data *pd) { return NULL; }
 static inline void blkg_get(struct blkcg_gq *blkg) { }
 static inline void blkg_put(struct blkcg_gq *blkg) { }
+static inline void bio_clear_blkcg(struct bio *bio) { }
 static inline void blk_cgroup_bio_start(struct bio *bio) { }
 static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bio) { return true; }
 
diff --git a/block/blk-core.c b/block/blk-core.c
index 196bccf27f58d..e396a316b8bf6 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -303,6 +303,7 @@ static void blk_free_queue(struct request_queue *q)
 	if (queue_is_mq(q))
 		blk_mq_release(q);
 
+	blkg_exit_queue(q);
 	ida_free(&blk_queue_ida, q->id);
 	lockdep_unregister_key(&q->io_lock_cls_key);
 	lockdep_unregister_key(&q->q_lock_cls_key);
@@ -481,7 +482,9 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)
 	init_waitqueue_head(&q->mq_freeze_wq);
 	mutex_init(&q->mq_freeze_lock);
 
-	blkg_init_queue(q);
+	error = blkg_init_queue(q);
+	if (error)
+		goto fail_stats;
 
 	/*
 	 * Init percpu_ref in atomic mode so that it's faster to shutdown.
@@ -491,7 +494,7 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)
 				blk_queue_usage_counter_release,
 				PERCPU_REF_INIT_ATOMIC, GFP_KERNEL);
 	if (error)
-		goto fail_stats;
+		goto fail_blkg;
 	lockdep_register_key(&q->io_lock_cls_key);
 	lockdep_register_key(&q->q_lock_cls_key);
 	lockdep_init_map(&q->io_lockdep_map, "&q->q_usage_counter(io)",
@@ -510,6 +513,8 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)
 
 	return q;
 
+fail_blkg:
+	blkg_exit_queue(q);
 fail_stats:
 	blk_free_queue_stats(q->stats);
 fail_id:
diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c
index 2a5c52ab74b4c..5ef4baab444bb 100644
--- a/block/blk-crypto-fallback.c
+++ b/block/blk-crypto-fallback.c
@@ -187,7 +187,7 @@ static struct bio *blk_crypto_alloc_enc_bio(struct bio *bio_src,
 	bio->bi_write_hint	= bio_src->bi_write_hint;
 	bio->bi_write_stream	= bio_src->bi_write_stream;
 	bio->bi_iter.bi_sector	= bio_src->bi_iter.bi_sector;
-	bio_clone_blkg_association(bio, bio_src);
+	bio_clone_blkcg_association(bio, bio_src);
 
 	/*
 	 * Move page array up in the allocated memory for the bio vecs as far as
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 2745bffcd5eef..57f2b4d4af20a 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -2686,7 +2686,7 @@ iocg_handle_over_budget(struct rq_qos *rqos, struct ioc_gq *iocg,
 
 static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)
 {
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg_gq *blkg = bio_blkg(bio);
 	struct ioc *ioc = rqos_to_ioc(rqos);
 	struct ioc_gq *iocg = blkg_to_iocg(blkg);
 	struct ioc_now now;
@@ -2775,7 +2775,7 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)
 static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq,
 			   struct bio *bio)
 {
-	struct ioc_gq *iocg = blkg_to_iocg(bio->bi_blkg);
+	struct ioc_gq *iocg = blkg_to_iocg(bio_blkg(bio));
 	struct ioc *ioc = rqos_to_ioc(rqos);
 	sector_t bio_end = bio_end_sector(bio);
 	struct ioc_now now;
@@ -2833,9 +2833,13 @@ static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq,
 
 static void ioc_rqos_done_bio(struct rq_qos *rqos, struct bio *bio)
 {
-	struct ioc_gq *iocg = blkg_to_iocg(bio->bi_blkg);
+	struct ioc_gq *iocg;
+
+	if (!bio->bi_iocost_cost)
+		return;
 
-	if (iocg && bio->bi_iocost_cost)
+	iocg = blkg_to_iocg(bio_blkg_lookup(bio));
+	if (iocg)
 		atomic64_add(bio->bi_iocost_cost, &iocg->done_vtime);
 }
 
diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index 2caa79a008ad1..7220edafd72b6 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -463,7 +463,7 @@ static void check_scale_change(struct iolatency_grp *iolat)
 static void blkcg_iolatency_throttle(struct rq_qos *rqos, struct bio *bio)
 {
 	struct blk_iolatency *blkiolat = BLKIOLATENCY(rqos);
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg_gq *blkg = bio_blkg(bio);
 	bool issue_as_root = bio_issue_as_root_blkg(bio);
 
 	if (!blkiolat->enabled)
@@ -590,11 +590,14 @@ static void blkcg_iolatency_done_bio(struct rq_qos *rqos, struct bio *bio)
 	bool issue_as_root = bio_issue_as_root_blkg(bio);
 	int inflight = 0;
 
-	blkg = bio->bi_blkg;
-	if (!blkg || !bio_flagged(bio, BIO_QOS_THROTTLED))
+	if (!bio_flagged(bio, BIO_QOS_THROTTLED))
 		return;
 
-	iolat = blkg_to_lat(bio->bi_blkg);
+	blkg = bio_blkg_lookup(bio);
+	if (!blkg)
+		return;
+
+	iolat = blkg_to_lat(blkg);
 	if (!iolat)
 		return;
 
diff --git a/block/blk-ioprio.c b/block/blk-ioprio.c
index 8fa8bca350629..5e3f8e49550da 100644
--- a/block/blk-ioprio.c
+++ b/block/blk-ioprio.c
@@ -132,7 +132,7 @@ static struct blkcg_policy ioprio_policy = {
 
 void blkcg_set_ioprio(struct bio *bio)
 {
-	struct ioprio_blkcg *blkcg = blkcg_to_ioprio_blkcg(bio->bi_blkg->blkcg);
+	struct ioprio_blkcg *blkcg = blkcg_to_ioprio_blkcg(bio_blkcg(bio));
 	u16 prio;
 
 	if (!blkcg || blkcg->prio_policy == POLICY_NO_CHANGE)
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index ffc3b70065d4b..3828c38579002 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1792,7 +1792,7 @@ static bool tg_within_limit(struct throtl_grp *tg, struct bio *bio, bool rw)
 bool __blk_throtl_bio(struct bio *bio)
 {
 	struct request_queue *q = bdev_get_queue(bio->bi_bdev);
-	struct blkcg_gq *blkg = bio->bi_blkg;
+	struct blkcg_gq *blkg = bio_blkg(bio);
 	struct throtl_qnode *qn = NULL;
 	struct throtl_grp *tg = blkg_to_tg(blkg);
 	struct throtl_service_queue *sq;
diff --git a/block/blk-throttle.h b/block/blk-throttle.h
index 1b5775771e078..45f30179fd9c0 100644
--- a/block/blk-throttle.h
+++ b/block/blk-throttle.h
@@ -171,7 +171,7 @@ static inline bool blk_should_throtl(struct bio *bio)
 	if (!blk_throtl_activated(bio->bi_bdev->bd_queue))
 		return false;
 
-	tg = blkg_to_tg(bio->bi_blkg);
+	tg = blkg_to_tg(bio_blkg(bio));
 	if (!cgroup_subsys_on_dfl(io_cgrp_subsys)) {
 		if (!bio_flagged(bio, BIO_CGROUP_ACCT)) {
 			bio_set_flag(bio, BIO_CGROUP_ACCT);
diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
index 3fa3b13a410f4..c0f945b8d9413 100644
--- a/drivers/md/bcache/request.c
+++ b/drivers/md/bcache/request.c
@@ -848,7 +848,7 @@ static CLOSURE_CALLBACK(cached_dev_read_done)
 		s->iop.bio->bi_iter.bi_sector =
 			s->cache_miss->bi_iter.bi_sector;
 		s->iop.bio->bi_iter.bi_size = s->insert_bio_sectors << 9;
-		bio_clone_blkg_association(s->iop.bio, s->cache_miss);
+		bio_clone_blkcg_association(s->iop.bio, s->cache_miss);
 		bch_bio_map(s->iop.bio, NULL);
 
 		bio_copy_data(s->cache_miss, s->iop.bio);
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index d413bfaf35278..cd68eec77f5a1 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1373,7 +1373,7 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone)
 	if (!tgt_clone)
 		tgt_clone = clone;
 
-	bio_clone_blkg_association(tgt_clone, io->orig_bio);
+	bio_clone_blkcg_association(tgt_clone, io->orig_bio);
 
 	/*
 	 * Account io->origin_bio to DM dev on behalf of target
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 680b34a63cb3d..fd78d6cc8da17 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -9386,7 +9386,7 @@ void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,
 		return;
 
 	bio_chain(discard_bio, bio);
-	bio_clone_blkg_association(discard_bio, bio);
+	bio_clone_blkcg_association(discard_bio, bio);
 	mddev_trace_remap(mddev, discard_bio, bio->bi_iter.bi_sector);
 	submit_bio_noacct(discard_bio);
 }
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 4176046627beb..54e4adb0ccb7d 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -121,7 +121,7 @@ int async_pmem_flush(struct nd_region *nd_region, struct bio *bio)
 
 		if (!child)
 			return -ENOMEM;
-		bio_clone_blkg_association(child, bio);
+		bio_clone_blkcg_association(child, bio);
 		child->bi_iter.bi_sector = -1;
 		bio_chain(child, bio);
 		submit_bio(child);
diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c
index 6dabe73ad790d..6512dbd9516fc 100644
--- a/fs/gfs2/lops.c
+++ b/fs/gfs2/lops.c
@@ -484,7 +484,7 @@ static struct bio *gfs2_chain_bio(struct bio *prev, unsigned int nr_iovecs,
 	struct bio *new;
 
 	new = bio_alloc(prev->bi_bdev, nr_iovecs, opf, GFP_NOIO);
-	bio_clone_blkg_association(new, prev);
+	bio_clone_blkcg_association(new, prev);
 	new->bi_iter.bi_sector = sector;
 	bio_chain(new, prev);
 	submit_bio(prev);
@@ -1114,4 +1114,3 @@ const struct gfs2_log_operations *gfs2_log_ops[] = {
 	&gfs2_revoke_lops,
 	NULL,
 };
-
diff --git a/include/linux/bio.h b/include/linux/bio.h
index bb3235497e673..11637f85c38d4 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -542,18 +542,20 @@ static inline void bio_release_pages(struct bio *bio, bool mark_dirty)
 	disk_devt((bio)->bi_bdev->bd_disk)
 
 #ifdef CONFIG_BLK_CGROUP
-void bio_associate_blkg(struct bio *bio);
-void bio_associate_blkg_from_css(struct bio *bio,
-				 struct cgroup_subsys_state *css);
-void bio_clone_blkg_association(struct bio *dst, struct bio *src);
+void bio_associate_blkcg(struct bio *bio);
+void bio_associate_blkcg_from_css(struct bio *bio,
+				  struct cgroup_subsys_state *css);
+void bio_clone_blkcg_association(struct bio *dst, struct bio *src);
+void bio_put_blkg_ref(struct bio *bio);
 void blkcg_punt_bio_submit(struct bio *bio);
 #else	/* CONFIG_BLK_CGROUP */
-static inline void bio_associate_blkg(struct bio *bio) { }
-static inline void bio_associate_blkg_from_css(struct bio *bio,
-					       struct cgroup_subsys_state *css)
+static inline void bio_associate_blkcg(struct bio *bio) { }
+static inline void bio_associate_blkcg_from_css(struct bio *bio,
+						struct cgroup_subsys_state *css)
 { }
-static inline void bio_clone_blkg_association(struct bio *dst,
-					      struct bio *src) { }
+static inline void bio_clone_blkcg_association(struct bio *dst,
+					       struct bio *src) { }
+static inline void bio_put_blkg_ref(struct bio *bio) { }
 static inline void blkcg_punt_bio_submit(struct bio *bio)
 {
 	submit_bio(bio);
@@ -563,10 +565,12 @@ static inline void blkcg_punt_bio_submit(struct bio *bio)
 static inline void bio_set_dev(struct bio *bio, struct block_device *bdev)
 {
 	bio_clear_flag(bio, BIO_REMAPPED);
-	if (bio->bi_bdev != bdev)
+	if (bio->bi_bdev != bdev) {
+		bio_put_blkg_ref(bio);
 		bio_clear_flag(bio, BIO_BPS_THROTTLED);
+	}
 	bio->bi_bdev = bdev;
-	bio_associate_blkg(bio);
+	bio_associate_blkcg(bio);
 }
 
 /*
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index d49d97a050d0e..53ba307b91cb3 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -246,12 +246,10 @@ struct bio {
 	void			*bi_private;
 #ifdef CONFIG_BLK_CGROUP
 	/*
-	 * Represents the association of the css and request_queue for the bio.
-	 * If a bio goes direct to device, it will not have a blkg as it will
-	 * not have a request_queue associated with it.  The reference is put
-	 * on release of the bio.
+	 * Represents the blkcg css association for the bio.  The reference is
+	 * put on release of the bio.
 	 */
-	struct blkcg_gq		*bi_blkg;
+	struct blkcg		*bi_blkcg;
 	/* Time that this bio was issued. */
 	u64			issue_time_ns;
 #ifdef CONFIG_BLK_CGROUP_IOCOST
@@ -309,6 +307,7 @@ enum {
 	BIO_TRACE_COMPLETION,	/* bio_endio() should trace the final completion
 				 * of this bio. */
 	BIO_CGROUP_ACCT,	/* has been accounted to a cgroup */
+	BIO_BLKG_REF,		/* bio pins the associated blkg */
 	BIO_QOS_THROTTLED,	/* bio went through rq_qos throttle path */
 	/*
 	 * This bio has completed bps throttling at the single tg granularity,
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 20cb8ed7d987a..f3b0e39d3f6b4 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -27,6 +27,7 @@
 #include <linux/xarray.h>
 #include <linux/file.h>
 #include <linux/lockdep.h>
+#include <linux/rhashtable-types.h>
 
 struct module;
 struct request_queue;
@@ -581,6 +582,7 @@ struct request_queue {
 #ifdef CONFIG_BLK_CGROUP
 	DECLARE_BITMAP		(blkcg_pols, BLKCG_MAX_POLS);
 	struct blkcg_gq		*root_blkg;
+	struct rhashtable	blkg_hash;
 	struct list_head	blkg_list;
 	struct mutex		blkcg_mutex;
 #endif
diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index 62552a2ce5b9e..4f869fe9cc90f 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -262,7 +262,7 @@ static inline void wbc_init_bio(struct writeback_control *wbc, struct bio *bio)
 	 * regular writeback instead of writing things out itself.
 	 */
 	if (wbc->wb)
-		bio_associate_blkg_from_css(bio, wbc->wb->blkcg_css);
+		bio_associate_blkcg_from_css(bio, wbc->wb->blkcg_css);
 }
 
 void inode_switch_wbs_work_fn(struct work_struct *work);
diff --git a/mm/page_io.c b/mm/page_io.c
index b23f494fcc83d..a4a1b8a8a0e38 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -316,7 +316,7 @@ static inline void count_swpout_vm_event(struct folio *folio)
 }
 
 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
-static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)
+static void bio_associate_blkcg_from_page(struct bio *bio, struct folio *folio)
 {
 	struct cgroup_subsys_state *css;
 	struct mem_cgroup *memcg;
@@ -331,12 +331,15 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)
 		css = NULL;
 	rcu_read_unlock();
 
-	bio_associate_blkg_from_css(bio, css);
+	bio_associate_blkcg_from_css(bio, css);
 	if (css)
 		css_put(css);
 }
 #else
-#define bio_associate_blkg_from_page(bio, folio)		do { } while (0)
+static inline void bio_associate_blkcg_from_page(struct bio *bio,
+						 struct folio *folio)
+{
+}
 #endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */
 
 struct swap_iocb {
@@ -436,7 +439,7 @@ static void swap_writepage_bdev_sync(struct folio *folio,
 	bio.bi_iter.bi_sector = swap_folio_sector(folio);
 	bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
 
-	bio_associate_blkg_from_page(&bio, folio);
+	bio_associate_blkcg_from_page(&bio, folio);
 	count_swpout_vm_event(folio);
 
 	folio_start_writeback(folio);
@@ -456,7 +459,7 @@ static void swap_writepage_bdev_async(struct folio *folio,
 	bio->bi_end_io = end_swap_bio_write;
 	bio_add_folio_nofail(bio, folio, folio_size(folio), 0);
 
-	bio_associate_blkg_from_page(bio, folio);
+	bio_associate_blkcg_from_page(bio, folio);
 	count_swpout_vm_event(folio);
 	folio_start_writeback(folio);
 	folio_unlock(folio);

gemini-3.1-pro-preview

							input: 21142
							output: 279
							thoughts: 1171

**KMSAN Assessment: Block Cgroup Association Refactor**

Okay, so I've got this patch series that's overhauling how `bio` structures relate to block cgroups, a common area for performance and isolation concerns. It looks like the main thrust is to move from a direct `bio->bi_blkg` pointer to a `bio->bi_blkcg` pointer and use a hash table (`q->blkg_hash`) to manage the `blkcg_gq` (now `blkg`) lookups, instead of that radix tree. This touches a lot of code, from the block layer down to cgroups and even MD, bcache, and swap, so we need to be very careful. My primary concern, as always, is memory safety and potential leaks.

First, I checked for uninitialized memory risks. The diffs show that `bio->bi_blkcg` *is* properly initialized to `NULL` in `bio_init()`. Then `bio_clear_blkcg()` handles the cleanup by decrementing the refcounts of the `blkcg` and `blkg`. The `blkg_hash` is initialized in `blkg_init_queue()` during queue setup. `blkcg_gq` structures are allocated with `kzalloc_node`, so they're zeroed as expected. I'm not seeing any new structures being copied to userspace, no uninitialized variables used in conditionals, and no complex data structures allocated without proper initialization. So, from the initial overview, it looks pretty safe.

Next, I specifically looked for KMSAN-related concerns. KMSAN (Kernel Memory Sanitizer) is designed to flag uninitialized memory reads. This patch mostly changes pointers and uses a hash table for lookups, as well as introducing a `BIO_BLKG_REF` flag to the `bio->bi_flags` structure. Since we're just shifting how we manage the association between `bio` and cgroups, I can't see any new avenues for leaking uninitialized kernel memory or reading from it. The existing allocations are handled correctly.

So, the conclusion is: This patch series should *not* require a dedicated KMSAN fuzzing session. The changes are largely structural refactorings concerning how the block cgroup association is managed within the kernel. The introduction of the `bi_blkcg` pointer, the hash table for lookups, and the modified initialization/cleanup routines are sound, and memory is properly initialized and cleaned up. Potential bugs would most likely appear as use-after-free conditions, memory leaks or NULL pointer dereferences that would be caught by existing KASAN and other tools. There's no indication of any uninitialized memory reads or kernel information leaking to userspace due to these changes. The `set-results` tool should handle the final outcome based on this assessment.

{
  "KMSANReasoning": "The patch refactors the block cgroup (blkcg) association for `bio` structures, replacing the direct `bi_blkg` pointer with `bi_blkcg` and dynamically looking up or pinning the queue-local `blkcg_gq` (blkg) using a hash table instead of a radix tree.\n\nThese changes involve modifying internal kernel structures (`bio`, `request_queue`, `blkcg_gq`) and their initialization/cleanup routines (`bio_init`, `bio_uninit`, `bio_clear_blkcg`). No new structures are copied to user space, nor are there any new buffers or complex data structures allocated without proper initialization. The modified fields in `bio` are explicitly initialized to `NULL` or appropriate values in `bio_init()`.\n\nAny bugs introduced by these changes would likely manifest as NULL pointer dereferences, use-after-free (e.g., incorrect reference counting of `blkcg` or `blkg`), or memory leaks, all of which are effectively caught by standard KASAN and other default debugging tools. There is no risk of uninitialized memory reads or kernel-to-user info-leaks that would necessitate a dedicated KMSAN fuzzing session.",
  "NeedsKMSAN": false
}

{
  "KMSANReasoning": "The patch refactors the block cgroup (blkcg) association for `bio` structures, replacing the direct `bi_blkg` pointer with `bi_blkcg` and dynamically looking up or pinning the queue-local `blkcg_gq` (blkg) using a hash table instead of a radix tree.\n\nThese changes involve modifying internal kernel structures (`bio`, `request_queue`, `blkcg_gq`) and their initialization/cleanup routines (`bio_init`, `bio_uninit`, `bio_clear_blkcg`). No new structures are copied to user space, nor are there any new buffers or complex data structures allocated without proper initialization. The modified fields in `bio` are explicitly initialized to `NULL` or appropriate values in `bio_init()`.\n\nAny bugs introduced by these changes would likely manifest as NULL pointer dereferences, use-after-free (e.g., incorrect reference counting of `blkcg` or `blkg`), or memory leaks, all of which are effectively caught by standard KASAN and other default debugging tools. There is no risk of uninitialized memory reads or kernel-to-user info-leaks that would necessitate a dedicated KMSAN fuzzing session.",
  "NeedsKMSAN": false
}

LLM Calls Summary:
Total Calls	Total Tokens	Avg Tokens	Total Duration (Seconds)	Avg Duration (Seconds)

Tool Calls Summary:
Total Calls	Total Duration (Seconds)	Avg Duration (Seconds)