A `struct page` has a reference count, and the lifetime of most pages is
managed by it. Kernel C code routinely takes its own references to such
pages, for example when a page is inserted into a VMA with
`vm_insert_page()`.
However, not every page is managed by its reference count, and not every
page may have references taken. For instance, a frozen page has a
reference count of zero, and `get_page()` must not be used on pages owned
by the slab allocator. Thus, a single `Page` type cannot represent both
pages that are known to be reference counted and pages for which this is
not known.
Make `Page` generic over a sealed `PageKind`, and add two page kinds:
- `Base` is the base kind. Nothing is known about a `Page` beyond
the validity of the `struct page`.
- `RefCounted` is the default kind. A `Page` is managed by
the page reference count, and taking additional references to it is
permitted.
Implement the `RefCounted` trait for `Page`, backed by
`get_page()` and `put_page()`, mark it `AlwaysRefCounted`, and return
`ARef` from `Page::alloc_page()`. The page is freed when the last
reference to it is dropped; for the order-0 pages allocated here, this is
equivalent to `__free_pages()`.
The `RefCounted` page kind shares its name with the `RefCounted` trait.
The name matches the mm term for a page that is not frozen, and the kind
is the default, so it is rarely spelled out. In `page.rs`, the trait is
referred to as `aref::RefCounted`.
Operations that are valid for any page are available for all page
kinds.
Let `Page` dereference to `Page`. The purpose of the
dereference is coercion: a `&Page` can be passed to code
that takes a `&Page`.
This also allows pages to be handed out as plain references without
owning the `struct page`. Remove `BorrowedPage` and update users:
`Vmalloc::to_page()` and `AsPageIter` hand out `&Page`, and binder
uses `&Page` and `ARef`.
Signed-off-by: Andreas Hindborg
---
drivers/android/binder/page_range.rs | 8 +-
rust/helpers/page.c | 10 ++
rust/kernel/alloc/allocator.rs | 26 ++---
rust/kernel/alloc/allocator/iter.rs | 7 +-
rust/kernel/page.rs | 195 ++++++++++++++++++++---------------
5 files changed, 142 insertions(+), 104 deletions(-)
diff --git a/drivers/android/binder/page_range.rs b/drivers/android/binder/page_range.rs
index b0c797523bc2..451369e07e7f 100644
--- a/drivers/android/binder/page_range.rs
+++ b/drivers/android/binder/page_range.rs
@@ -198,7 +198,7 @@ unsafe impl Send for Inner {}
#[repr(C)]
struct PageInfo {
lru: bindings::list_head,
- page: Option,
+ page: Option>,
range: *const ShrinkablePageRange,
}
@@ -206,7 +206,7 @@ impl PageInfo {
/// # Safety
///
/// The caller ensures that writing to `me.page` is ok, and that the page is not currently set.
- unsafe fn set_page(me: *mut PageInfo, page: Page) {
+ unsafe fn set_page(me: *mut PageInfo, page: ARef) {
// SAFETY: This pointer offset is in bounds.
let ptr = unsafe { &raw mut (*me).page };
@@ -229,13 +229,13 @@ unsafe fn get_page<'a>(me: *const PageInfo) -> Option<&'a Page> {
let ptr = unsafe { &raw const (*me).page };
// SAFETY: The pointer is valid for reading.
- unsafe { (*ptr).as_ref() }
+ unsafe { (*ptr).as_deref() }
}
/// # Safety
///
/// The caller ensures that writing to `me.page` is ok for the duration of 'a.
- unsafe fn take_page(me: *mut PageInfo) -> Option {
+ unsafe fn take_page(me: *mut PageInfo) -> Option> {
// SAFETY: This pointer offset is in bounds.
let ptr = unsafe { &raw mut (*me).page };
diff --git a/rust/helpers/page.c b/rust/helpers/page.c
index f8463fbed2a2..906eb4b18f02 100644
--- a/rust/helpers/page.c
+++ b/rust/helpers/page.c
@@ -10,6 +10,16 @@ __rust_helper struct page *rust_helper_alloc_pages(gfp_t gfp_mask,
return alloc_pages(gfp_mask, order);
}
+__rust_helper void rust_helper_get_page(struct page *page)
+{
+ get_page(page);
+}
+
+__rust_helper void rust_helper_put_page(struct page *page)
+{
+ put_page(page);
+}
+
__rust_helper void *rust_helper_kmap_local_page(struct page *page)
{
return kmap_local_page(page);
diff --git a/rust/kernel/alloc/allocator.rs b/rust/kernel/alloc/allocator.rs
index cd4203f27aed..bf20714156c5 100644
--- a/rust/kernel/alloc/allocator.rs
+++ b/rust/kernel/alloc/allocator.rs
@@ -17,7 +17,10 @@
use crate::{
bindings,
- page, //
+ page::{
+ Base,
+ Page, //
+ }, //
};
use core::{
@@ -169,7 +172,7 @@ unsafe fn realloc(
}
impl Vmalloc {
- /// Convert a pointer to a [`Vmalloc`] allocation to a [`page::BorrowedPage`].
+ /// Convert a pointer to a [`Vmalloc`] allocation to a [`Page`] reference.
///
/// # Examples
///
@@ -202,20 +205,19 @@ impl Vmalloc {
///
/// - `ptr` must be a valid pointer to a [`Vmalloc`] allocation.
/// - `ptr` must remain valid for the entire duration of `'a`.
- pub unsafe fn to_page<'a>(ptr: NonNull) -> page::BorrowedPage<'a> {
+ pub unsafe fn to_page<'a>(ptr: NonNull) -> &'a Page {
// SAFETY: `ptr` is a valid pointer to `Vmalloc` memory.
let page = unsafe { bindings::vmalloc_to_page(ptr.as_ptr().cast()) };
- // SAFETY: `vmalloc_to_page` returns a valid pointer to a `struct page` for a valid pointer
- // to `Vmalloc` memory.
- let page = unsafe { NonNull::new_unchecked(page) };
-
+ // INVARIANT: `Base` has no invariants in addition to the validity of the `struct page`,
+ // which is argued below.
// SAFETY:
- // - `page` is a valid pointer to a `struct page`, given that by the safety requirements of
- // this function `ptr` is a valid pointer to a `Vmalloc` allocation.
- // - By the safety requirements of this function `ptr` is valid for the entire lifetime of
- // `'a`.
- unsafe { page::BorrowedPage::from_raw(page) }
+ // - `vmalloc_to_page` returns a valid, non-null pointer to a `struct page` for a valid
+ // pointer to `Vmalloc` memory, given that by the safety requirements of this function
+ // `ptr` is a valid pointer to a `Vmalloc` allocation.
+ // - By the safety requirements of this function `ptr`, and hence the `struct page`, is
+ // valid for the entire lifetime of `'a`.
+ unsafe { &*page.cast() }
}
}
diff --git a/rust/kernel/alloc/allocator/iter.rs b/rust/kernel/alloc/allocator/iter.rs
index 02fda3ea5cae..7c69ef31fb0c 100644
--- a/rust/kernel/alloc/allocator/iter.rs
+++ b/rust/kernel/alloc/allocator/iter.rs
@@ -9,7 +9,8 @@
ptr::NonNull, //
};
-/// An [`Iterator`] of [`page::BorrowedPage`] items owned by a [`Vmalloc`] allocation.
+/// An [`Iterator`] of [`Page`](page::Page) references to the pages of a [`Vmalloc`]
+/// allocation.
///
/// # Guarantees
///
@@ -28,11 +29,11 @@ pub struct VmallocPageIter<'a> {
size: usize,
/// The current page index of the [`Iterator`].
index: usize,
- _p: PhantomData>,
+ _p: PhantomData<&'a page::Page>,
}
impl<'a> Iterator for VmallocPageIter<'a> {
- type Item = page::BorrowedPage<'a>;
+ type Item = &'a page::Page;
fn next(&mut self) -> Option {
let offset = self.index.checked_mul(page::PAGE_SIZE)?;
diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index 1c0796ea229f..b2a17c86e7de 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -12,11 +12,16 @@
code::*,
Result, //
},
+ sync::aref::{
+ self,
+ ARef,
+ AlwaysRefCounted, //
+ },
+ types::Opaque,
uaccess::UserSliceReader, //
};
use core::{
marker::PhantomData,
- mem::ManuallyDrop,
ops::Deref,
ptr::{
self,
@@ -65,104 +70,106 @@ pub const fn page_align(addr: usize) -> Option {
Some(sum & PAGE_MASK)
}
-/// Representation of a non-owning reference to a [`Page`].
-///
-/// This type provides a borrowed version of a [`Page`] that is owned by some other entity, e.g. a
-/// [`Vmalloc`] allocation such as [`VBox`].
-///
-/// # Example
-///
-/// ```
-/// # use kernel::{bindings, prelude::*};
-/// use kernel::page::{BorrowedPage, Page, PAGE_SIZE};
-/// # use core::{mem::MaybeUninit, ptr, ptr::NonNull };
+/// Trait to be implemented by types which provide an [`Iterator`] of [`Page`] references,
+/// such as [`VmallocPageIter`](kernel::alloc::allocator::VmallocPageIter).
+pub trait AsPageIter {
+ /// The [`Iterator`] type, e.g. [`VmallocPageIter`](kernel::alloc::allocator::VmallocPageIter).
+ type Iter<'a>: Iterator- >
+ where
+ Self: 'a;
+
+ /// Returns an [`Iterator`] of [`Page`] references over all pages owned by `self`.
+ fn page_iter(&mut self) -> Self::Iter<'_>;
+}
+
+mod private {
+ pub trait Sealed {}
+}
+
+/// Marker trait for the kinds of a [`Page`].
///
-/// fn borrow_page<'a>(vbox: &'a mut VBox>) -> BorrowedPage<'a> {
-/// let ptr = ptr::from_ref(&**vbox);
+/// A page kind describes what is known about a `struct page`, and thereby what a [`Page`] of that
+/// kind can be used for. See [`Base`] and [`RefCounted`].
///
-/// // SAFETY: `ptr` is a valid pointer to `Vmalloc` memory.
-/// let page = unsafe { bindings::vmalloc_to_page(ptr.cast()) };
+/// A [`Page`] of any other kind dereferences to a [`Page`] of the base kind [`Base`]. The
+/// purpose of the dereference is coercion: a reference to a page of any kind can be passed to code
+/// that takes a `&Page`.
+pub trait PageKind: private::Sealed {}
+
+/// The base [`PageKind`]: a valid page with no further guarantees.
///
-/// // SAFETY: `vmalloc_to_page` returns a valid pointer to a `struct page` for a valid
-/// // pointer to `Vmalloc` memory.
-/// let page = unsafe { NonNull::new_unchecked(page) };
+/// Nothing is known about how the allocation of a [`Page`] is managed. In particular,
+/// the page is not necessarily managed by the page reference count. For instance, pages owned by
+/// the slab allocator are not. Thus, it is not possible to obtain an [`ARef`] from a
+/// `&Page`.
///
-/// // SAFETY:
-/// // - `self.0` is a valid pointer to a `struct page`.
-/// // - `self.0` is valid for the entire lifetime of `self`.
-/// unsafe { BorrowedPage::from_raw(page) }
-/// }
+/// # Invariants
///
-/// let mut vbox = VBox::<[u8; PAGE_SIZE]>::new_uninit(GFP_KERNEL)?;
-/// let page = borrow_page(&mut vbox);
+/// A [`Page`] has no invariants in addition to the ones of [`Page`].
+pub struct Base;
+
+impl private::Sealed for Base {}
+impl PageKind for Base {}
+
+/// The default [`PageKind`]: a page that is managed by the page reference count.
///
-/// // SAFETY: There is no concurrent read or write to this page.
-/// unsafe { page.fill_zero_raw(0, PAGE_SIZE)? };
-/// # Ok::<(), Error>(())
-/// ```
+/// A [`Page`] is accessed through a shared reference or through an owning
+/// [`ARef`]. The page allocation is freed when the last reference to it is dropped.
///
/// # Invariants
///
-/// The borrowed underlying pointer to a `struct page` is valid for the entire lifetime `'a`.
+/// For a [`Page`]:
///
-/// [`VBox`]: kernel::alloc::VBox
-/// [`Vmalloc`]: kernel::alloc::allocator::Vmalloc
-pub struct BorrowedPage<'a>(ManuallyDrop, PhantomData<&'a Page>);
-
-impl<'a> BorrowedPage<'a> {
- /// Constructs a [`BorrowedPage`] from a raw pointer to a `struct page`.
- ///
- /// # Safety
- ///
- /// - `ptr` must point to a valid `bindings::page`.
- /// - `ptr` must remain valid for the entire lifetime `'a`.
- pub unsafe fn from_raw(ptr: NonNull) -> Self {
- let page = Page { page: ptr };
-
- // INVARIANT: The safety requirements guarantee that `ptr` is valid for the entire lifetime
- // `'a`.
- Self(ManuallyDrop::new(page), PhantomData)
- }
-}
+/// - The allocation of the page is managed by the page reference count, and the reference count is
+/// non-zero.
+/// - It is permitted to take additional references to the page with `get_page()`.
+pub struct RefCounted;
-impl<'a> Deref for BorrowedPage<'a> {
- type Target = Page;
+impl private::Sealed for RefCounted {}
+impl PageKind for RefCounted {}
- fn deref(&self) -> &Self::Target {
- &self.0
- }
-}
-
-/// Trait to be implemented by types which provide an [`Iterator`] implementation of
-/// [`BorrowedPage`] items, such as [`VmallocPageIter`](kernel::alloc::allocator::VmallocPageIter).
-pub trait AsPageIter {
- /// The [`Iterator`] type, e.g. [`VmallocPageIter`](kernel::alloc::allocator::VmallocPageIter).
- type Iter<'a>: Iterator
- >
- where
- Self: 'a;
-
- /// Returns an [`Iterator`] of [`BorrowedPage`] items over all pages owned by `self`.
- fn page_iter(&mut self) -> Self::Iter<'_>;
-}
-
-/// A pointer to a page that owns the page allocation.
+/// A `struct page`.
+///
+/// A `Page` is generic over a [`PageKind`], which describes what is known about the page:
+///
+/// - `Page` is a page for which nothing is known beyond its validity.
+/// - `Page`, or just `Page`, is a page that is managed by the page reference count.
+///
+/// Operations that are valid for any page are available for all page kinds. All other page kinds
+/// dereference to [`Page`], so that a reference to a page of any kind coerces to a
+/// `&Page`.
///
/// # Invariants
///
-/// The pointer is valid, and has ownership over the page.
-pub struct Page {
- page: NonNull,
+/// - The `Page` is backed by a valid `struct page`.
+/// - The invariants of `Kind` hold for the page, see the documentation of the [`PageKind`] types.
+#[repr(transparent)]
+pub struct Page {
+ page: Opaque,
+ _kind: PhantomData,
}
// SAFETY: Pages have no logic that relies on them staying on a given thread, so moving them across
// threads is safe.
-unsafe impl Send for Page {}
+unsafe impl Send for Page {}
// SAFETY: Pages have no logic that relies on them not being accessed concurrently, so accessing
// them concurrently is safe.
-unsafe impl Sync for Page {}
+unsafe impl Sync for Page {}
-impl Page {
+impl Deref for Page {
+ type Target = Page;
+
+ #[inline]
+ fn deref(&self) -> &Self::Target {
+ // SAFETY: `Page` is transparent over `struct page` for all page kinds, so the pointer cast
+ // is valid. `Base` has no invariants in addition to the ones of `Page`, which hold for
+ // `self`.
+ unsafe { &*ptr::from_ref(self).cast() }
+ }
+}
+
+impl Page {
/// Allocates a new page.
///
/// # Examples
@@ -185,19 +192,22 @@ impl Page {
/// # Ok::<(), kernel::alloc::AllocError>(())
/// ```
#[inline]
- pub fn alloc_page(flags: Flags) -> Result {
+ pub fn alloc_page(flags: Flags) -> Result, AllocError> {
// SAFETY: Depending on the value of `gfp_flags`, this call may sleep. Other than that, it
// is always safe to call this method.
let page = unsafe { bindings::alloc_pages(flags.as_raw(), 0) };
let page = NonNull::new(page).ok_or(AllocError)?;
- // INVARIANT: We just successfully allocated a page, so we now have ownership of the newly
- // allocated page. We transfer that ownership to the new `Page` object.
- Ok(Self { page })
+ // SAFETY: We just successfully allocated a page, so we own the reference count returned
+ // by `alloc_pages`, and we transfer that reference to the new `ARef`. Since `Page`
+ // is transparent, we can cast the pointer directly.
+ Ok(unsafe { ARef::from_raw(page.cast()) })
}
+}
+impl Page {
/// Returns a raw pointer to the page.
pub fn as_ptr(&self) -> *mut bindings::page {
- self.page.as_ptr()
+ self.page.get()
}
/// Get the node id containing this page.
@@ -373,10 +383,25 @@ pub unsafe fn copy_from_user_slice_raw(
}
}
-impl Drop for Page {
+// SAFETY: By the type invariant, a `Page` is managed by the page reference count.
+// Incrementing it via `get_page()` keeps the page alive at least until a matching `put_page()`,
+// which frees the page when the count reaches zero.
+unsafe impl aref::RefCounted for Page {
+ #[inline]
+ fn inc_ref(&self) {
+ // SAFETY: By the type invariant, `self.as_ptr()` points to a valid `struct page` with a
+ // non-zero reference count.
+ unsafe { bindings::get_page(self.as_ptr()) };
+ }
+
#[inline]
- fn drop(&mut self) {
- // SAFETY: By the type invariants, we have ownership of the page and can free it.
- unsafe { bindings::__free_pages(self.page.as_ptr(), 0) };
+ unsafe fn dec_ref(obj: NonNull) {
+ // SAFETY: By the function safety requirements, the caller relinquishes a reference count
+ // on the page. Since `Page` is transparent, we can cast the pointer directly.
+ unsafe { bindings::put_page(obj.cast().as_ptr()) };
}
}
+
+// SAFETY: All instances of `Page` are reference counted, so a `&Page` proves that the refcount is
+// non-zero and it is okay to obtain an `ARef` from it.
+unsafe impl AlwaysRefCounted for Page {}
--
2.54.0