Add the page kind `Exclusive`. A `Page` is a page that is uniquely owned through an `Owned>`, with the added invariant that the page data area does not incur data races. This means a `Page` cannot be mapped to user space or shared with devices, and it becomes simpler to directly reference the contents of the page. Back `Page` by a frozen page, allocated with `alloc_frozen_pages()` and freed with `free_frozen_pages()`. The reference count of a frozen page is zero and references to it cannot be taken, so the exclusive ownership is upheld by mm and not only by the Rust type system. Add `Page::alloc_exclusive()` to allocate a `Page`. The function is not named `alloc_page()`, because inherent functions of the same name on two page kinds would make the call `Page::alloc_page()` ambiguous. `Page` dereferences to `Page`. This does not allow safe code to break the aliasing invariant of `Page`: `Page` does not implement `AlwaysRefCounted`, so no `ARef` to the page can be obtained, and the raw data accessors are unsafe and already require their callers to rule out races with all other accesses to the page. Suggested-by: Matthew Wilcox Signed-off-by: Andreas Hindborg --- rust/helpers/page.c | 6 ++++ rust/kernel/page.rs | 96 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 100 insertions(+), 2 deletions(-) diff --git a/rust/helpers/page.c b/rust/helpers/page.c index 906eb4b18f02..8f9f624ea141 100644 --- a/rust/helpers/page.c +++ b/rust/helpers/page.c @@ -10,6 +10,12 @@ __rust_helper struct page *rust_helper_alloc_pages(gfp_t gfp_mask, return alloc_pages(gfp_mask, order); } +__rust_helper struct page *rust_helper_alloc_frozen_pages(gfp_t gfp_mask, + unsigned int order) +{ + return alloc_frozen_pages(gfp_mask, order); +} + __rust_helper void rust_helper_get_page(struct page *page) { get_page(page); diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs index 2ceeac0237d5..c885bcf4f886 100644 --- a/rust/kernel/page.rs +++ b/rust/kernel/page.rs @@ -17,7 +17,11 @@ ARef, AlwaysRefCounted, // }, - types::Opaque, + types::{ + Opaque, + Ownable, + Owned, // + }, uaccess::UserSliceReader, // }; use core::{ @@ -89,7 +93,7 @@ pub trait Sealed {} /// Marker trait for the kinds of a [`Page`]. /// /// A page kind describes what is known about a `struct page`, and thereby what a [`Page`] of that -/// kind can be used for. See [`Base`] and [`RefCounted`]. +/// kind can be used for. See [`Base`], [`RefCounted`] and [`Exclusive`]. /// /// A [`Page`] of any other kind dereferences to a [`Page`] of the base kind [`Base`]. The /// purpose of the dereference is coercion: a reference to a page of any kind can be passed to code @@ -134,6 +138,7 @@ impl PageKind for RefCounted {} /// /// - `Page` is a page for which nothing is known beyond its validity. /// - `Page`, or just `Page`, is a page that is managed by the page reference count. +/// - `Page` is a uniquely owned page whose data follows standard Rust aliasing rules. /// /// Operations that are valid for any page are available for all page kinds. All other page kinds /// dereference to [`Page`], so that a reference to a page of any kind coerces to a @@ -429,3 +434,90 @@ unsafe fn dec_ref(obj: NonNull) { // SAFETY: All instances of `Page` are reference counted, so a `&Page` proves that the refcount is // non-zero and it is okay to obtain an `ARef` from it. unsafe impl AlwaysRefCounted for Page {} + +/// A [`PageKind`] for a page whose data area follows standard Rust aliasing rules. +/// +/// A [`Page`] is accessed through an owning [`Owned>`]. It has the same +/// usage constraints as other Rust types. Thus, it cannot be mapped to user space or shared with +/// devices. This makes it safe to reference the contents of the page while the page is mapped in +/// kernel space. +/// +/// A [`Page`] is backed by a frozen page. The reference count of a frozen page is zero, +/// and references to it cannot be taken. Thus, the exclusive ownership of the page is upheld by +/// the kernel, and not only by the Rust type system. +/// +/// # Invariants +/// +/// For a [`Page`]: +/// +/// - The page is an order 0 frozen page: it was allocated with `alloc_frozen_pages()`, and its +/// reference count is zero for as long as the [`Page`] exists. +/// - The page is owned by a unique [`Owned>`]. No references to the page exist +/// together with the [`Owned>`], other than the ones borrowed from it. +/// - The data of the page is accessed only through references to the [`Page`], +/// including the [`Page`] references it dereferences to. +/// - Safe code reads the data of the page only through a shared reference, and writes it only +/// through a mutable reference. The unsafe raw accessors of [`Page`] can write through a shared +/// reference, their callers are required to rule out races with all other accesses. +pub struct Exclusive; + +impl private::Sealed for Exclusive {} +impl PageKind for Exclusive {} + +impl Deref for Page { + type Target = Page; + + #[inline] + fn deref(&self) -> &Self::Target { + // SAFETY: `Page` is transparent over `struct page` for all page kinds, so the pointer cast + // is valid. `Base` has no invariants in addition to the ones of `Page`, which hold for + // `self`. + unsafe { &*ptr::from_ref(self).cast() } + } +} + +impl Page { + /// Allocates a new exclusive page. + /// + /// # Examples + /// + /// ``` + /// use kernel::page::Page; + /// + /// let page = Page::alloc_exclusive(GFP_KERNEL | __GFP_ZERO)?; + /// # Ok::<(), kernel::alloc::AllocError>(()) + /// ``` + #[inline] + pub fn alloc_exclusive(flags: Flags) -> Result, AllocError> { + // SAFETY: Depending on the value of `gfp_flags`, this call may sleep. Other than that, it + // is always safe to call this method. + let page = unsafe { bindings::alloc_frozen_pages(flags.as_raw(), 0) }; + let page = NonNull::new(page).ok_or(AllocError)?; + + // INVARIANT: The page was just allocated with `alloc_frozen_pages()`, so it is frozen, we + // own it, and its data is only accessible through the returned `Owned>`. + // SAFETY: + // - We just successfully allocated the page, so `page` points to a valid `struct page`. + // Since `Page` is transparent over `struct page`, we can cast the pointer directly. + // - The page was just allocated, so no other `Owned` exists for it. + // - A `struct page` is never moved, so the page is pinned. + // - `Page` allows no other references to exist together with the `Owned`, and + // none exist, as the page is newly allocated and references cannot be taken to a frozen + // page. Thus, it is sound to hand out `&Page` and + // `Pin<&mut Page>`. + // - Pages have no logic that relies on them staying on a given thread or on them not being + // accessed concurrently. + Ok(unsafe { Owned::from_raw(page.cast()) }) + } +} + +impl Ownable for Page { + #[inline] + unsafe fn release(this: NonNull) { + // SAFETY: By the function safety requirements, we have exclusive ownership of the page, and + // by the type invariant the page is an order 0 frozen page, so we can free it with + // `free_frozen_pages()`. Since `Page` is transparent over `struct page`, we can cast the + // pointer directly. + unsafe { bindings::free_frozen_pages(this.cast().as_ptr(), 0) }; + } +} -- 2.54.0