The raw accessors of `Page` are used to access pages that may be mapped to user space or shared with devices. A plain access to such a page is a data race when the other party writes to the page. Add the private method `Page::with_page_as_sysmem()`, which maps a range of a page and passes it to a closure as a `SysMem` view. Access the data of the page through this view in `Page::read_raw()`, `Page::write_raw()` and `Page::fill_zero_raw()`, so that all accesses are byte-wise atomic. Zero the data with `SysMem::fill_zero()`. Narrow the safety requirements of these methods, and of `Page::read_sysmem()` and `Page::write_sysmem()`, accordingly: a call must not race with a plain access to the same data, but it is allowed to race with other non-plain accesses. `Page::copy_from_user_slice_raw()` still writes to the page with plain accesses. State that in the safety requirements. Signed-off-by: Andreas Hindborg --- rust/kernel/page.rs | 153 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 107 insertions(+), 46 deletions(-) diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs index 3c48ecde4d14..ff191bfcd3f9 100644 --- a/rust/kernel/page.rs +++ b/rust/kernel/page.rs @@ -334,28 +334,74 @@ fn with_pointer_into_page( } } + /// Runs a piece of code with a range of this page mapped as a [`SysMem`] view. + /// + /// If `f` is called, then it is called with a view of the `len` bytes of the page that start + /// at `offset`. Accesses through the view are volatile or byte-wise atomic, so the view is + /// suitable for a page whose data is subject to concurrent writes by hardware or user space. + /// The view is valid for the duration of the call to `f`. + /// + /// If `offset` and `len` refer to a region outside of this page, then this method returns + /// [`EINVAL`] and does not call `f`. + /// + /// # Safety + /// + /// * Callers must ensure that no plain access to the mapped range races with an access + /// performed through the view, see [`SysMem`]. + /// * Callers must ensure that no mutable reference to the data of the page that overlaps with + /// the mapped range exists during this call. If `f` writes through the view, no reference to + /// that data may exist during this call. Such references can be obtained from a + /// [`Page`], see [`Page::as_slice`]. + /// * If the page is a [`Page`], `f` must only write initialized bytes through the + /// view. + /// * Callers must ensure that the view is only used on the current task, as this method may + /// use a local mapping. + unsafe fn with_page_as_sysmem( + &self, + offset: usize, + len: usize, + f: impl FnOnce(SysMem<'_, [u8]>) -> Result, + ) -> Result { + self.with_pointer_into_page(offset, len, move |ptr| { + // SAFETY: + // - If `with_pointer_into_page` calls into this closure, then it has performed a + // bounds check and guarantees that `ptr` is valid for `len` bytes for the duration + // of the closure. A byte slice has an alignment of 1. + // - By the safety requirements of this function, no plain access to the range races + // with an access through the view. + f(unsafe { SysMem::new(ptr::slice_from_raw_parts_mut(ptr, len)) }) + }) + } + /// Maps the page and reads from it into the given buffer. /// /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes /// outside of the page, then this call returns [`EINVAL`]. /// + /// The copy is byte-wise atomic, so it is well-defined even if the data of the page is + /// concurrently written by hardware, by user space, or by other kernel code using atomic or + /// volatile accesses. In that case, the data that is read may be arbitrary. + /// /// # Safety /// - /// * Callers must ensure that `dst` is valid for writing `len` bytes. - /// * Callers must ensure that this call does not race with a write to the same page that - /// overlaps with this read. + /// * Callers must ensure that `dst` is valid for writing `len` bytes, and that this memory + /// does not overlap with the data of the page. + /// * Callers must ensure that this call does not race with a plain write to the same page that + /// overlaps with this read, see [`SysMem`]. /// * Callers must ensure that no mutable reference to the data of the page that overlaps with /// this read exists during this call. pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result { - self.with_pointer_into_page(offset, len, move |src| { - // SAFETY: If `with_pointer_into_page` calls into this closure, then - // it has performed a bounds check and guarantees that `src` is - // valid for `len` bytes. - // - // There caller guarantees that there is no data race. - unsafe { ptr::copy_nonoverlapping(src, dst, len) }; + let copy = |src: SysMem<'_, [u8]>| { + // SAFETY: By the safety requirements of this function, `dst` is valid for writing + // `len` bytes, which is equal to `src.size()`, and it does not overlap with `src`. + unsafe { SysMemBackend::copy_from_io(src, dst) }; Ok(()) - }) + }; + + // SAFETY: By the safety requirements of this function, no plain write races with the read + // through the view, and no mutable reference to the data that is read exists. `copy` does + // not write through the view, and it only uses the view on the current task. + unsafe { self.with_page_as_sysmem(offset, len, copy) } } /// Maps the page and reads from it into the given system memory region using byte-wise atomic @@ -366,14 +412,14 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result /// /// The writes to `dst` are byte-wise atomic, so they are well-defined even if `dst` is /// concurrently accessed by hardware, by user space, or by other kernel code using atomic or - /// volatile accesses. The reads from the page are plain accesses, so the page must not be - /// written to concurrently. + /// volatile accesses. The reads from the page are byte-wise atomic as well. /// /// # Safety /// /// Callers must ensure that: /// - /// - This call does not race with a write to the source page that overlaps with this read. + /// - This call does not race with a plain write to the source page that overlaps with this + /// read, see [`SysMem`]. /// - No mutable reference to the data of the page that overlaps with this read exists during /// this call. /// - The memory region of `dst` does not overlap with the memory of the page. @@ -393,8 +439,8 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result /// // way while `dst` is alive. /// let dst = unsafe { SysMem::new(&raw mut buf as *mut [u8]) }; /// - /// // SAFETY: `page` is not written to during this call, no reference to the data of `page` - /// // exists, and `buf` is not part of the page. + /// // SAFETY: No plain write to `page` happens during this call, no reference to the data of + /// // `page` exists, and `buf` is not part of the page. /// unsafe { page.read_sysmem(dst, 0)? }; /// assert_eq!(buf, [0u8; 16]); /// # Ok::<(), Error>(()) @@ -405,8 +451,8 @@ pub unsafe fn read_sysmem(&self, dst: SysMem<'_, [u8]>, offset: usize) -> Result // - If `with_pointer_into_page` calls into this closure, then it has performed a // bounds check and guarantees that `src` is valid for `dst.len()` bytes, which is // equal to `dst.size()`. - // - By the safety requirements of this function, the page is not written to during - // this call, so `src` is valid for volatile reads. + // - By the safety requirements of this function, no plain write to the page races + // with this call, so `src` is valid for volatile reads. // - By the safety requirements of this function, `src` does not overlap with `dst`. unsafe { SysMemBackend::copy_to_io(dst, src) }; Ok(()) @@ -418,25 +464,34 @@ pub unsafe fn read_sysmem(&self, dst: SysMem<'_, [u8]>, offset: usize) -> Result /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes /// outside of the page, then this call returns [`EINVAL`]. /// + /// The copy is byte-wise atomic, so it is well-defined even if the data of the page is + /// concurrently accessed by hardware, by user space, or by other kernel code using atomic or + /// volatile accesses. + /// /// # Safety /// - /// * Callers must ensure that `src` is valid for reading `len` bytes. - /// * Callers must ensure that this call does not race with a read or write to the same page - /// that overlaps with this write. + /// * Callers must ensure that `src` is valid for reading `len` bytes, and that this memory + /// does not overlap with the data of the page. + /// * Callers must ensure that this call does not race with a plain read or write to the same + /// page that overlaps with this write, see [`SysMem`]. /// * Callers must ensure that no reference to the data of the page that overlaps with this /// write exists during this call. Such references can be obtained from a /// [`Page`], see [`Page::as_slice`]. /// * If the page is a [`Page`], callers must ensure that the `len` bytes at `src` /// are initialized. pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Result { - self.with_pointer_into_page(offset, len, move |dst| { - // SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a - // bounds check and guarantees that `dst` is valid for `len` bytes. - // - // There caller guarantees that there is no data race. - unsafe { ptr::copy_nonoverlapping(src, dst, len) }; + let copy = |dst: SysMem<'_, [u8]>| { + // SAFETY: By the safety requirements of this function, `src` is valid for reading + // `len` bytes, which is equal to `dst.size()`, and it does not overlap with `dst`. + unsafe { SysMemBackend::copy_to_io(dst, src) }; Ok(()) - }) + }; + + // SAFETY: By the safety requirements of this function, no plain access races with the + // write through the view, no reference to the data that is written exists, and the bytes + // written are initialized if the page requires it. `copy` only uses the view on the + // current task. + unsafe { self.with_page_as_sysmem(offset, len, copy) } } /// Maps the page and writes into it from the given system memory region using byte-wise atomic @@ -447,15 +502,14 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res /// /// The reads from `src` are byte-wise atomic, so they are well-defined even if `src` is /// concurrently accessed by hardware, by user space, or by other kernel code using atomic or - /// volatile accesses. The writes to the page are plain accesses, so the page must not be read - /// from or written to concurrently. + /// volatile accesses. The writes to the page are byte-wise atomic as well. /// /// # Safety /// /// Callers must ensure that: /// - /// - This call does not race with a read or write to the destination page that overlaps with - /// this write. + /// - This call does not race with a plain read or write to the destination page that overlaps + /// with this write, see [`SysMem`]. /// - No reference to the data of the page that overlaps with this write exists during this /// call. Such references can be obtained from a [`Page`], see /// [`Page::as_slice`]. @@ -483,8 +537,9 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res /// ) /// }; /// - /// // SAFETY: `page` is not accessed in any other way during these calls, no reference to the - /// // data of `page` exists, and neither `data` nor `buf` is part of the page. + /// // SAFETY: No plain access to `page` happens during these calls, no reference to the data + /// // of `page` exists, `data` is initialized, and neither `data` nor `buf` is part of the + /// // page. /// unsafe { /// page.write_sysmem(src, 0)?; /// page.read_sysmem(dst, 0)?; @@ -498,8 +553,8 @@ pub unsafe fn write_sysmem(&self, src: SysMem<'_, [u8]>, offset: usize) -> Resul // - If `with_pointer_into_page` calls into this closure, then it has performed a // bounds check and guarantees that `dst` is valid for `src.len()` bytes, which is // equal to `src.size()`. - // - By the safety requirements of this function, the page is not read from or written - // to during this call, so `dst` is valid for volatile writes. + // - By the safety requirements of this function, no plain access to the page races + // with this call, so `dst` is valid for volatile writes. // - By the safety requirements of this function, `dst` does not overlap with `src`. unsafe { SysMemBackend::copy_from_io(src, dst) }; Ok(()) @@ -511,22 +566,27 @@ pub unsafe fn write_sysmem(&self, src: SysMem<'_, [u8]>, offset: usize) -> Resul /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes /// outside of the page, then this call returns [`EINVAL`]. /// + /// The writes are byte-wise atomic, so they are well-defined even if the data of the page is + /// concurrently accessed by hardware, by user space, or by other kernel code using atomic or + /// volatile accesses. + /// /// # Safety /// - /// * Callers must ensure that this call does not race with a read or write to the same page - /// that overlaps with this write. + /// * Callers must ensure that this call does not race with a plain read or write to the same + /// page that overlaps with this write, see [`SysMem`]. /// * Callers must ensure that no reference to the data of the page that overlaps with this /// write exists during this call. Such references can be obtained from a /// [`Page`], see [`Page::as_slice`]. pub unsafe fn fill_zero_raw(&self, offset: usize, len: usize) -> Result { - self.with_pointer_into_page(offset, len, move |dst| { - // SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a - // bounds check and guarantees that `dst` is valid for `len` bytes. - // - // There caller guarantees that there is no data race. - unsafe { ptr::write_bytes(dst, 0u8, len) }; + let fill = |dst: SysMem<'_, [u8]>| { + dst.fill_zero(); Ok(()) - }) + }; + + // SAFETY: By the safety requirements of this function, no plain access races with the + // write through the view, and no reference to the data that is written exists. The bytes + // written are initialized. `fill` only uses the view on the current task. + unsafe { self.with_page_as_sysmem(offset, len, fill) } } /// Copies data from userspace into this page. @@ -540,7 +600,8 @@ pub unsafe fn fill_zero_raw(&self, offset: usize, len: usize) -> Result { /// # Safety /// /// * Callers must ensure that this call does not race with a read or write to the same page - /// that overlaps with this write. + /// that overlaps with this write. This includes byte-wise atomic accesses, as this method + /// writes to the page with plain accesses. /// * Callers must ensure that no reference to the data of the page that overlaps with this /// write exists during this call. Such references can be obtained from a /// [`Page`], see [`Page::as_slice`]. -- 2.54.0