projects/linux·blob master · refs

1797 lines · 58.5 KiB

1// SPDX-License-Identifier: Apache-2.0 OR MIT
2
3//! Library to safely and fallibly initialize pinned `struct`s using in-place constructors.
4//!
5//! [Pinning][pinning] is Rust's way of ensuring data does not move.
6//!
7//! It also allows in-place initialization of big `struct`s that would otherwise produce a stack
8//! overflow.
9//!
10//! This library's main use-case is in [Rust-for-Linux]. Although this version can be used
11//! standalone.
12//!
13//! There are cases when you want to in-place initialize a struct. For example when it is very big
14//! and moving it from the stack is not an option, because it is bigger than the stack itself.
15//! Another reason would be that you need the address of the object to initialize it. This stands
16//! in direct conflict with Rust's normal process of first initializing an object and then moving
17//! it into it's final memory location. For more information, see
18//! <https://rust-for-linux.com/the-safe-pinned-initialization-problem>.
19//!
20//! This library allows you to do in-place initialization safely.
21//!
22//! ## Nightly Needed for `alloc` feature
23//!
24//! This library requires the [`allocator_api` unstable feature] when the `alloc` feature is
25//! enabled and thus this feature can only be used with a nightly compiler. When enabling the
26//! `alloc` feature, the user will be required to activate `allocator_api` as well.
27//!
28//! [`allocator_api` unstable feature]: https://doc.rust-lang.org/nightly/unstable-book/library-features/allocator-api.html
29//!
30//! The feature is enabled by default, thus by default `pin-init` will require a nightly compiler.
31//! However, using the crate on stable compilers is possible by disabling `alloc`. In practice this
32//! will require the `std` feature, because stable compilers have neither `Box` nor `Arc` in no-std
33//! mode.
34//!
35//! ## Nightly needed for `unsafe-pinned` feature
36//!
37//! This feature enables the `Wrapper` implementation on the unstable `core::pin::UnsafePinned` type.
38//! This requires the [`unsafe_pinned` unstable feature](https://github.com/rust-lang/rust/issues/125735)
39//! and therefore a nightly compiler. Note that this feature is not enabled by default.
40//!
41//! # Overview
42//!
43//! To initialize a `struct` with an in-place constructor you will need two things:
44//! - an in-place constructor,
45//! - a memory location that can hold your `struct` (this can be the [stack], an [`Arc<T>`],
46//! [`Box<T>`] or any other smart pointer that supports this library).
47//!
48//! To get an in-place constructor there are generally three options:
49//! - directly creating an in-place constructor using the [`pin_init!`] macro,
50//! - a custom function/macro returning an in-place constructor provided by someone else,
51//! - using the unsafe function [`pin_init_from_closure()`] to manually create an initializer.
52//!
53//! Aside from pinned initialization, this library also supports in-place construction without
54//! pinning, the macros/types/functions are generally named like the pinned variants without the
55//! `pin_` prefix.
56//!
57//! # Examples
58//!
59//! Throughout the examples we will often make use of the `CMutex` type which can be found in
60//! `../examples/mutex.rs`. It is essentially a userland rebuild of the `struct mutex` type from
61//! the Linux kernel. It also uses a wait list and a basic spinlock. Importantly the wait list
62//! requires it to be pinned to be locked and thus is a prime candidate for using this library.
63//!
64//! ## Using the [`pin_init!`] macro
65//!
66//! If you want to use [`PinInit`], then you will have to annotate your `struct` with
67//! `#[`[`pin_data`]`]`. It is a macro that uses `#[pin]` as a marker for
68//! [structurally pinned fields]. After doing this, you can then create an in-place constructor via
69//! [`pin_init!`]. The syntax is almost the same as normal `struct` initializers. The difference is
70//! that you need to write `<-` instead of `:` for fields that you want to initialize in-place.
71//!
72//! ```rust
73//! # #![feature(allocator_api)]
74//! # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
75//! # use core::pin::Pin;
76//! use pin_init::{pin_data, pin_init, InPlaceInit};
77//!
78//! #[pin_data]
79//! struct Foo {
80//! #[pin]
81//! a: CMutex<usize>,
82//! b: u32,
83//! }
84//!
85//! let foo = pin_init!(Foo {
86//! a <- CMutex::new(42),
87//! b: 24,
88//! });
89//! # let _ = Box::pin_init(foo);
90//! ```
91//!
92//! `foo` now is of the type [`impl PinInit<Foo>`]. We can now use any smart pointer that we like
93//! (or just the stack) to actually initialize a `Foo`:
94//!
95//! ```rust
96//! # #![feature(allocator_api)]
97//! # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
98//! # use core::{alloc::AllocError, pin::Pin};
99//! # use pin_init::*;
100//! #
101//! # #[pin_data]
102//! # struct Foo {
103//! # #[pin]
104//! # a: CMutex<usize>,
105//! # b: u32,
106//! # }
107//! #
108//! # let foo = pin_init!(Foo {
109//! # a <- CMutex::new(42),
110//! # b: 24,
111//! # });
112//! let foo: Result<Pin<Box<Foo>>, AllocError> = Box::pin_init(foo);
113//! ```
114//!
115//! For more information see the [`pin_init!`] macro.
116//!
117//! ## Using a custom function/macro that returns an initializer
118//!
119//! Many types that use this library supply a function/macro that returns an initializer, because
120//! the above method only works for types where you can access the fields.
121//!
122//! ```rust
123//! # #![feature(allocator_api)]
124//! # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
125//! # use pin_init::*;
126//! # use std::sync::Arc;
127//! # use core::pin::Pin;
128//! let mtx: Result<Pin<Arc<CMutex<usize>>>, _> = Arc::pin_init(CMutex::new(42));
129//! ```
130//!
131//! To declare an init macro/function you just return an [`impl PinInit<T, E>`]:
132//!
133//! ```rust
134//! # #![feature(allocator_api)]
135//! # use pin_init::*;
136//! # #[path = "../examples/error.rs"] mod error; use error::Error;
137//! # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
138//! #[pin_data]
139//! struct DriverData {
140//! #[pin]
141//! status: CMutex<i32>,
142//! buffer: Box<[u8; 1_000_000]>,
143//! }
144//!
145//! impl DriverData {
146//! fn new() -> impl PinInit<Self, Error> {
147//! pin_init!(Self {
148//! status <- CMutex::new(0),
149//! buffer: Box::init(pin_init::init_zeroed())?,
150//! }? Error)
151//! }
152//! }
153//! ```
154//!
155//! ## Manual creation of an initializer
156//!
157//! Often when working with primitives the previous approaches are not sufficient. That is where
158//! [`pin_init_from_closure()`] comes in. This `unsafe` function allows you to create a
159//! [`impl PinInit<T, E>`] directly from a closure. Of course you have to ensure that the closure
160//! actually does the initialization in the correct way. Here are the things to look out for
161//! (we are calling the parameter to the closure `slot`):
162//! - when the closure returns `Ok(())`, then it has completed the initialization successfully, so
163//! `slot` now contains a valid bit pattern for the type `T`,
164//! - when the closure returns `Err(e)`, then the caller may deallocate the memory at `slot`, so
165//! you need to take care to clean up anything if your initialization fails mid-way,
166//! - you may assume that `slot` will stay pinned even after the closure returns until `drop` of
167//! `slot` gets called.
168//!
169//! ```rust
170//! # #![feature(extern_types)]
171//! use pin_init::{pin_data, pinned_drop, PinInit, PinnedDrop, pin_init_from_closure};
172//! use core::{
173//! marker::PhantomPinned,
174//! cell::UnsafeCell,
175//! pin::Pin,
176//! mem::MaybeUninit,
177//! };
178//! mod bindings {
179//! #[repr(C)]
180//! pub struct foo {
181//! /* fields from C ... */
182//! }
183//! extern "C" {
184//! pub fn init_foo(ptr: *mut foo);
185//! pub fn destroy_foo(ptr: *mut foo);
186//! #[must_use = "you must check the error return code"]
187//! pub fn enable_foo(ptr: *mut foo, flags: u32) -> i32;
188//! }
189//! }
190//!
191//! /// # Invariants
192//! ///
193//! /// `foo` is always initialized
194//! #[pin_data(PinnedDrop)]
195//! pub struct RawFoo {
196//! #[pin]
197//! _p: PhantomPinned,
198//! #[pin]
199//! foo: UnsafeCell<MaybeUninit<bindings::foo>>,
200//! }
201//!
202//! impl RawFoo {
203//! pub fn new(flags: u32) -> impl PinInit<Self, i32> {
204//! // SAFETY:
205//! // - when the closure returns `Ok(())`, then it has successfully initialized and
206//! // enabled `foo`,
207//! // - when it returns `Err(e)`, then it has cleaned up before
208//! unsafe {
209//! pin_init_from_closure(move |slot: *mut Self| {
210//! // `slot` contains uninit memory, avoid creating a reference.
211//! let foo = &raw mut (*slot).foo;
212//! let foo = UnsafeCell::raw_get(foo).cast::<bindings::foo>();
213//!
214//! // Initialize the `foo`
215//! bindings::init_foo(foo);
216//!
217//! // Try to enable it.
218//! let err = bindings::enable_foo(foo, flags);
219//! if err != 0 {
220//! // Enabling has failed, first clean up the foo and then return the error.
221//! bindings::destroy_foo(foo);
222//! Err(err)
223//! } else {
224//! // All fields of `RawFoo` have been initialized, since `_p` is a ZST.
225//! Ok(())
226//! }
227//! })
228//! }
229//! }
230//! }
231//!
232//! #[pinned_drop]
233//! impl PinnedDrop for RawFoo {
234//! fn drop(self: Pin<&mut Self>) {
235//! // SAFETY: Since `foo` is initialized, destroying is safe.
236//! unsafe { bindings::destroy_foo(self.foo.get().cast::<bindings::foo>()) };
237//! }
238//! }
239//! ```
240//!
241//! For more information on how to use [`pin_init_from_closure()`], take a look at the uses inside
242//! the `kernel` crate. The [`sync`] module is a good starting point.
243//!
244//! [`sync`]: https://rust.docs.kernel.org/kernel/sync/index.html
245//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
246//! [structurally pinned fields]:
247//! https://doc.rust-lang.org/std/pin/index.html#projections-and-structural-pinning
248//! [stack]: crate::stack_pin_init
249#![cfg_attr(
250 kernel,
251 doc = "[`Arc<T>`]: https://rust.docs.kernel.org/kernel/sync/struct.Arc.html"
252)]
253#![cfg_attr(
254 kernel,
255 doc = "[`Box<T>`]: https://rust.docs.kernel.org/kernel/alloc/kbox/struct.Box.html"
256)]
257#![cfg_attr(not(kernel), doc = "[`Arc<T>`]: alloc::alloc::sync::Arc")]
258#![cfg_attr(not(kernel), doc = "[`Box<T>`]: alloc::alloc::boxed::Box")]
259//! [`impl PinInit<Foo>`]: crate::PinInit
260//! [`impl PinInit<T, E>`]: crate::PinInit
261//! [`impl Init<T, E>`]: crate::Init
262//! [Rust-for-Linux]: https://rust-for-linux.com/
263
264#![forbid(missing_docs, unsafe_op_in_unsafe_fn)]
265#![cfg_attr(not(feature = "std"), no_std)]
266#![cfg_attr(feature = "alloc", feature(allocator_api))]
267#![cfg_attr(
268 all(feature = "unsafe-pinned", CONFIG_RUSTC_HAS_UNSAFE_PINNED),
269 feature(unsafe_pinned)
270)]
271#![cfg_attr(all(USE_RUSTC_FEATURES, doc), allow(internal_features))]
272#![cfg_attr(all(USE_RUSTC_FEATURES, doc), feature(rustdoc_internals))]
273
274use core::{
275 cell::UnsafeCell,
276 convert::Infallible,
277 marker::PhantomData,
278 mem::MaybeUninit,
279 num::*,
280 pin::Pin,
281 ptr::{self, NonNull},
282};
283
284// This is used by doc-tests -- the proc-macros expand to `::pin_init::...` and without this the
285// doc-tests wouldn't have an extern crate named `pin_init`.
286#[allow(unused_extern_crates)]
287extern crate self as pin_init;
288
289#[doc(hidden)]
290pub mod __internal;
291
292#[cfg(any(feature = "std", feature = "alloc"))]
293mod alloc;
294#[cfg(any(feature = "std", feature = "alloc"))]
295pub use alloc::InPlaceInit;
296
297/// Used to specify the pinning information of the fields of a struct.
298///
299/// This is somewhat similar in purpose as
300/// [pin-project-lite](https://crates.io/crates/pin-project-lite).
301/// Place this macro on a struct definition and then `#[pin]` in front of the attributes of each
302/// field you want to structurally pin.
303///
304/// This macro enables the use of the [`pin_init!`] macro. When pin-initializing a `struct`,
305/// then `#[pin]` directs the type of initializer that is required.
306///
307/// If your `struct` implements `Drop`, then you need to add `PinnedDrop` as arguments to this
308/// macro, and change your `Drop` implementation to `PinnedDrop` annotated with
309/// `#[`[`macro@pinned_drop`]`]`, since dropping pinned values requires extra care.
310///
311/// # Examples
312///
313/// ```
314/// # #![feature(allocator_api)]
315/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
316/// use pin_init::pin_data;
317///
318/// enum Command {
319/// /* ... */
320/// }
321///
322/// #[pin_data]
323/// struct DriverData {
324/// #[pin]
325/// queue: CMutex<Vec<Command>>,
326/// buf: Box<[u8; 1024 * 1024]>,
327/// }
328/// ```
329///
330/// ```
331/// # #![feature(allocator_api)]
332/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
333/// # mod bindings { pub struct info; pub unsafe fn destroy_info(_: *mut info) {} }
334/// use core::pin::Pin;
335/// use pin_init::{pin_data, pinned_drop, PinnedDrop};
336///
337/// enum Command {
338/// /* ... */
339/// }
340///
341/// #[pin_data(PinnedDrop)]
342/// struct DriverData {
343/// #[pin]
344/// queue: CMutex<Vec<Command>>,
345/// buf: Box<[u8; 1024 * 1024]>,
346/// raw_info: *mut bindings::info,
347/// }
348///
349/// #[pinned_drop]
350/// impl PinnedDrop for DriverData {
351/// fn drop(self: Pin<&mut Self>) {
352/// unsafe { bindings::destroy_info(self.raw_info) };
353/// }
354/// }
355/// ```
356pub use ::pin_init_internal::pin_data;
357
358/// Used to implement `PinnedDrop` safely.
359///
360/// Only works on structs that are annotated via `#[`[`macro@pin_data`]`]`.
361///
362/// # Examples
363///
364/// ```
365/// # #![feature(allocator_api)]
366/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
367/// # mod bindings { pub struct info; pub unsafe fn destroy_info(_: *mut info) {} }
368/// use core::pin::Pin;
369/// use pin_init::{pin_data, pinned_drop, PinnedDrop};
370///
371/// enum Command {
372/// /* ... */
373/// }
374///
375/// #[pin_data(PinnedDrop)]
376/// struct DriverData {
377/// #[pin]
378/// queue: CMutex<Vec<Command>>,
379/// buf: Box<[u8; 1024 * 1024]>,
380/// raw_info: *mut bindings::info,
381/// }
382///
383/// #[pinned_drop]
384/// impl PinnedDrop for DriverData {
385/// fn drop(self: Pin<&mut Self>) {
386/// unsafe { bindings::destroy_info(self.raw_info) };
387/// }
388/// }
389/// ```
390pub use ::pin_init_internal::pinned_drop;
391
392/// Derives the [`Zeroable`] trait for the given `struct` or `union`.
393///
394/// This can only be used for `struct`s/`union`s where every field implements the [`Zeroable`]
395/// trait.
396///
397/// # Examples
398///
399/// ```
400/// use pin_init::Zeroable;
401///
402/// #[derive(Zeroable)]
403/// pub struct DriverData {
404/// pub(crate) id: i64,
405/// buf_ptr: *mut u8,
406/// len: usize,
407/// }
408/// ```
409///
410/// ```
411/// use pin_init::Zeroable;
412///
413/// #[derive(Zeroable)]
414/// pub union SignCast {
415/// signed: i64,
416/// unsigned: u64,
417/// }
418/// ```
419pub use ::pin_init_internal::Zeroable;
420
421/// Derives the [`Zeroable`] trait for the given `struct` or `union` if all fields implement
422/// [`Zeroable`].
423///
424/// Contrary to the derive macro named [`macro@Zeroable`], this one silently fails when a field
425/// doesn't implement [`Zeroable`].
426///
427/// # Examples
428///
429/// ```
430/// use pin_init::MaybeZeroable;
431///
432/// // implements `Zeroable`
433/// #[derive(MaybeZeroable)]
434/// pub struct DriverData {
435/// pub(crate) id: i64,
436/// buf_ptr: *mut u8,
437/// len: usize,
438/// }
439///
440/// // does not implement `Zeroable`
441/// #[derive(MaybeZeroable)]
442/// pub struct DriverData2 {
443/// pub(crate) id: i64,
444/// buf_ptr: *mut u8,
445/// len: usize,
446/// // this field doesn't implement `Zeroable`
447/// other_data: &'static i32,
448/// }
449/// ```
450pub use ::pin_init_internal::MaybeZeroable;
451
452/// Initialize and pin a type directly on the stack.
453///
454/// # Examples
455///
456/// ```rust
457/// # #![feature(allocator_api)]
458/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
459/// # use pin_init::*;
460/// # use core::pin::Pin;
461/// #[pin_data]
462/// struct Foo {
463/// #[pin]
464/// a: CMutex<usize>,
465/// b: Bar,
466/// }
467///
468/// #[pin_data]
469/// struct Bar {
470/// x: u32,
471/// }
472///
473/// stack_pin_init!(let foo = pin_init!(Foo {
474/// a <- CMutex::new(42),
475/// b: Bar {
476/// x: 64,
477/// },
478/// }));
479/// let foo: Pin<&mut Foo> = foo;
480/// println!("a: {}", &*foo.a.lock());
481/// ```
482///
483/// # Syntax
484///
485/// A normal `let` binding with optional type annotation. The expression is expected to implement
486/// [`PinInit`]/[`Init`] with the error type [`Infallible`]. If you want to use a different error
487/// type, then use [`stack_try_pin_init!`].
488#[macro_export]
489macro_rules! stack_pin_init {
490 (let $var:ident $(: $t:ty)? = $val:expr) => {
491 let val = $val;
492 let mut $var = ::core::pin::pin!($crate::__internal::StackInit$(::<$t>)?::uninit());
493 let Ok(mut $var) = $crate::__internal::StackInit::init($var, val);
494 };
495}
496
497/// Initialize and pin a type directly on the stack.
498///
499/// # Examples
500///
501/// ```rust
502/// # #![feature(allocator_api)]
503/// # #[path = "../examples/error.rs"] mod error; use error::Error;
504/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
505/// # use pin_init::*;
506/// #[pin_data]
507/// struct Foo {
508/// #[pin]
509/// a: CMutex<usize>,
510/// b: Box<Bar>,
511/// }
512///
513/// struct Bar {
514/// x: u32,
515/// }
516///
517/// stack_try_pin_init!(let foo: Foo = pin_init!(Foo {
518/// a <- CMutex::new(42),
519/// b: Box::try_new(Bar {
520/// x: 64,
521/// })?,
522/// }? Error));
523/// let foo = foo.unwrap();
524/// println!("a: {}", &*foo.a.lock());
525/// ```
526///
527/// ```rust
528/// # #![feature(allocator_api)]
529/// # #[path = "../examples/error.rs"] mod error; use error::Error;
530/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
531/// # use pin_init::*;
532/// #[pin_data]
533/// struct Foo {
534/// #[pin]
535/// a: CMutex<usize>,
536/// b: Box<Bar>,
537/// }
538///
539/// struct Bar {
540/// x: u32,
541/// }
542///
543/// stack_try_pin_init!(let foo: Foo =? pin_init!(Foo {
544/// a <- CMutex::new(42),
545/// b: Box::try_new(Bar {
546/// x: 64,
547/// })?,
548/// }? Error));
549/// println!("a: {}", &*foo.a.lock());
550/// # Ok::<_, Error>(())
551/// ```
552///
553/// # Syntax
554///
555/// A normal `let` binding with optional type annotation. The expression is expected to implement
556/// [`PinInit`]/[`Init`]. This macro assigns a result to the given variable, adding a `?` after the
557/// `=` will propagate this error.
558#[macro_export]
559macro_rules! stack_try_pin_init {
560 (let $var:ident $(: $t:ty)? = $val:expr) => {
561 let val = $val;
562 let mut $var = ::core::pin::pin!($crate::__internal::StackInit$(::<$t>)?::uninit());
563 let mut $var = $crate::__internal::StackInit::init($var, val);
564 };
565 (let $var:ident $(: $t:ty)? =? $val:expr) => {
566 let val = $val;
567 let mut $var = ::core::pin::pin!($crate::__internal::StackInit$(::<$t>)?::uninit());
568 let mut $var = $crate::__internal::StackInit::init($var, val)?;
569 };
570}
571
572/// Construct an in-place, fallible pinned initializer for `struct`s.
573///
574/// The error type defaults to [`Infallible`]; if you need a different one, write `? Error` at the
575/// end, after the struct initializer.
576///
577/// The syntax is almost identical to that of a normal `struct` initializer:
578///
579/// ```rust
580/// # use pin_init::*;
581/// # use core::pin::Pin;
582/// #[pin_data]
583/// struct Foo {
584/// a: usize,
585/// b: Bar,
586/// }
587///
588/// #[pin_data]
589/// struct Bar {
590/// x: u32,
591/// }
592///
593/// # fn demo() -> impl PinInit<Foo> {
594/// let a = 42;
595///
596/// let initializer = pin_init!(Foo {
597/// a,
598/// b: Bar {
599/// x: 64,
600/// },
601/// });
602/// # initializer }
603/// # Box::pin_init(demo()).unwrap();
604/// ```
605///
606/// Arbitrary Rust expressions can be used to set the value of a variable.
607///
608/// The fields are initialized in the order that they appear in the initializer. So it is possible
609/// to read already initialized fields using raw pointers.
610///
611/// IMPORTANT: You are not allowed to create references to fields of the struct inside of the
612/// initializer.
613///
614/// # Init-functions
615///
616/// When working with this library it is often desired to let others construct your types without
617/// giving access to all fields. This is where you would normally write a plain function `new` that
618/// would return a new instance of your type. With this library that is also possible. However,
619/// there are a few extra things to keep in mind.
620///
621/// To create an initializer function, simply declare it like this:
622///
623/// ```rust
624/// # use pin_init::*;
625/// # use core::pin::Pin;
626/// # #[pin_data]
627/// # struct Foo {
628/// # a: usize,
629/// # b: Bar,
630/// # }
631/// # #[pin_data]
632/// # struct Bar {
633/// # x: u32,
634/// # }
635/// impl Foo {
636/// fn new() -> impl PinInit<Self> {
637/// pin_init!(Self {
638/// a: 42,
639/// b: Bar {
640/// x: 64,
641/// },
642/// })
643/// }
644/// }
645/// ```
646///
647/// Users of `Foo` can now create it like this:
648///
649/// ```rust
650/// # use pin_init::*;
651/// # use core::pin::Pin;
652/// # #[pin_data]
653/// # struct Foo {
654/// # a: usize,
655/// # b: Bar,
656/// # }
657/// # #[pin_data]
658/// # struct Bar {
659/// # x: u32,
660/// # }
661/// # impl Foo {
662/// # fn new() -> impl PinInit<Self> {
663/// # pin_init!(Self {
664/// # a: 42,
665/// # b: Bar {
666/// # x: 64,
667/// # },
668/// # })
669/// # }
670/// # }
671/// let foo = Box::pin_init(Foo::new());
672/// ```
673///
674/// They can also easily embed it into their own `struct`s:
675///
676/// ```rust
677/// # use pin_init::*;
678/// # use core::pin::Pin;
679/// # #[pin_data]
680/// # struct Foo {
681/// # a: usize,
682/// # b: Bar,
683/// # }
684/// # #[pin_data]
685/// # struct Bar {
686/// # x: u32,
687/// # }
688/// # impl Foo {
689/// # fn new() -> impl PinInit<Self> {
690/// # pin_init!(Self {
691/// # a: 42,
692/// # b: Bar {
693/// # x: 64,
694/// # },
695/// # })
696/// # }
697/// # }
698/// #[pin_data]
699/// struct FooContainer {
700/// #[pin]
701/// foo1: Foo,
702/// #[pin]
703/// foo2: Foo,
704/// other: u32,
705/// }
706///
707/// impl FooContainer {
708/// fn new(other: u32) -> impl PinInit<Self> {
709/// pin_init!(Self {
710/// foo1 <- Foo::new(),
711/// foo2 <- Foo::new(),
712/// other,
713/// })
714/// }
715/// }
716/// ```
717///
718/// Here we see that when using `pin_init!` with `PinInit`, one needs to write `<-` instead of `:`.
719/// This signifies that the given field is initialized in-place. As with `struct` initializers, just
720/// writing the field (in this case `other`) without `:` or `<-` means `other: other,`.
721///
722/// # Syntax
723///
724/// As already mentioned in the examples above, inside of `pin_init!` a `struct` initializer with
725/// the following modifications is expected:
726/// - Fields that you want to initialize in-place have to use `<-` instead of `:`.
727/// - You can use `_: { /* run any user-code here */ },` anywhere where you can place fields in
728/// order to run arbitrary code.
729/// - In front of the initializer you can write `&this in` to have access to a [`NonNull<Self>`]
730/// pointer named `this` inside of the initializer.
731/// - Using struct update syntax one can place `..Zeroable::init_zeroed()` at the very end of the
732/// struct, this initializes every field with 0 and then runs all initializers specified in the
733/// body. This can only be done if [`Zeroable`] is implemented for the struct.
734///
735/// For instance:
736///
737/// ```rust
738/// # use pin_init::*;
739/// # use core::marker::PhantomPinned;
740/// #[pin_data]
741/// #[derive(Zeroable)]
742/// struct Buf {
743/// // `ptr` points into `buf`.
744/// ptr: *mut u8,
745/// buf: [u8; 64],
746/// #[pin]
747/// pin: PhantomPinned,
748/// }
749///
750/// let init = pin_init!(&this in Buf {
751/// buf: [0; 64],
752/// // SAFETY: TODO.
753/// ptr: unsafe { (&raw mut (*this.as_ptr()).buf).cast() },
754/// pin: PhantomPinned,
755/// });
756/// let init = pin_init!(Buf {
757/// buf: [1; 64],
758/// ..Zeroable::init_zeroed()
759/// });
760/// ```
761///
762/// [`NonNull<Self>`]: core::ptr::NonNull
763pub use pin_init_internal::pin_init;
764
765/// Construct an in-place, fallible initializer for `struct`s.
766///
767/// This macro defaults the error to [`Infallible`]; if you need a different one, write `? Error`
768/// at the end, after the struct initializer.
769///
770/// The syntax is identical to [`pin_init!`] and its safety caveats also apply:
771/// - `unsafe` code must guarantee either full initialization or return an error and allow
772/// deallocation of the memory.
773/// - the fields are initialized in the order given in the initializer.
774/// - no references to fields are allowed to be created inside of the initializer.
775///
776/// This initializer is for initializing data in-place that might later be moved. If you want to
777/// pin-initialize, use [`pin_init!`].
778///
779/// # Examples
780///
781/// ```rust
782/// # #![feature(allocator_api)]
783/// # #[path = "../examples/error.rs"] mod error; use error::Error;
784/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
785/// # use pin_init::InPlaceInit;
786/// use pin_init::{init, Init, init_zeroed};
787///
788/// struct BigBuf {
789/// small: [u8; 1024 * 1024],
790/// }
791///
792/// impl BigBuf {
793/// fn new() -> impl Init<Self> {
794/// init!(Self {
795/// small <- init_zeroed(),
796/// })
797/// }
798/// }
799/// # let _ = Box::init(BigBuf::new());
800/// ```
801pub use pin_init_internal::init;
802
803/// Asserts that a field on a struct using `#[pin_data]` is marked with `#[pin]` ie. that it is
804/// structurally pinned.
805///
806/// # Examples
807///
808/// This will succeed:
809/// ```
810/// use pin_init::{pin_data, assert_pinned};
811///
812/// #[pin_data]
813/// struct MyStruct {
814/// #[pin]
815/// some_field: u64,
816/// }
817///
818/// assert_pinned!(MyStruct, some_field, u64);
819/// ```
820///
821/// This will fail:
822/// ```compile_fail
823/// use pin_init::{pin_data, assert_pinned};
824///
825/// #[pin_data]
826/// struct MyStruct {
827/// some_field: u64,
828/// }
829///
830/// assert_pinned!(MyStruct, some_field, u64);
831/// ```
832///
833/// Some uses of the macro may trigger the `can't use generic parameters from outer item` error. To
834/// work around this, you may pass the `inline` parameter to the macro. The `inline` parameter can
835/// only be used when the macro is invoked from a function body.
836/// ```
837/// # use core::pin::Pin;
838/// use pin_init::{pin_data, assert_pinned};
839///
840/// #[pin_data]
841/// struct Foo<T> {
842/// #[pin]
843/// elem: T,
844/// }
845///
846/// impl<T> Foo<T> {
847/// fn project_this(self: Pin<&mut Self>) -> Pin<&mut T> {
848/// assert_pinned!(Foo<T>, elem, T, inline);
849///
850/// // SAFETY: The field is structurally pinned.
851/// unsafe { self.map_unchecked_mut(|me| &mut me.elem) }
852/// }
853/// }
854/// ```
855#[macro_export]
856macro_rules! assert_pinned {
857 ($ty:ty, $field:ident, $field_ty:ty, inline) => {
858 // SAFETY: This code is unreachable.
859 let _ = move |ptr: *mut $ty| unsafe {
860 let data = <$ty as $crate::__internal::HasPinData>::__pin_data();
861 _ = data
862 .$field(ptr)
863 .init($crate::__internal::AlwaysFail::<$field_ty>::new());
864 };
865 };
866
867 ($ty:ty, $field:ident, $field_ty:ty) => {
868 const _: () = {
869 $crate::assert_pinned!($ty, $field, $field_ty, inline);
870 };
871 };
872}
873
874/// A pin-initializer for the type `T`.
875///
876/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
877/// be [`Box<T>`], [`Arc<T>`] or even the stack (see [`stack_pin_init!`]).
878///
879/// Also see the [module description](self).
880///
881/// # Safety
882///
883/// When implementing this trait you will need to take great care. Also there are probably very few
884/// cases where a manual implementation is necessary. Use [`pin_init_from_closure`] where possible.
885///
886/// The [`PinInit::__init`] function:
887/// - returns `Ok(())` if it initialized every field of `slot`,
888/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
889/// - `slot` can be deallocated without UB occurring,
890/// - `slot` does not need to be dropped,
891/// - `slot` is not partially initialized.
892/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
893///
894#[cfg_attr(
895 kernel,
896 doc = "[`Arc<T>`]: https://rust.docs.kernel.org/kernel/sync/struct.Arc.html"
897)]
898#[cfg_attr(
899 kernel,
900 doc = "[`Box<T>`]: https://rust.docs.kernel.org/kernel/alloc/kbox/struct.Box.html"
901)]
902#[cfg_attr(not(kernel), doc = "[`Arc<T>`]: alloc::alloc::sync::Arc")]
903#[cfg_attr(not(kernel), doc = "[`Box<T>`]: alloc::alloc::boxed::Box")]
904#[must_use = "An initializer must be used in order to create its value."]
905pub unsafe trait PinInit<T: ?Sized, E = Infallible>: Sized {
906 /// Alias of [`PinInit::__init`].
907 ///
908 /// New code should use `__init` instead.
909 ///
910 /// # Safety
911 ///
912 /// Same as `__init`.
913 #[inline(always)]
914 #[cfg(not(kernel))]
915 #[deprecated = "use `raw_try_init` instead"]
916 unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
917 // SAFETY: Per safety requirement.
918 unsafe { self.__init(slot) }
919 }
920
921 /// Initializes `slot`.
922 ///
923 /// It is not recommended to call this directly. Use [`raw_init`] or [`raw_try_init`].
924 ///
925 /// # Safety
926 ///
927 /// - `slot` is a valid pointer to uninitialized memory.
928 /// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
929 /// deallocate.
930 /// - `slot` will not move until it is dropped, i.e. it will be pinned.
931 /// If `Self: Init<T, E>`, this requirement is cancelled and it may be moved.
932 unsafe fn __init(self, slot: *mut T) -> Result<(), E>;
933
934 /// First initializes the value using `self` then calls the function `f` with the initialized
935 /// value.
936 ///
937 /// If `f` returns an error the value is dropped and the initializer will forward the error.
938 ///
939 /// # Examples
940 ///
941 /// ```rust
942 /// # #![feature(allocator_api)]
943 /// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
944 /// # use pin_init::*;
945 /// let mtx_init = CMutex::new(42);
946 /// // Make the initializer print the value.
947 /// let mtx_init = mtx_init.pin_chain(|mtx| {
948 /// println!("{:?}", mtx.get_data_mut());
949 /// Ok(())
950 /// });
951 /// ```
952 #[inline]
953 fn pin_chain<F>(self, f: F) -> ChainPinInit<Self, F, T, E>
954 where
955 F: FnOnce(Pin<&mut T>) -> Result<(), E>,
956 {
957 ChainPinInit(self, f, __internal::PhantomInvariant::new())
958 }
959}
960
961/// Initializes `slot` with an initializer.
962///
963/// # Safety
964///
965/// - `slot` is a valid pointer to uninitialized memory.
966/// - `slot` will not move until it is dropped, i.e. it will be pinned.
967/// If `init` implements `Init<T, E>`, this requirement is cancelled and it may be moved.
968#[inline(always)]
969pub unsafe fn raw_init<T>(slot: *mut T, init: impl PinInit<T>) {
970 // SAFETY: Per safety requirement.
971 unsafe { init.__init(slot).unwrap_or_else(|e| match e {}) }
972}
973
974/// Fallibly initializes `slot` with an initializer.
975///
976/// # Safety
977///
978/// - `slot` is a valid pointer to uninitialized memory.
979/// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
980/// deallocate.
981/// - `slot` will not move until it is dropped, i.e. it will be pinned.
982/// If `init` implements `Init<T, E>`, this requirement is cancelled and it may be moved.
983#[inline(always)]
984pub unsafe fn raw_try_init<T, E>(slot: *mut T, init: impl PinInit<T, E>) -> Result<(), E> {
985 // SAFETY: Per safety requirement.
986 unsafe { init.__init(slot) }
987}
988
989/// An initializer returned by [`PinInit::pin_chain`].
990pub struct ChainPinInit<I, F, T: ?Sized, E>(I, F, __internal::PhantomInvariant<(E, T)>);
991
992// SAFETY: The `__init` function is implemented such that it
993// - returns `Ok(())` on successful initialization,
994// - returns `Err(err)` on error and in this case `slot` will be dropped.
995// - considers `slot` pinned.
996unsafe impl<T: ?Sized, E, I, F> PinInit<T, E> for ChainPinInit<I, F, T, E>
997where
998 I: PinInit<T, E>,
999 F: FnOnce(Pin<&mut T>) -> Result<(), E>,
1000{
1001 #[inline]
1002 unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
1003 // SAFETY: All requirements fulfilled since this function is `__init`.
1004 let slot = unsafe { __internal::Slot::<__internal::Pinned, _>::new(slot) };
1005 let mut guard = slot.init(self.0)?;
1006 (self.1)(guard.let_binding())?;
1007 core::mem::forget(guard);
1008 Ok(())
1009 }
1010}
1011
1012/// An initializer for `T`.
1013///
1014/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
1015/// be [`Box<T>`], [`Arc<T>`] or even the stack (see [`stack_pin_init!`]). Because
1016/// [`PinInit<T, E>`] is a super trait, you can use every function that takes it as well.
1017///
1018/// Also see the [module description](self).
1019///
1020/// # Safety
1021///
1022/// When implementing this trait you will need to take great care. Also there are probably very few
1023/// cases where a manual implementation is necessary. Use [`init_from_closure`] where possible.
1024///
1025/// The [`PinInit::__init`] function must work without the pinning requirement; the caller is
1026/// allowed to move the pointee after initialization.
1027///
1028#[cfg_attr(
1029 kernel,
1030 doc = "[`Arc<T>`]: https://rust.docs.kernel.org/kernel/sync/struct.Arc.html"
1031)]
1032#[cfg_attr(
1033 kernel,
1034 doc = "[`Box<T>`]: https://rust.docs.kernel.org/kernel/alloc/kbox/struct.Box.html"
1035)]
1036#[cfg_attr(not(kernel), doc = "[`Arc<T>`]: alloc::alloc::sync::Arc")]
1037#[cfg_attr(not(kernel), doc = "[`Box<T>`]: alloc::alloc::boxed::Box")]
1038#[must_use = "An initializer must be used in order to create its value."]
1039pub unsafe trait Init<T: ?Sized, E = Infallible>: PinInit<T, E> {
1040 /// First initializes the value using `self` then calls the function `f` with the initialized
1041 /// value.
1042 ///
1043 /// If `f` returns an error the value is dropped and the initializer will forward the error.
1044 ///
1045 /// # Examples
1046 ///
1047 /// ```rust
1048 /// use pin_init::{init, init_zeroed, Init};
1049 ///
1050 /// struct Foo {
1051 /// buf: [u8; 1_000_000],
1052 /// }
1053 ///
1054 /// impl Foo {
1055 /// fn setup(&mut self) {
1056 /// println!("Setting up foo");
1057 /// }
1058 /// }
1059 ///
1060 /// let foo = init!(Foo {
1061 /// buf <- init_zeroed()
1062 /// }).chain(|foo| {
1063 /// foo.setup();
1064 /// Ok(())
1065 /// });
1066 /// ```
1067 #[inline]
1068 fn chain<F>(self, f: F) -> ChainInit<Self, F, T, E>
1069 where
1070 F: FnOnce(&mut T) -> Result<(), E>,
1071 {
1072 ChainInit(self, f, __internal::PhantomInvariant::new())
1073 }
1074}
1075
1076/// An initializer returned by [`Init::chain`].
1077pub struct ChainInit<I, F, T: ?Sized, E>(I, F, __internal::PhantomInvariant<(E, T)>);
1078
1079// SAFETY: The `__init` function does not rely on the pinning requirement.
1080unsafe impl<T: ?Sized, E, I, F> Init<T, E> for ChainInit<I, F, T, E>
1081where
1082 I: Init<T, E>,
1083 F: FnOnce(&mut T) -> Result<(), E>,
1084{
1085}
1086
1087// SAFETY: The `__init` function is implemented such that it
1088// - returns `Ok(())` on successful initialization,
1089// - returns `Err(err)` on error and in this case `slot` will be dropped.
1090unsafe impl<T: ?Sized, E, I, F> PinInit<T, E> for ChainInit<I, F, T, E>
1091where
1092 I: Init<T, E>,
1093 F: FnOnce(&mut T) -> Result<(), E>,
1094{
1095 #[inline]
1096 unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
1097 // SAFETY: All requirements fulfilled since this function is `__init`.
1098 let slot = unsafe { __internal::Slot::<__internal::Unpinned, _>::new(slot) };
1099 let mut guard = slot.init(self.0)?;
1100 (self.1)(guard.let_binding())?;
1101 core::mem::forget(guard);
1102 Ok(())
1103 }
1104}
1105
1106/// Implement `PinInit` and `Init` for closures.
1107///
1108/// It is unsafe to create this type, since the closure needs to fulfill the same safety
1109/// requirement as the `__init` functions.
1110struct InitClosure<F, T: ?Sized>(F, __internal::PhantomInvariant<T>);
1111
1112// SAFETY: When constructing via `init_from_closure`, the `__init` function does not rely on the
1113// pinning requirement. When constructing via `pin_init_from_closure`, the opaque type prevents this
1114// implementation from being visible.
1115unsafe impl<T: ?Sized, F, E> Init<T, E> for InitClosure<F, T> where
1116 F: FnOnce(*mut T) -> Result<(), E>
1117{
1118}
1119
1120// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
1121// `__init` invariants.
1122unsafe impl<T: ?Sized, F, E> PinInit<T, E> for InitClosure<F, T>
1123where
1124 F: FnOnce(*mut T) -> Result<(), E>,
1125{
1126 #[inline]
1127 unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
1128 (self.0)(slot)
1129 }
1130}
1131
1132/// Creates a new [`PinInit<T, E>`] from the given closure.
1133///
1134/// # Safety
1135///
1136/// The closure:
1137/// - returns `Ok(())` if it initialized every field of `slot`,
1138/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
1139/// - `slot` can be deallocated without UB occurring,
1140/// - `slot` does not need to be dropped,
1141/// - `slot` is not partially initialized.
1142/// - may assume that the `slot` does not move if `T: !Unpin`,
1143/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
1144#[inline]
1145pub const unsafe fn pin_init_from_closure<T: ?Sized, E>(
1146 f: impl FnOnce(*mut T) -> Result<(), E>,
1147) -> impl PinInit<T, E> {
1148 InitClosure(f, __internal::PhantomInvariant::new())
1149}
1150
1151/// Creates a new [`Init<T, E>`] from the given closure.
1152///
1153/// # Safety
1154///
1155/// The closure:
1156/// - returns `Ok(())` if it initialized every field of `slot`,
1157/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
1158/// - `slot` can be deallocated without UB occurring,
1159/// - `slot` does not need to be dropped,
1160/// - `slot` is not partially initialized.
1161/// - the `slot` may move after initialization.
1162/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
1163#[inline]
1164pub const unsafe fn init_from_closure<T: ?Sized, E>(
1165 f: impl FnOnce(*mut T) -> Result<(), E>,
1166) -> impl Init<T, E> {
1167 InitClosure(f, __internal::PhantomInvariant::new())
1168}
1169
1170/// Changes the to be initialized type.
1171///
1172/// # Safety
1173///
1174/// - `*mut U` must be castable to `*mut T` and any value of type `T` written through such a
1175/// pointer must result in a valid `U`.
1176#[inline]
1177pub const unsafe fn cast_pin_init<T, U, E>(init: impl PinInit<T, E>) -> impl PinInit<U, E> {
1178 // SAFETY: initialization delegated to a valid initializer. Cast is valid by function safety
1179 // requirements.
1180 unsafe { pin_init_from_closure(|ptr: *mut U| init.__init(ptr.cast::<T>())) }
1181}
1182
1183/// Changes the to be initialized type.
1184///
1185/// # Safety
1186///
1187/// - `*mut U` must be castable to `*mut T` and any value of type `T` written through such a
1188/// pointer must result in a valid `U`.
1189#[inline]
1190pub const unsafe fn cast_init<T, U, E>(init: impl Init<T, E>) -> impl Init<U, E> {
1191 // SAFETY: initialization delegated to a valid initializer. Cast is valid by function safety
1192 // requirements.
1193 unsafe { init_from_closure(|ptr: *mut U| init.__init(ptr.cast::<T>())) }
1194}
1195
1196/// An initializer that leaves the memory uninitialized.
1197///
1198/// The initializer is a no-op. The `slot` memory is not changed.
1199#[inline]
1200pub fn uninit<T, E>() -> impl Init<MaybeUninit<T>, E> {
1201 // SAFETY: The memory is allowed to be uninitialized.
1202 unsafe { init_from_closure(|_| Ok(())) }
1203}
1204
1205/// Array initializer from element initializer.
1206struct ArrayInit<T: ?Sized, F>(F, __internal::PhantomInvariant<T>);
1207
1208// SAFETY: On success, all `N` elements of the array have been initialized. On error or panic, the
1209// elements that have been initialized so far are dropped, thus leaving the array uninitialized and
1210// ready to deallocate.
1211unsafe impl<T, F, I, E, const N: usize> PinInit<[T; N], E> for ArrayInit<T, F>
1212where
1213 F: FnMut(usize) -> I,
1214 I: PinInit<T, E>,
1215{
1216 unsafe fn __init(mut self, slot: *mut [T; N]) -> Result<(), E> {
1217 /// # Invariants
1218 ///
1219 /// - `ptr[..num_init]` contains initialized elements of type `T`
1220 /// - `ptr[num_init..N]` (where N is the size of the array) contains uninitialized memory
1221 struct ArrayInitGuard<T> {
1222 /// A pointer to the first element of the array.
1223 ptr: *mut T,
1224 /// The number of initialized elements in the array.
1225 num_init: usize,
1226 }
1227
1228 impl<T> Drop for ArrayInitGuard<T> {
1229 #[inline]
1230 fn drop(&mut self) {
1231 // SAFETY: Per type invariant, `self.ptr[..self.num_init]` are initialized.
1232 unsafe {
1233 core::ptr::drop_in_place(core::ptr::slice_from_raw_parts_mut(
1234 self.ptr,
1235 self.num_init,
1236 ))
1237 };
1238 }
1239 }
1240
1241 // INVARIANT: nothing is initialized yet.
1242 let mut guard = ArrayInitGuard {
1243 ptr: slot.cast::<T>(),
1244 num_init: 0,
1245 };
1246
1247 for i in 0..N {
1248 // INVARIANT: Elements `self.ptr[..self.num_init]` have been initialized
1249 // thus far. This holds true for every `self.num_init = i`.
1250 guard.num_init = i;
1251
1252 let init = (self.0)(i);
1253 // SAFETY:
1254 // - The subslot is derived from `slot` with a valid offset.
1255 // - If `Err` is touched, the subslot is not touched further, the guard will drop
1256 // previously initialized elements only.
1257 // - `slot` is pinned so is the subslot.
1258 unsafe { init.__init(&raw mut (*slot)[i]) }?;
1259 }
1260
1261 // Dismiss the drop guard now that all elements are initialized.
1262 core::mem::forget(guard);
1263 Ok(())
1264 }
1265}
1266
1267// SAFETY: `I: Init` cancels out the pinning requirement on subslots, which is the only place in the
1268// `__init` function that relies on `slot` being pinned.
1269unsafe impl<T, F, I, E, const N: usize> Init<[T; N], E> for ArrayInit<T, F>
1270where
1271 F: FnMut(usize) -> I,
1272 I: Init<T, E>,
1273{
1274}
1275
1276/// Initializes an array by initializing each element via the provided initializer.
1277///
1278/// # Examples
1279///
1280/// ```rust
1281/// # use pin_init::*;
1282/// use pin_init::init_array_from_fn;
1283/// let array: Box<[usize; 1_000]> = Box::init(init_array_from_fn(|i| i)).unwrap();
1284/// assert_eq!(array.len(), 1_000);
1285/// ```
1286#[inline]
1287pub fn init_array_from_fn<I, const N: usize, T, E>(
1288 make_init: impl FnMut(usize) -> I,
1289) -> impl Init<[T; N], E>
1290where
1291 I: Init<T, E>,
1292{
1293 ArrayInit(make_init, __internal::PhantomInvariant::new())
1294}
1295
1296/// Initializes an array by initializing each element via the provided initializer.
1297///
1298/// # Examples
1299///
1300/// ```rust
1301/// # #![feature(allocator_api)]
1302/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
1303/// # use pin_init::*;
1304/// # use core::pin::Pin;
1305/// use pin_init::pin_init_array_from_fn;
1306/// use std::sync::Arc;
1307/// let array: Pin<Arc<[CMutex<usize>; 1_000]>> =
1308/// Arc::pin_init(pin_init_array_from_fn(|i| CMutex::new(i))).unwrap();
1309/// assert_eq!(array.len(), 1_000);
1310/// ```
1311#[inline]
1312pub fn pin_init_array_from_fn<I, const N: usize, T, E>(
1313 make_init: impl FnMut(usize) -> I,
1314) -> impl PinInit<[T; N], E>
1315where
1316 I: PinInit<T, E>,
1317{
1318 ArrayInit(make_init, __internal::PhantomInvariant::new())
1319}
1320
1321/// Construct an initializer in a closure and run it.
1322///
1323/// Returns an initializer that first runs the closure and then the initializer returned by it.
1324///
1325/// See also [`init_scope`].
1326///
1327/// # Examples
1328///
1329/// ```
1330/// # use pin_init::*;
1331/// # #[pin_data]
1332/// # struct Foo { a: u64, b: isize }
1333/// # struct Bar { a: u32, b: isize }
1334/// # fn lookup_bar() -> Result<Bar, Error> { todo!() }
1335/// # struct Error;
1336/// fn init_foo() -> impl PinInit<Foo, Error> {
1337/// pin_init_scope(|| {
1338/// let bar = lookup_bar()?;
1339/// Ok(pin_init!(Foo { a: bar.a.into(), b: bar.b }? Error))
1340/// })
1341/// }
1342/// ```
1343///
1344/// This initializer will first execute `lookup_bar()`, match on it, if it returned an error, the
1345/// initializer itself will fail with that error. If it returned `Ok`, then it will run the
1346/// initializer returned by the [`pin_init!`] invocation.
1347#[inline]
1348pub fn pin_init_scope<T, E, F, I>(make_init: F) -> impl PinInit<T, E>
1349where
1350 F: FnOnce() -> Result<I, E>,
1351 I: PinInit<T, E>,
1352{
1353 // SAFETY:
1354 // - If `make_init` returns `Err`, `Err` is returned and `slot` is completely uninitialized,
1355 // - If `make_init` returns `Ok`, safety requirement are fulfilled by `init.__init`.
1356 // - The safety requirements of `init.__init` are fulfilled, since it's being called from an
1357 // initializer.
1358 unsafe {
1359 pin_init_from_closure(move |slot: *mut T| -> Result<(), E> {
1360 let init = make_init()?;
1361 init.__init(slot)
1362 })
1363 }
1364}
1365
1366/// Construct an initializer in a closure and run it.
1367///
1368/// Returns an initializer that first runs the closure and then the initializer returned by it.
1369///
1370/// See also [`pin_init_scope`].
1371///
1372/// # Examples
1373///
1374/// ```
1375/// # use pin_init::*;
1376/// # struct Foo { a: u64, b: isize }
1377/// # struct Bar { a: u32, b: isize }
1378/// # fn lookup_bar() -> Result<Bar, Error> { todo!() }
1379/// # struct Error;
1380/// fn init_foo() -> impl Init<Foo, Error> {
1381/// init_scope(|| {
1382/// let bar = lookup_bar()?;
1383/// Ok(init!(Foo { a: bar.a.into(), b: bar.b }? Error))
1384/// })
1385/// }
1386/// ```
1387///
1388/// This initializer will first execute `lookup_bar()`, match on it, if it returned an error, the
1389/// initializer itself will fail with that error. If it returned `Ok`, then it will run the
1390/// initializer returned by the [`init!`] invocation.
1391#[inline]
1392pub fn init_scope<T, E, F, I>(make_init: F) -> impl Init<T, E>
1393where
1394 F: FnOnce() -> Result<I, E>,
1395 I: Init<T, E>,
1396{
1397 // SAFETY:
1398 // - If `make_init` returns `Err`, `Err` is returned and `slot` is completely uninitialized,
1399 // - If `make_init` returns `Ok`, safety requirement are fulfilled by `init.__init`.
1400 // - The safety requirements of `init.__init` are fulfilled, since it's being called from an
1401 // initializer.
1402 unsafe {
1403 init_from_closure(move |slot: *mut T| -> Result<(), E> {
1404 let init = make_init()?;
1405 init.__init(slot)
1406 })
1407 }
1408}
1409
1410// SAFETY: The `__init` function does not rely on slot being pinned after it returns.
1411unsafe impl<T> Init<T> for T {}
1412
1413// SAFETY: the `__init` function always returns `Ok(())` and initializes every field of
1414// `slot`. Additionally, all pinning invariants of `T` are upheld.
1415unsafe impl<T> PinInit<T> for T {
1416 #[inline]
1417 unsafe fn __init(self, slot: *mut T) -> Result<(), Infallible> {
1418 // SAFETY: `slot` is valid for writes by the safety requirements of this function.
1419 unsafe { slot.write(self) };
1420 Ok(())
1421 }
1422}
1423
1424// SAFETY: The `__init` function does not rely on slot being pinned after it returns.
1425unsafe impl<T, E> Init<T, E> for Result<T, E> {}
1426
1427// SAFETY: when the `__init` function returns with
1428// - `Ok(())`, `slot` was initialized and all pinned invariants of `T` are upheld.
1429// - `Err(err)`, slot was not written to.
1430unsafe impl<T, E> PinInit<T, E> for Result<T, E> {
1431 #[inline]
1432 unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
1433 // SAFETY: `slot` is valid for writes by the safety requirements of this function.
1434 unsafe { slot.write(self?) };
1435 Ok(())
1436 }
1437}
1438
1439/// Smart pointer containing uninitialized memory and that can write a value.
1440pub trait InPlaceWrite<T> {
1441 /// The type `Self` turns into when the contents are initialized.
1442 type Initialized;
1443
1444 /// Use the given initializer to write a value into `self`.
1445 ///
1446 /// Does not drop the current value and considers it as uninitialized memory.
1447 fn write_init<E>(self, init: impl Init<T, E>) -> Result<Self::Initialized, E>;
1448
1449 /// Use the given pin-initializer to write a value into `self`.
1450 ///
1451 /// Does not drop the current value and considers it as uninitialized memory.
1452 fn write_pin_init<E>(self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E>;
1453}
1454
1455impl<T> InPlaceWrite<T> for &'static mut MaybeUninit<T> {
1456 type Initialized = &'static mut T;
1457
1458 #[inline]
1459 fn write_init<E>(self, init: impl Init<T, E>) -> Result<Self::Initialized, E> {
1460 let slot = self.as_mut_ptr();
1461
1462 // SAFETY: `slot` is a valid pointer to uninitialized memory.
1463 unsafe { init.__init(slot)? };
1464
1465 // SAFETY: The above call initialized the memory.
1466 unsafe { Ok(self.assume_init_mut()) }
1467 }
1468
1469 #[inline]
1470 fn write_pin_init<E>(self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E> {
1471 let slot = self.as_mut_ptr();
1472
1473 // SAFETY: `slot` is a valid pointer to uninitialized memory.
1474 //
1475 // The `'static` borrow guarantees the data will not be
1476 // moved/invalidated until it gets dropped (which is never).
1477 unsafe { init.__init(slot)? };
1478
1479 // SAFETY: The above call initialized the memory.
1480 Ok(Pin::static_mut(unsafe { self.assume_init_mut() }))
1481 }
1482}
1483
1484/// Trait facilitating pinned destruction.
1485///
1486/// Use [`pinned_drop`] to implement this trait safely:
1487///
1488/// ```rust
1489/// # #![feature(allocator_api)]
1490/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
1491/// # use pin_init::*;
1492/// use core::pin::Pin;
1493/// #[pin_data(PinnedDrop)]
1494/// struct Foo {
1495/// #[pin]
1496/// mtx: CMutex<usize>,
1497/// }
1498///
1499/// #[pinned_drop]
1500/// impl PinnedDrop for Foo {
1501/// fn drop(self: Pin<&mut Self>) {
1502/// println!("Foo is being dropped!");
1503/// }
1504/// }
1505/// ```
1506///
1507/// # Safety
1508///
1509/// This trait must be implemented via the [`pinned_drop`] proc-macro attribute on the impl.
1510pub unsafe trait PinnedDrop: __internal::HasPinData {
1511 /// Executes the pinned destructor of this type.
1512 ///
1513 /// While this function is marked safe, it is actually unsafe to call it manually. For this
1514 /// reason it takes an additional parameter. This type can only be constructed by `unsafe` code
1515 /// and thus prevents this function from being called where it should not.
1516 ///
1517 /// This extra parameter will be generated by the `#[pinned_drop]` proc-macro attribute
1518 /// automatically.
1519 fn drop(self: Pin<&mut Self>, only_call_from_drop: __internal::OnlyCallFromDrop);
1520}
1521
1522/// Marker trait for types that can be initialized by writing just zeroes.
1523///
1524/// # Safety
1525///
1526/// The bit pattern consisting of only zeroes is a valid bit pattern for this type. In other words,
1527/// this is not UB:
1528///
1529/// ```rust,ignore
1530/// let val: Self = unsafe { core::mem::zeroed() };
1531/// ```
1532pub unsafe trait Zeroable {
1533 /// Create a new zeroed `Self`.
1534 ///
1535 /// The returned initializer will write `0x00` to every byte of the given `slot`.
1536 #[inline]
1537 fn init_zeroed() -> impl Init<Self>
1538 where
1539 Self: Sized,
1540 {
1541 init_zeroed()
1542 }
1543
1544 /// Create a `Self` consisting of all zeroes.
1545 ///
1546 /// Whenever a type implements [`Zeroable`], this function should be preferred over
1547 /// [`core::mem::zeroed()`] or using `MaybeUninit<T>::zeroed().assume_init()`.
1548 ///
1549 /// As const traits are not yet stable, [`pin_init::zeroed()`] can be used instead
1550 /// when initialization is required in a `const` context.
1551 ///
1552 /// # Examples
1553 ///
1554 /// ```
1555 /// use pin_init::Zeroable;
1556 ///
1557 /// #[derive(Zeroable)]
1558 /// struct Point {
1559 /// x: u32,
1560 /// y: u32,
1561 /// }
1562 ///
1563 /// let point: Point = Zeroable::zeroed();
1564 /// assert_eq!(point.x, 0);
1565 /// assert_eq!(point.y, 0);
1566 /// ```
1567 #[inline]
1568 fn zeroed() -> Self
1569 where
1570 Self: Sized,
1571 {
1572 zeroed()
1573 }
1574}
1575
1576/// Create an initializer for a zeroed `T`.
1577///
1578/// The returned initializer will write `0x00` to every byte of the given `slot`.
1579#[inline]
1580pub fn init_zeroed<T: Zeroable>() -> impl Init<T> {
1581 // SAFETY: Because `T: Zeroable`, all bytes zero is a valid bit pattern for `T`
1582 // and because we write all zeroes, the memory is initialized.
1583 unsafe {
1584 init_from_closure(|slot: *mut T| {
1585 slot.write_bytes(0, 1);
1586 Ok(())
1587 })
1588 }
1589}
1590
1591/// Create a `T` consisting of all zeroes.
1592///
1593/// Whenever a type implements [`Zeroable`], this function should be preferred over
1594/// [`core::mem::zeroed()`] or using `MaybeUninit<T>::zeroed().assume_init()`.
1595///
1596/// While const traits remain unstable, this function serves as the `const` version of
1597/// [`Zeroable::zeroed()`].
1598///
1599/// # Examples
1600///
1601/// ```
1602/// use pin_init::{Zeroable, zeroed};
1603///
1604/// #[derive(Zeroable)]
1605/// struct Point {
1606/// x: u32,
1607/// y: u32,
1608/// }
1609///
1610/// let point: Point = zeroed();
1611/// assert_eq!(point.x, 0);
1612/// assert_eq!(point.y, 0);
1613/// ```
1614#[inline]
1615pub const fn zeroed<T: Zeroable>() -> T {
1616 // SAFETY:By the type invariants of `Zeroable`, all zeroes is a valid bit pattern for `T`.
1617 unsafe { core::mem::zeroed() }
1618}
1619
1620macro_rules! impl_zeroable {
1621 ($($({$($generics:tt)*})? $t:ty, )*) => {
1622 // SAFETY: Safety comments written in the macro invocation.
1623 $(unsafe impl$($($generics)*)? Zeroable for $t {})*
1624 };
1625}
1626
1627impl_zeroable! {
1628 // SAFETY: All primitives that are allowed to be zero.
1629 bool,
1630 char,
1631 u8, u16, u32, u64, u128, usize,
1632 i8, i16, i32, i64, i128, isize,
1633 f32, f64,
1634
1635 // Note: do not add uninhabited types (such as `!` or `core::convert::Infallible`) to this list;
1636 // creating an instance of an uninhabited type is immediate undefined behavior. For more on
1637 // uninhabited/empty types, consult The Rustonomicon:
1638 // <https://doc.rust-lang.org/stable/nomicon/exotic-sizes.html#empty-types>. The Rust Reference
1639 // also has information on undefined behavior:
1640 // <https://doc.rust-lang.org/stable/reference/behavior-considered-undefined.html>.
1641 //
1642 // SAFETY: These are inhabited ZSTs; there is nothing to zero and a valid value exists.
1643 {<T: ?Sized>} PhantomData<T>, core::marker::PhantomPinned, (),
1644
1645 // SAFETY: Type is allowed to take any value, including all zeros.
1646 {<T>} MaybeUninit<T>,
1647
1648 // SAFETY: `T: Zeroable` and `UnsafeCell` is `repr(transparent)`.
1649 {<T: ?Sized + Zeroable>} UnsafeCell<T>,
1650
1651 // SAFETY: `null` pointer is valid.
1652 //
1653 // We cannot use `T: ?Sized`, since the VTABLE pointer part of fat pointers is not allowed to be
1654 // null.
1655 //
1656 // When `Pointee` gets stabilized, we could use
1657 // `T: ?Sized where <T as Pointee>::Metadata: Zeroable`
1658 {<T>} *mut T, {<T>} *const T,
1659
1660 // SAFETY: `null` pointer is valid and the metadata part of these fat pointers is allowed to be
1661 // zero.
1662 {<T>} *mut [T], {<T>} *const [T], *mut str, *const str,
1663
1664 // SAFETY: `T` is `Zeroable`.
1665 {<const N: usize, T: Zeroable>} [T; N], {<T: Zeroable>} Wrapping<T>,
1666}
1667
1668macro_rules! impl_tuple_zeroable {
1669 ($first:ident, $(,)?) => {
1670 #[cfg_attr(all(USE_RUSTC_FEATURES, doc), doc(fake_variadic))]
1671 /// Implemented for tuples up to 10 items long.
1672 // SAFETY: All elements are zeroable and padding can be zero.
1673 unsafe impl<$first: Zeroable> Zeroable for ($first,) {}
1674 };
1675 ($first:ident, $($t:ident),* $(,)?) => {
1676 #[cfg_attr(doc, doc(hidden))]
1677 // SAFETY: All elements are zeroable and padding can be zero.
1678 unsafe impl<$first: Zeroable, $($t: Zeroable),*> Zeroable for ($first, $($t),*) {}
1679 impl_tuple_zeroable!($($t),* ,);
1680 }
1681}
1682
1683impl_tuple_zeroable!(A, B, C, D, E, F, G, H, I, J);
1684
1685/// Marker trait for types that allow `Option<Self>` to be set to all zeroes in order to write
1686/// `None` to that location.
1687///
1688/// # Safety
1689///
1690/// The implementer needs to ensure that `unsafe impl Zeroable for Option<Self> {}` is sound.
1691pub unsafe trait ZeroableOption {}
1692
1693// SAFETY: by the safety requirement of `ZeroableOption`, this is valid.
1694unsafe impl<T: ZeroableOption> Zeroable for Option<T> {}
1695
1696macro_rules! impl_fn_zeroable_option {
1697 ([$($abi:literal),* $(,)?] $args:tt) => {
1698 $(impl_fn_zeroable_option!({extern $abi} $args);)*
1699 $(impl_fn_zeroable_option!({unsafe extern $abi} $args);)*
1700 };
1701 ({$($prefix:tt)*} {$(,)?}) => {};
1702 ({$($prefix:tt)*} {$ret:ident, $arg:ident $(,)?}) => {
1703 #[cfg_attr(all(USE_RUSTC_FEATURES, doc), doc(fake_variadic))]
1704 /// Implemented for function pointers with up to 20 arity.
1705 // SAFETY: function pointers are part of the option layout optimization:
1706 // <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
1707 unsafe impl<$ret, $arg> ZeroableOption for $($prefix)* fn($arg) -> $ret {}
1708 impl_fn_zeroable_option!({$($prefix)*} {$arg,});
1709 };
1710 ({$($prefix:tt)*} {$ret:ident, $($rest:ident),* $(,)?}) => {
1711 #[cfg_attr(doc, doc(hidden))]
1712 // SAFETY: function pointers are part of the option layout optimization:
1713 // <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
1714 unsafe impl<$ret, $($rest),*> ZeroableOption for $($prefix)* fn($($rest),*) -> $ret {}
1715 impl_fn_zeroable_option!({$($prefix)*} {$($rest),*,});
1716 };
1717}
1718
1719impl_fn_zeroable_option!(["Rust", "C"] { A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U });
1720
1721macro_rules! impl_zeroable_option {
1722 ($($({$($generics:tt)*})? $t:ty, )*) => {
1723 // SAFETY: Safety comments written in the macro invocation.
1724 $(unsafe impl$($($generics)*)? ZeroableOption for $t {})*
1725 };
1726}
1727
1728impl_zeroable_option! {
1729 // SAFETY: `Option<&T>` is part of the option layout optimization guarantee:
1730 // <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
1731 {<T: ?Sized>} &T,
1732 // SAFETY: `Option<&mut T>` is part of the option layout optimization guarantee:
1733 // <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
1734 {<T: ?Sized>} &mut T,
1735 // SAFETY: `Option<NonNull<T>>` is part of the option layout optimization guarantee:
1736 // <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
1737 {<T: ?Sized>} NonNull<T>,
1738 // SAFETY: All zeros is equivalent to `None` (option layout optimization guarantee:
1739 // <https://doc.rust-lang.org/stable/std/option/index.html#representation>).
1740 NonZero<u8>, NonZero<u16>, NonZero<u32>, NonZero<u64>, NonZero<u128>, NonZero<usize>,
1741 NonZero<i8>, NonZero<i16>, NonZero<i32>, NonZero<i64>, NonZero<i128>, NonZero<isize>,
1742}
1743
1744/// This trait allows creating an instance of `Self` which contains exactly one
1745/// [structurally pinned value](https://doc.rust-lang.org/std/pin/index.html#projections-and-structural-pinning).
1746///
1747/// This is useful when using wrapper `struct`s like [`UnsafeCell`] or with new-type `struct`s.
1748///
1749/// # Examples
1750///
1751/// ```
1752/// # use core::cell::UnsafeCell;
1753/// # use pin_init::{pin_data, pin_init, Wrapper};
1754///
1755/// #[pin_data]
1756/// struct Foo {}
1757///
1758/// #[pin_data]
1759/// struct Bar {
1760/// #[pin]
1761/// content: UnsafeCell<Foo>
1762/// };
1763///
1764/// let foo_initializer = pin_init!(Foo{});
1765/// let initializer = pin_init!(Bar {
1766/// content <- UnsafeCell::pin_init(foo_initializer)
1767/// });
1768/// ```
1769pub trait Wrapper<T> {
1770 /// Creates an pin-initializer for a [`Self`] containing `T` from the `value_init` initializer.
1771 fn pin_init<E>(value_init: impl PinInit<T, E>) -> impl PinInit<Self, E>;
1772}
1773
1774impl<T> Wrapper<T> for UnsafeCell<T> {
1775 #[inline]
1776 fn pin_init<E>(value_init: impl PinInit<T, E>) -> impl PinInit<Self, E> {
1777 // SAFETY: `UnsafeCell<T>` has a compatible layout to `T`.
1778 unsafe { cast_pin_init(value_init) }
1779 }
1780}
1781
1782impl<T> Wrapper<T> for MaybeUninit<T> {
1783 #[inline]
1784 fn pin_init<E>(value_init: impl PinInit<T, E>) -> impl PinInit<Self, E> {
1785 // SAFETY: `MaybeUninit<T>` has a compatible layout to `T`.
1786 unsafe { cast_pin_init(value_init) }
1787 }
1788}
1789
1790#[cfg(all(feature = "unsafe-pinned", CONFIG_RUSTC_HAS_UNSAFE_PINNED))]
1791impl<T> Wrapper<T> for core::pin::UnsafePinned<T> {
1792 #[inline]
1793 fn pin_init<E>(init: impl PinInit<T, E>) -> impl PinInit<Self, E> {
1794 // SAFETY: `UnsafePinned<T>` has a compatible layout to `T`.
1795 unsafe { cast_pin_init(init) }
1796 }
1797}