An excellent point! One that raises so many different answers. Transfer of consciousness, this is all just one exceedingly long and complicated shared dream, we're all a simulacrum, clones, the list goes on and on.

After all, our memories vanish when we return home but certain other things vanish as well. [ And here, the Doctor reaches up to the tattoo on his wrist, blazing 'UNSETTLED' and lightly taps it. ]