In this blog, I just want to talk about my recently accepted research paper, which is part of my PhD studying Mixed Reality Digital Twins (MRDTs). My research is situated in the Human-Computer Interaction (HCI) space and the philosophy of technology.
Mixed reality (MR) is part of the immersive technology of extended reality (XR). Digital twins (DTs) is its own established technology where a physical asset has a virtual counterpart, using bi-directional data to keep it live and transfer those physical properties onto the digital. When you combine both, you get this kind of real-time data stream hologram representation, hence Mixed Reality Digital Twins
The Blind Spot in the Literature
What I discovered is that these systems are usually assessed purely based on system performance. Even though developers and researchers term this “user-experience (UX) centred design,” there is a major blind spot: there is very little literature accounting for the actual human user and their own lived experience.
To bridge this gap, I am adopting a framework to study these systems in a more phenomenological way. Specifically, I’m using the post-phenomenology framework, which studies the relations between the user and their world as mediated by technology.
In this space, philosopher Don Ihde specifies four core human-technology relations:
- Embodiment
- Hermeneutic
- Alterity
- Background
My core proposal and the main contribution of this paper is that mixed reality digital twins actually offer a concurrent, simultaneous activation of all four of these relations at the exact same time. All of these relations are at play at once, creating what I call a Blended Life-World.
Defining the “Blended Life-World”
The term “Life-World” is a direct homage to Edmund Husserl’s lebenswelt, where he talks about our pre-reflective experience or experiencing the world exactly as it is presented to us.
With mixed reality, the physical blends with the virtual spatial environment; the user experiences both the real environment and the virtual one through a headset (I am using the HoloLens 2). At the same time, the DT side blends physical and digital assets together through real-time data. When a user puts on the headset and interacts with the physical counterpart, you can see all four relations collapse into two simultaneous experiences:
- Perceptual Blending (Embodiment + Hermeneutic)
You embody the HoloLens so completely that it becomes part of you and recedes into the background. It becomes transparent because you are quite literally perceiving the world through the lens. Yet, at the same time, you are actively interpreting the data and information being presented to you through that digital overlay. - Operational Reciprocity (Alterity + Background)
You are interacting with a hologram as a quasi-other (alterity), while the digital twin’s background relation silently maintains the real-time, bi-directional communication infrastructure.
Ultimately, the technology is no longer just a separate tool. It becomes the very medium through which a cyber-physical reality is revealed to the user.
The Empirical Study: 120 Lines of Code
To validate this conceptual proposal, I put users through a three-phase interview process to map their empirical experiences:
- Phase 1 (Expectations): Establishing the user’s prior worldview regarding digital-physical boundaries and their familiarity with XR.
- Phase 2 (Lived Experience): Having them describe exactly what they are experiencing while actively interacting with the MRDT.
- Phase 3 (Relational Recall): Asking open-ended questions after the experience to capture their insights.
After coding 120 lines of user accounts, where I tracked exactly where these relations intersect, four dominant themes emerged:
Embodied Data Perception
A transition from the cognitive decoding of units to a direct sensory feeling of data. And because my MRDT design visualised temperature sensors as particles that change colour or accelerate, the users noted in their own words that they could literally “feel the data.”
Reciprocal Agency Loop
A bi-directional synchronicity where digital and physical actions are perceived as a singular agency. When they moved the physical object, the virtual object moved, and vice versa. This is where a relational flip happens, where they no longer see two separate entities, but one.
Safe Mediation Through the Twin
Users often perceive physical assets as somewhat fragile. Yet, seeing this digital proxy makes them feel more inclined to explore and understand it. They are less scared because it feels enticing and palatable. Participants said things like,
“I was afraid of breaking it, but then I could move it. There’s freedom to explore and grab it with your fingertips without worrying about destroying the physical.”
Relational Breakdown
This is when the illusion of unity breaks. Due to latency or a bit of jitter from the background relation, the illusion shatters. It’s like someone wearing glasses; they only notice the glasses when they fall off their face. Latency affects far more than just the graphical representation; it breaks the blended entity.
Reflections and Future Work
As the designer of the MRDT myself, I have to acknowledge my own design choices and biases. I assumed that temperature sensors needed to be visualised as particles that change colour or accelerate, and I used an accelerometer so that changing the physical three-axis would move the virtual object instantly. All of this happens incredibly fast. It is a conceptual proposal, but it is grounded in a highly empirical study that validates it.
I wanted to document my whole feeling journey through developing this first paper and navigating the interview process. To be honest, I don’t always feel confident executing it, and going to France to present this paper makes me feel a bit out of my depth! But looking back at the script I’ve put together, I realise I have a much better grasp of these frameworks than I thought.
The definite next step for my future work is moving into micro-phenomenology. For my next paper, I want to upgrade the MRDT and the interviews so they aren’t so retrospective or structured like a “think-aloud” usability test. I want to home in on the actual, granular moment in time of their experience, evoking the immediate experience and emotions rather than a post-rationalisation.
But for now, it’s time to take this first paper to France!
