Practising the Hardest Conversations in Care: VR De-escalation Training With AI Patients
Back to Blog
VR

Practising the Hardest Conversations in Care: VR De-escalation Training With AI Patients

How we built VR de-escalation training with AI patients for AZ Groeninge: a hybrid AI pipeline, a private 5G network, an emotion model and Flemish speech.

Nurses can now rehearse their most difficult conversations in VR, with an AI patient whose mood shifts with every word they say.

Aggression in the hospital is not an exception. A confused patient refuses his medication. A daughter has waited for hours in the emergency department and wants answers now. A visitor will not leave after visiting hours. Every nurse meets these moments, yet most only ever practise them on paper or in a one-off role play.

Together with AZ Groeninge, VIVES University of Applied Sciences and Proximus, OneBonsai built a training that changes that. This post covers what the training does, and then, in more detail than usual, how it works under the hood.

Part of 5G H.E.A.L.T.H.

The training is one of six use cases in 5G H.E.A.L.T.H. (High-speed Enabled Advanced Life-Saving 5G Technological Hospital), a 5G pilot project supported by the Belgian federal government and run from 2024 to 2026. The consortium, led by AZ Groeninge in Kortrijk, also includes Proximus, OneBonsai, Televic, Remote-OR and VIVES.

The project built a private 5G network across the AZ Groeninge campus and tested what it makes possible in a hospital: wireless nurse call and alarms, remote patient monitoring, live streaming from the operating room, an AI assistant that turns 2D scans into 3D models for surgical planning, and this VR training.

Four partners, one training

Each partner brought a piece the others could not.

  • AZ Groeninge brought the hospital reality: the situations its nurses actually face on the ward and in the emergency department, through its working group on aggression.
  • VIVES University of Applied Sciences brought the method. The scenarios build on its course on aggression and de-escalation skills, communication psychology and its approach to simulation-based learning.
  • Proximus provided the private 5G network and worked with us on the server, so the AI runs off the headset while the headset stays wireless.
  • OneBonsai designed and built the VR environments, the AI patients and the model behind their emotions.

Just talk

The trainee puts on a headset, reads a short briefing and starts talking. To speak to the patient, they hold a button, say what they want to say and release it. There are no scripts and no multiple-choice answers: the nurse uses their own words, as on the ward.

The briefing for the waiting-room scenario, with push-to-talk instructions
The briefing for the waiting-room scenario, with push-to-talk instructions

The training covers three situations nurses recognise at once:

  • Giving blood-pressure medication to an elderly man with dementia who is convinced he is being poisoned.
  • Calming a daughter who has waited for hours in the emergency department while her father lies on a stretcher.
  • Handling an intoxicated visitor who refuses to leave after visiting hours, until security arrives.

The first two were chosen after a literature study, together with AZ Groeninge's aggression working group and VIVES, as common and representative cases. For each scenario the team worked out the desired communication strategies, acceptable alternatives and the ones to avoid. That didactic groundwork is the basis of the VR experience and of how the patient responds.

Each session follows the rhythm of simulation training: a briefing, the scenario itself, and a debriefing where the trainee reflects on what worked and what did not.

A patient with a state of mind

The AI patient does not follow a script. It has an emotional state, and that state decides how it reacts.

We modelled that state on Russell's circumplex model of affect (1980), a well-established model from psychology. It places every emotion on two axes:

  • Valence: how pleasant or unpleasant the feeling is.
  • Arousal: how activated the person is, from calm to highly agitated.

Together they form four quadrants. An angry or anxious patient sits at high arousal and negative valence. The goal of de-escalation is to guide them towards calm and content.

Russell's circumplex model of affect: de-escalation moves the patient from angry and anxious towards calm and content
Russell's circumplex model of affect: de-escalation moves the patient from angry and anxious towards calm and content

After every sentence, the AI judges how well the nurse followed the de-escalation procedure. Did they introduce themselves, listen, acknowledge the emotion, explain clearly, offer a choice? Or did they rush, belittle, argue or threaten? Based on that judgement, the AI moves the patient's valence and arousal. The patient's next words, tone and attitude follow from where they now stand.

Keeping the state in an explicit model, rather than leaving it to a language model to decide freely how upset the patient is, keeps the patient's mood consistent from one sentence to the next, and makes it something the trainee can see.

The trainee sees the result live on an Emotional Status gauge, a simplified view of the patient's position in the model. Red means the situation is escalating; green means it is calming down.

The conversation log and the Emotional Status gauge beside the AI patient at the hospital front desk
The conversation log and the Emotional Status gauge beside the AI patient at the hospital front desk

Same patient, two conversations

The difference a few words make is easiest to see in practice.

A nurse who opens with a friendly "Hey, how are you?" gets a cautious but open answer: the patient is "alright, though still a bit unsettled". The door to a real conversation is open.

A friendly opening keeps the patient's Emotional Status gauge in the middle
A friendly opening keeps the patient's Emotional Status gauge in the middle

A nurse who walks in with "Here are your pills. Take two." gets the opposite. The patient feels ordered around, refuses flatly and lashes out. No explanation, no choice, no connection: exactly the pattern the VIVES course warns against.

A blunt instruction makes the patient refuse his medication and the gauge turns red
A blunt instruction makes the patient refuse his medication and the gauge turns red

The same happens in the waiting room. A polite but vague "let me check and get back to you" does not calm a worried daughter. What works is naming her frustration, giving concrete information about what is happening, and promising a follow-up the nurse can actually keep.

A vague answer pushes the waiting daughter's gauge towards red
A vague answer pushes the waiting daughter's gauge towards red

Under the hood

A hybrid architecture

A believable conversation needs three things at once: a scene that never stutters, speech that is understood, and an answer that comes fast enough to feel like a reply. The work is split across the headset, a server in the hospital and an online service.

Architecture of the VR de-escalation training: a standalone VR headset, a private 5G network on the AZ Groeninge campus, an on-site AI server with GPU for speech recognition, a local language model and the patient's emotional state, and an online LLM for Dutch dialogue
Architecture of the VR de-escalation training: a standalone VR headset, a private 5G network on the AZ Groeninge campus, an on-site AI server with GPU for speech recognition, a local language model and the patient's emotional state, and an online LLM for Dutch dialogue
  • The headset renders everything you see: the hospital room, the virtual patient, the gauge and the conversation log. Rendering on the headset keeps the image smooth and the experience fluid, whatever the network does.
  • An AI server on site, with a GPU, does the listening. The trainee's speech is processed there, by speech recognition and a local large language model. Keeping that step inside the hospital gives low latency and high privacy.
  • An online language model service generates the patient's rich, contextual Dutch dialogue.
  • The 5G network connects headset and server in real time. Where 5G is not available, a fast local network does the same job.

One turn of the conversation

  1. The trainee holds the button and speaks. The headset sends the audio over the private 5G network.
  2. The on-site server turns the speech into text.
  3. The AI judges the sentence against the de-escalation procedure and updates the patient's valence and arousal.
  4. The patient's reply is generated from the conversation so far, the scenario and the new emotional state.
  5. The reply goes back to the headset, where the patient responds in words and body language, and the gauge moves.

Push-to-talk also keeps turn-taking simple: the system knows exactly when a sentence starts and ends, so it never answers a half-finished thought.

Why a private 5G network

The project's private 5G network has its own core on the AZ Groeninge campus, a "network in a box" that handles data locally and keeps running even if the hospital's outside connection fails. Coverage indoors comes from distributed antenna systems, with Nokia small cells for critical rooms.

In the project's measurements, the private 5G network gave a low and consistent latency of about 20 to 27 milliseconds with almost no packet loss. The hospital's professional Wi-Fi averaged about 100 milliseconds, with occasional spikes of several seconds. For a conversation, the spikes are what hurt: a reply that sometimes arrives seconds late breaks the illusion of talking to someone.

Two more properties matter in a hospital. Devices get onto the network with a SIM rather than a shared password, with encryption built in, so only authorised devices connect. And because the core sits on campus, data is processed locally, which helps the hospital meet GDPR and its own policies.

Flemish is the hard part

Recognising Flemish speech reliably was the biggest technical challenge. Flemish varies strongly from region to region and is poorly supported internationally compared with standard Dutch or English. Nurses do not switch to textbook Dutch under pressure, nor should they have to in a training meant to feel real.

Using local AI models for part of this step made recognition more reliable. It remains the main area for further tuning, together with making the patient's verbal and non-verbal reactions ever more realistic, and steering the conversation in real time to the content, emotional charge and communication style of each trainee.

Feedback after every session

After each session the trainee gets direct feedback and a report with insights into their communication style, the choices they made and the effect those choices had on the situation.

At group level, training programmes and the hospital can follow and evaluate how skills develop across a team.

From pilot to hospital-wide training

VIVES set up a train-the-trainer programme: a group of AZ Groeninge staff learned to run the VR training, and they now lead it for their colleagues. That way the knowledge stays in the hospital instead of depending on the project team.

After each training, participants fill in an evaluation on their self-confidence, skills and overall experience. The feedback so far is clear: nurses say they feel better prepared for aggression and act more calmly after the VR exercises. Those are self-reported results, and the evaluations continue so the programme can keep improving.

Why VR and AI together

  • Safe to fail. A nurse can try the wrong approach, see the patient escalate and start again. No real patient pays the price.
  • Never the same twice. Because the patient responds to what is actually said, every conversation unfolds differently. Trainees cannot learn the answers by heart.
  • Practice on demand. No actors to book and no training day to wait for. Staff can train whenever it suits the ward.
  • Wireless and mobile. Thanks to the 5G connection and the on-site server, the heavy AI work happens off the headset, so the training can be taken wherever the nurses are.
  • Grounded in the curriculum. The patient's reactions follow the same de-escalation principles nurses learn at VIVES, so what they practise in VR is what they were taught.

Beyond the hospital

Difficult conversations are not limited to healthcare. The same approach, an AI character with a modelled state of mind inside a realistic VR setting, fits anyone who has to stay calm when the person in front of them does not: security staff, prison officers, front-desk teams, customer service.

For the wider picture, read De-escalation Training in VR: Practising With AI-Driven Characters and the de-escalation training case study.

Want to see an AI patient react to your own words? Get in touch for a demo.

References

  1. [1]Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178.
  2. [2]5G H.E.A.L.T.H. consortium (AZ Groeninge, Proximus, OneBonsai, Remote-OR, Televic, VIVES). 5G H.E.A.L.T.H. Whitepaper 2023–2026.
  3. [3]Moore, N., Ahmadpour, N., Brown, M., Poronnik, P., & Davids, J. (2022). Designing Virtual Reality–Based Conversational Agents to Train Clinicians in Verbal De-escalation Skills: Exploratory Usability Study. JMIR Serious Games, 10(3), e38669. https://doi.org/10.2196/38669
  4. [4]Hsu, R., McCabe, C., Ellison, C., Howells, M., & Loetscher, T. (2024). Bridging the Gap: Virtual Reality as a Tool for De-Escalation Training in Disability Support Settings. Multimodal Technologies and Interaction, 8(11), 100. https://doi.org/10.3390/mti8110100

About the Author

Evarest Schoofs

About Evarest Schoofs

CEO of OneBonsai, the AI and software company in Leuven that builds VR training, training simulators and production AI. He is driven by technology and how it gets put to work: he loves new gadgets, builds his own tools, and is always looking for ways to make systems work better with tech.

Evarest Schoofs on LinkedIn

Ready to talk it through?

Let's discuss how this applies to your organization