diff options
Diffstat (limited to 'docs/use_cases/pilot_gaze_monitoring/introduction.md')
| -rw-r--r-- | docs/use_cases/pilot_gaze_monitoring/introduction.md | 43 |
1 files changed, 43 insertions, 0 deletions
diff --git a/docs/use_cases/pilot_gaze_monitoring/introduction.md b/docs/use_cases/pilot_gaze_monitoring/introduction.md new file mode 100644 index 0000000..0faf4b1 --- /dev/null +++ b/docs/use_cases/pilot_gaze_monitoring/introduction.md @@ -0,0 +1,43 @@ +Overview +======== + +**ArGaze** enabled a cognitive assistant to support a pilot's situation awareness. + +The following use case has integrated the [ArUco marker pipeline](../../user_guide/aruco_marker_pipeline/introduction.md) to map pilot gaze onto many cockpit instruments in real-time and then enable AOI fixation matching with the [gaze analysis pipeline](../../user_guide/gaze_analysis_pipeline/introduction.md). + +## Background + +The [HAIKU project](https://haikuproject.eu/) aims to pave the way for human-centric intelligent assistants in the aviation domain by supporting, among others, the pilot during startle or surprise events. +One of the features provided by the assistant through **ArGaze** is a situation awareness support that ensures the pilot updates his awareness of the aircraft state by monitoring his gaze and flight parameters. +When this support is active, relevant information is highlighted on the Primary Flight Display (PFD). + + + +## Experiment context + +Pilot eye tracking data were provided by Tobii Pro Glasses 2, a head-mounted eye tracker. +The gaze and scene camera video were captured through the Tobii SDK and processed in real-time on NVIDIA Jetson Xavier. +Since the eye tracker model is head-mounted, ArUco markers were placed at various locations within a A320 cockpit simulator to ensure that several of them were constantly visible in the field of view of the eye tracker camera. + + + +The [ArUco marker pipeline](../../user_guide/aruco_marker_pipeline/introduction.md) has enabled real-time gaze mapping onto multiple screens and panels around pilot-in-command position while [gaze analysis pipeline](../../user_guide/gaze_analysis_pipeline/introduction.md) was identifying fixations and matching them with dynamic AOIs related to each instruments. +To identify the relevant AOIs, a 3D model of the cockpit describing the AOI and the position of the markers has been realized. + + + +Finally, fixation events were sent in real-time through [Ivy bus middleware](https://gitlab.com/ivybus/ivy-python/) to the situation awareness software in charge of displaying attention getter onto the PFD screen. + +## ArGaze setup + +The project defines 3 main files to integrate **ArGaze** to the experiment: + +* The context that captures gaze data and scene camera video: [live_streaming_context.json](context.md) +* The pipeline that processes gaze data and scene camera video: [live_processing_pipeline.json](pipeline.md) +* The observers that send fixation events to Ivy bus middleware: [observers.py](observers.md) + +The project is loaded by executing the following command: + +```shell +python -m argaze load live_streaming_context.json +``` |