diff options
| -rw-r--r-- | docs/img/argaze_load_gui_opencv_frame.png | bin | 0 -> 339281 bytes | |||
| -rw-r--r-- | docs/img/argaze_load_gui_opencv_pipeline.png | bin | 0 -> 517856 bytes | |||
| -rw-r--r-- | docs/index.md | 6 | ||||
| -rw-r--r-- | docs/user_guide/aruco_marker_pipeline/advanced_topics/scripting.md | 77 | ||||
| -rw-r--r-- | docs/user_guide/aruco_marker_pipeline/aoi_3d_frame.md | 37 | ||||
| -rw-r--r-- | docs/user_guide/aruco_marker_pipeline/configuration_and_execution.md | 88 |
6 files changed, 105 insertions, 103 deletions
diff --git a/docs/img/argaze_load_gui_opencv_frame.png b/docs/img/argaze_load_gui_opencv_frame.png Binary files differnew file mode 100644 index 0000000..3ab3b5e --- /dev/null +++ b/docs/img/argaze_load_gui_opencv_frame.png diff --git a/docs/img/argaze_load_gui_opencv_pipeline.png b/docs/img/argaze_load_gui_opencv_pipeline.png Binary files differnew file mode 100644 index 0000000..227a91d --- /dev/null +++ b/docs/img/argaze_load_gui_opencv_pipeline.png diff --git a/docs/index.md b/docs/index.md index 00b8ed7..2e5a99d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -43,3 +43,9 @@ This ArUco marker pipeline can be combined with any wearable eye tracking device !!! note *ArUco marker pipeline is greatly inspired by [Andrew T. Duchowski, Vsevolod Peysakhovich and Krzysztof Krejtz article](https://git.recherche.enac.fr/attachments/download/1990/Using_Pose_Estimation_to_Map_Gaze_to_Detected_Fidu.pdf) about using pose estimation to map gaze to detected fiducial markers.* + +## Demonstration + + + +[Test **ArGaze** by reading the dedicated user guide section](./user_guide/utils/demonstrations_scripts.md).
\ No newline at end of file diff --git a/docs/user_guide/aruco_marker_pipeline/advanced_topics/scripting.md b/docs/user_guide/aruco_marker_pipeline/advanced_topics/scripting.md index c81d57d..d3a340b 100644 --- a/docs/user_guide/aruco_marker_pipeline/advanced_topics/scripting.md +++ b/docs/user_guide/aruco_marker_pipeline/advanced_topics/scripting.md @@ -74,35 +74,80 @@ from argaze import ArFeatures ... ``` -## Pipeline execution outputs +## Pipeline execution -The [ArUcoCamera.watch](../../../argaze.md/#argaze.ArFeatures.ArCamera.watch) method returns data about pipeline execution. +### Detect ArUco markers, estimate scene pose and project 3D AOI + +Pass each camera image with timestamp information to the [ArUcoCamera.watch](../../argaze.md/#argaze.ArFeatures.ArCamera.watch) method to execute the whole pipeline dedicated to ArUco marker detection, scene pose estimation and 3D AOI projection. + +!!! warning "Mandatory" + + The [ArUcoCamera.watch](../../argaze.md/#argaze.ArFeatures.ArCamera.watch) method must be called from a *try* block to catch pipeline exceptions. ```python -# Assuming that timestamped images are available +# Assuming that Full HD (1920x1080) images are available with timestamp values ...: + # Edit timestamped image + timestamped_image = DataFeatures.TimestampedImage(image, timestamp=timestamp) + try: - # Watch image with ArUco camera - aruco_camera.watch(image, timestamp=timestamp) + # Detect ArUco markers, estimate scene pose then, project 3D AOI into camera frame + aruco_camera.watch(timestamped_image) # Do something with pipeline exception except Exception as e: ... - # Do something with detected_markers - ... aruco_camera.aruco_detector.detected_markers() + # Display ArUcoCamera frame image to display detected ArUco markers, scene pose, 2D AOI projection and ArFrame visualization. + ... aruco_camera.image() +``` + +### Detection outputs + +The [ArUcoCamera.watch](../../../argaze.md/#argaze.ArFeatures.ArCamera.watch) method returns data about pipeline execution. + +```python +# Assuming that watch method has been called + +# Do something with detected_markers +... aruco_camera.aruco_detector.detected_markers() ``` Let's understand the meaning of each returned data. -### *aruco_camera.aruco_detector.detected_markers()* +#### *aruco_camera.aruco_detector.detected_markers()* A dictionary containing all detected markers is provided by [ArUcoDetector](../../../argaze.md/#argaze.ArUcoMarker.ArUcoDetector) class. +### Analyse timestamped gaze positions into the camera frame + +As mentioned above, [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) inherits from [ArFrame](../../argaze.md/#argaze.ArFeatures.ArFrame) and, so, benefits from all the services described in the [gaze analysis pipeline section](../gaze_analysis_pipeline/introduction.md). + +Particularly, timestamped gaze positions can be passed one by one to the [ArUcoCamera.look](../../argaze.md/#argaze.ArFeatures.ArFrame.look) method to execute the whole pipeline dedicated to gaze analysis. + +!!! warning "Mandatory" + + The [ArUcoCamera.look](../../argaze.md/#argaze.ArFeatures.ArFrame.look) method must be called from a *try* block to catch pipeline exceptions. + +```python +# Assuming that timestamped gaze positions are available +... + + try: + + # Look ArUcoCamera frame at a timestamped gaze position + aruco_camera.look(timestamped_gaze_position) + + # Do something with pipeline exception + except Exception as e: + + ... +``` + ## Setup ArUcoCamera image parameters Specific [ArUcoCamera.image](../../../argaze.md/#argaze.ArFeatures.ArFrame.image) method parameters can be configured thanks to a Python dictionary. @@ -133,4 +178,18 @@ aruco_camera_image = aruco_camera.image(**image_parameters) ``` !!! note - [ArUcoCamera](../../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) inherits from [ArFrame](../../../argaze.md/#argaze.ArFeatures.ArFrame) and, so, benefits from all image parameters described in [gaze analysis pipeline visualization section](../../gaze_analysis_pipeline/visualization.md).
\ No newline at end of file + [ArUcoCamera](../../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) inherits from [ArFrame](../../../argaze.md/#argaze.ArFeatures.ArFrame) and, so, benefits from all image parameters described in [gaze analysis pipeline visualization section](../../gaze_analysis_pipeline/visualization.md). + + +## Display ArUcoScene frames + +All [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) frames image can be displayed as any [ArFrame](../../argaze.md/#argaze.ArFeatures.ArFrame). + +```python + ... + + # Display all ArUcoScene frames + for frame in aruco_camera.scene_frames(): + + ... frame.image() +```
\ No newline at end of file diff --git a/docs/user_guide/aruco_marker_pipeline/aoi_3d_frame.md b/docs/user_guide/aruco_marker_pipeline/aoi_3d_frame.md index 7323f2e..3a029b0 100644 --- a/docs/user_guide/aruco_marker_pipeline/aoi_3d_frame.md +++ b/docs/user_guide/aruco_marker_pipeline/aoi_3d_frame.md @@ -69,7 +69,8 @@ Here is the previous extract where "Left_Screen" and "Right_Screen" AOI are defi } } } - } + }, + "copy_background_into_scenes_frames": true ... } } @@ -96,40 +97,18 @@ The names of 3D AOI **and** their related [ArFrames](../../argaze.md/#argaze.ArF [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) frame layers are projected into their dedicated [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) layers when the JSON configuration file is loaded. -## Pipeline execution - -### Map ArUcoCamera image into ArUcoScenes frames - -After the timestamped camera image is passed to the [ArUcoCamera.watch](../../argaze.md/#argaze.ArFeatures.ArCamera.watch) method, it is possible to apply a perspective transformation in order to project the watched image into each [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) [frame's background](../../argaze.md/#argaze.ArFeatures.ArFrame) image. - -```python -# Assuming that Full HD (1920x1080) timestamped images are available -...: - - # Detect ArUco markers, estimate scene pose then, project 3D AOI into camera frame - aruco_camera.watch(timestamped_image) +### *copy_background_into_scenes_frames* - # Map watched image into ArUcoScene frames background - aruco_camera.map(timestamp=timestamp) -``` +When the timestamped camera image is passed to the [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) method, it is possible to apply a perspective transformation in order to project the watched image into each [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) [frame's background](../../argaze.md/#argaze.ArFeatures.ArFrame) image. -### Analyze timestamped gaze positions into ArUcoScene frames +## Pipeline execution [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) frames benefits from all the services described in the [gaze analysis pipeline section](../gaze_analysis_pipeline/introduction.md). !!! note - Timestamped [GazePositions](../../argaze.md/#argaze.GazeFeatures.GazePosition) passed to the [ArUcoCamera.look](../../argaze.md/#argaze.ArFeatures.ArFrame.look) method are projected into [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) frames if applicable. - -### Display each ArUcoScene frames - -All [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) frames image can be displayed as any [ArFrame](../../argaze.md/#argaze.ArFeatures.ArFrame). - -```python - ... + Timestamped [GazePositions](../../argaze.md/#argaze.GazeFeatures.GazePosition) passed to the [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) are automatically projected into [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) frames if applicable. - # Display all ArUcoScene frames - for frame in aruco_camera.scene_frames(): +Each [ArUcoScene](../../argaze.md/#argaze.ArUcoMarker.ArUcoScene) frames image is displayed in a separate window. - ... frame.image() -```
\ No newline at end of file +
\ No newline at end of file diff --git a/docs/user_guide/aruco_marker_pipeline/configuration_and_execution.md b/docs/user_guide/aruco_marker_pipeline/configuration_and_execution.md index f4bd2d4..c2ee1b9 100644 --- a/docs/user_guide/aruco_marker_pipeline/configuration_and_execution.md +++ b/docs/user_guide/aruco_marker_pipeline/configuration_and_execution.md @@ -1,17 +1,17 @@ -Load and execute pipeline +Edit and execute pipeline ========================= Once [ArUco markers are placed into a scene](aruco_marker_description.md), they can be detected thanks to [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) class. As [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) inherits from [ArFrame](../../argaze.md/#argaze.ArFeatures.ArFrame), the [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) class also benefits from all the services described in the [gaze analysis pipeline section](../gaze_analysis_pipeline/introduction.md). - +Once defined, an ArUco marker pipeline needs to embedded inside a context that will provides it both gaze positions and camera images to process. -## Load JSON configuration file + -An [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) pipeline can be loaded from a JSON configuration file thanks to [argaze.load](../../argaze.md/#argaze.load) package method. +## Edit JSON configuration -Here is a simple JSON [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) configuration file example: +Here is a simple JSON [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) configuration example: ```json { @@ -52,19 +52,7 @@ Here is a simple JSON [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCam } ``` -Then, here is how to load the JSON file: - -```python -import argaze - -# Load ArUcoCamera -with argaze.load('./configuration.json') as aruco_camera: - - # Do something with ArUcoCamera - ... -``` - -Now, let's understand the meaning of each JSON entry. +Let's understand the meaning of each JSON entry. ### argaze.ArUcoMarker.ArUcoCamera.ArUcoCamera @@ -101,62 +89,32 @@ The usual [ArFrame visualization parameters](../gaze_analysis_pipeline/visualiza ## Pipeline execution -### Detect ArUco markers, estimate scene pose and project 3D AOI - -Pass each camera image with timestamp information to the [ArUcoCamera.watch](../../argaze.md/#argaze.ArFeatures.ArCamera.watch) method to execute the whole pipeline dedicated to ArUco marker detection, scene pose estimation and 3D AOI projection. - -!!! warning "Mandatory" - - The [ArUcoCamera.watch](../../argaze.md/#argaze.ArFeatures.ArCamera.watch) method must be called from a *try* block to catch pipeline exceptions. - -```python -# Assuming that Full HD (1920x1080) images are available with timestamp values -...: - - # Edit timestamped image - timestamped_image = DataFeatures.TimestampedImage(image, timestamp=timestamp) - - try: +A pipeline needs to be embedded into a context to be executed. - # Detect ArUco markers, estimate scene pose then, project 3D AOI into camera frame - aruco_camera.watch(timestamped_image) +Copy the gaze analysis pipeline configuration defined above inside the following context configuration. - # Do something with pipeline exception - except Exception as e: - - ... - - # Display ArUcoCamera frame image to display detected ArUco markers, scene pose, 2D AOI projection and ArFrame visualization. - ... aruco_camera.image() +```json +{ + "argaze.utils.contexts.OpenCV.Movie": { + "name": "Movie player", + "path": "./src/argaze/utils/demo/tobii_record/segments/1/fullstream.mp4", + "pipeline": JSON CONFIGURATION + } +} ``` -### Analyse timestamped gaze positions into the camera frame - -As mentioned above, [ArUcoCamera](../../argaze.md/#argaze.ArUcoMarker.ArUcoCamera) inherits from [ArFrame](../../argaze.md/#argaze.ArFeatures.ArFrame) and, so, benefits from all the services described in the [gaze analysis pipeline section](../gaze_analysis_pipeline/introduction.md). - -Particularly, timestamped gaze positions can be passed one by one to the [ArUcoCamera.look](../../argaze.md/#argaze.ArFeatures.ArFrame.look) method to execute the whole pipeline dedicated to gaze analysis. +Then, use the [*load* command](../utils/main_commands.md) to execute the context. -!!! warning "Mandatory" - - The [ArUcoCamera.look](../../argaze.md/#argaze.ArFeatures.ArFrame.look) method must be called from a *try* block to catch pipeline exceptions. - -```python -# Assuming that timestamped gaze positions are available -... - - try: +```shell +python -m argaze load CONFIGURATION +``` - # Look ArUcoCamera frame at a timestamped gaze position - aruco_camera.look(timestamped_gaze_position) +This command should open a GUI window with the detected markers and identified cursor fixations circles when the mouse moves over the window. - # Do something with pipeline exception - except Exception as e: - - ... -``` + !!! note "" - At this point, the [ArUcoCamera.watch](../../argaze.md/#argaze.ArFeatures.ArCamera.watch) method only detects ArUco marker and the [ArUcoCamera.look](../../argaze.md/#argaze.ArFeatures.ArCamera.look) method only processes gaze movement identification without any AOI support as no scene description is provided into the JSON configuration file. + At this point, the pipeline only processes gaze movement identification without any AOI support as no scene description is provided into the JSON configuration file. Read the next chapters to learn [how to estimate scene pose](pose_estimation.md), [how to describe a 3D scene's AOI](aoi_3d_description.md) and [how to project them into the camera frame](aoi_3d_projection.md).
\ No newline at end of file |
