SensiMARSensiMAR awarded at Conference
A publication related to the SensiMAR study was awarded as best communication of young researcher, last November 23 at the conference ICGI 2019. A proposal for designing a multisensory AR app is made and the current article presents some insights included in this awarded communcation.
SensiMAR is an outdoor multisensory AR study targeted for cultural heritage sites, aiming to provide users the chance to perceive different levels of immersion with the addition of visual stimulus, a spatialized soundscape and with related smells. Proposing such a complex system raised several questions, issues, and the need to propose a system design arose.
The goal of this research: provide knowledge to better understand if cultural heritage visits can be enhanced by having access to multisensory AR experiences in-situ.
The overall Conference
Aiming to bring together researchers, teachers and professionals in the areas of Computer Graphics, Image Processing, Computer Vision and Human-Computer Interaction, the ICGI’2019 – International Conference on Graphics and Interaction took place between November 21 and 22, 2019 at the University of Algarve in Faro, Portugal, a joint organization with the Eurographics Portuguese Chapter.
This conference allows the dissemination of work carried out or ongoing and the exchange of experiences between the academic, industrial and end-user communities.
The article entitled “SensiMAR – Designing a Multisensory AR System” was presented by the researcher Anabela Marto, the first author of this full paper, on November 21, during the first session of the event, entitled “Augmented Reality”.
It was with great pleasure that we saw this study being awarded as best communication of young researcher.
The Presented Paper
Context
Cultural heritage (CH) has been one of the many fields where technology has been increasingly requested to improve visitor’s experience in a way that they have been appealing to new technologies, such as augmented reality (AR) for the last decades to attract more visitants. In fact, AR, which differentiates from virtual reality (VR) solutions by the need to blend the real surroundingswith the virtual added content, appears to be a great advantage to use among CH sites, since, to fully take benefit of the experience, it must be enjoyed in-situ.
However, its implementation can raise several challenges, a window of opportunity that triggered the creation of an ambitious application called SensiMAR, which aims to offer an improved AR experience: a multisensory AR experience. Thus, this paper outlined and designed a multisensory AR system, intending to provide knowledge to better understand if cultural heritage visits can be enhanced by having access to multisensory AR experiences in loco.
Multisensory Applications in CH and Designing Multisensory Apps are two important topics covered in the article, providing also relevant insights about previous works – how they were implemented and how they were evaluated - that can be found in the original article.
Multisensory AR System Description
From a functional perspective, an implementation of a multisensory AR system for CH contexts should provide the user, when exploring a given site in ruins, the ability to perceive in real-time relevant virtual images to complement the regular visit: a 3D reconstruction of the site, avatars, and animations. The soundscape of the scene must be in a 3D scene, in such manner to give the user the sensation that the sounds are coming from different directions and various distances. The smells should be added to the augmented scenario at specific moments, depending on the timeline of the experience.
The following figure illustrates what the multisensory AR system should provide to the user, when implemented in a CH context, specifically, an archaeological site.
Figure 1: Illustration of what the augmented scenario of a multisensory AR system should provide to the user, considering the real and the virtual elements
The added content, corresponding to visual, audio, and smell stimuli, requires specific equipment. The article covers the technological approaches and techniques to implement this system in loco. In the overall, the user, carrying only a smartphone or a tablet in their hand, will be exposed to all three senses of this multisensory experience, in addition to the natural five stimuli coming from the surrounding environment. The app, running on the smartphone, will be responsible to trigger the other two stimuli which are included in this multisensory experience: hearing and smell.
System Evaluation
The first stage of the testing and evaluation process is to test the multisensory AR app at an archaeological site. This experiment will take place in the ancient Roman city of Conimbriga (Portugal), whose ruins are part of the Monographic Museum of Conimbriga-National Museum.
Several setups are proposed to be tested with an in-between subjects design. These setups will evaluate the sense of presence in the different levels of immersion, considering the addition of the several stimuli: visual, visual + audio, visual + smell, visual + audio + smell. And a group without any stimuli added, as a regular guided visit. With these stups, it is also intended to evaluate the gained knowledge and users satisfaction. The second stage of the evaluation includes the comparison of results coming from this multisensory AR system in loco, with a VR experience, using an HMD. The purpose of this evaluation is to compare the sense of presence between these two technologies – AR and VR.
What should participants perceive?
When designing a multisensory AR application, to define what are the virtual contents that will be added to the experience is very important. Due to the real scenario that will be also part of the experience, a balanced addition of virtual elements is described, considering the visual data, the hearing content, and the smells to be added to the experience.
In order to provide a multisensory sensation, stimuli should be evoked accordingly to the narrative in a way that all explorations are synchronized between them: images, audio, and smell. To ebale a successful communication between all elements, a storyboard was created in order to provide all actions, at every second of the experience. The following figure illustrates a sound map, to help positioning the sounds in the 3D space, as well as to know the route of several wagons that would pass by.
Figure 2: Perspective view of the sound sources for the second experiment.
The original article describes with more details all elements that will be part of the experience regarding to images, sound and smell.
Foreseeable issues
Important to highlight are the predictable issues regarding to the experience in-situ. Considering the visual stimulus, since the experience takes place outdoors, the sun may humper users’ experiences. Also unmanageable is the amount of visitors in the heritage site – the researcher must be aware that if there are a lot of visitors, the tracking experience may be affected. Another important thought must be considered regarding to the fact that the experience will be totally carried-out by the user. This means that the researcher cannot control the speed of how fast the user explores the surroundings. A developer knows that he should be careful exploring the surrounding in other to keep the tracking accurate, while the user, even if notified to be thoughtful, may not know how slow he should explore or even forget about the advice.
Regarding to sound, tests must be done in place before testing to understand how the sound levels diplay outdoors. Once again, if the place gets too crowded, it may interfere with the experience due to the noise of the surroundings.
Smell is known to be the more complex and challenging sense. Having this in mind, releasing smells outdoors has an increased difficulty related to the uncontrolled scenario, where wind and temperature are part of the experience and must be considered when implementing the system. Adding to the stated challenge, individual characteristics, such as gender, age, allergies, or smoking habits, are also moderators that may interfere with collected results.
Overall Conclusions
This study presented relevant topics regarding the amount and type of information to be added to the experience, aiming for a balanced and natural augmented scenario. The visual elements should not hamper the real perception of the space, thus the elements to be added in the scenario must be at a certain distance – not too close to the user – and must be relevant, as well as the sound and smell addition, in a manner that their presence in the experience, will provide useful and engaging information for the user.
Despite the fact of multisensory applications and AR systems being prospected with great expectations for CH sites in literature, we believe that a deeper research must be accomplished in this field to attain a solid and reliable solution. For immediate future research, it is proposed the implementation of this system, testing and evaluating the proposed variables, to solidify its strength in future implementations, as well as to validate a reliable evaluation methodology for multisensory AR systems.
For further details related to this designing process for an AR multisensory for cultural heritage, please refer to the original article.
