In addition to serving as a means for documenting events and capturing memories, digital images can help facilitate decision making. The use of digital images range from personal photos and social media to more complex applications in education and medicine. To this end, we provide the research community with access to the computational framework. The framework can potentially be applied to any multimodal data stream and to any visual domain. The alignments produced by our framework can be used to create a database of low-level image features and high-level semantic annotations corresponding to perceptually important image regions. We also find differences in system performances when identifying image regions using clustering methods that rely on gaze information rather than image features. The accuracy of these labels exceeds that of baseline alignments obtained using purely temporal correspondence between fixations and words. The resulting multimodal alignments are then used to annotate image regions with linguistic labels. Using an unsupervised bitext alignment algorithm originally developed for machine translation, we create meaningful mappings between participants’ eye movements over an image and their spoken descriptions of that image. Our work relies on the notion that gaze and spoken narratives can jointly model how humans inspect and analyze images. To begin to bridge this gap, we propose a framework that integrates human-elicited gaze and spoken language to label perceptually important regions in an image.
Despite many recent advances in the field of computer vision, there remains a disconnect between how computers process images and how humans understand them.
0 kommentar(er)
