Semantic and Visual Processing
One of the consequences of the digital revolution is the gradual, but inexorable availability of all kinds of text and images in a machine-readable format. Libraries around the world scan their collections. Newspapers offer their articles on the web. Governments put their archives and laws online. Users blog and upload their personal photos online. While digitization is well underway, turning the information contained in these texts and images into exploitable knowledge in the information society has become a major challenge as well as a major opportunity
Aims
The goal of the project is to carry out automatic semantic analyses across text and images and then extract usable information. In the case of text, semantics refers to the extraction of predicate—argument structures (or propositions) from text as well as entities. In images, the analyses also aim at recognizing entities and the relations between them.
Methods
We carried out an experiment to extract entities across text and images. We extracted the entity mentions from image captions and we computed a semantic similarity between the mentions and the region labels found in images using statistical associations between these mentions and the labels and syntactic relationships. We published the results in CoNLL 2015.
We also implemented a multilingual semantic role labeler to extract predicate—argument structures, which are so far only available for a handful of languages such as English or Chinese. Starting from PropBank in English and loosely parallel corpora such as versions of Wikipedia in different languages, we carried out a mapping of semantic propositions. We could train semantic parser on the generated Swedish and French.
Research group
PI:Kalle Åström
Dept. of mathematics, LTH, Lund University
PI:Pierre Nugues
Dept of Computer Science, LTH, Lund University
Magnus Oskarsson
Dept. of mathematics, LTH, Lund University
Peter Exner
Dept. of computer science, LTH, Lund University
Links and references
Improving the detection of relations between objects in an image using textual semantics..
Pattern Recognition Applications and Methods, volume 9443 of Lecture Notes in Computer Science, pages 133– 145. Ana Fred, Maria De Marsico, and Antoine Tabbone, editors, Springer International Publishing, 2015.
Peter Exner, Marcus Klang, and Pierre Nugues
CA distant supervision approach to semantic role labeling..
Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, p.239–248, Denver, Colorado, June 2015. Association for Computational Linguistics.
Rebecka Weegar, Kalle Åström, and Pierre Nugues
Linking entities across images and text.
Proceedings of the Nineteenth Conference on Computational Natural Language Learning, p.185–193, Beijing, July 2015.