Multi Modal, Multi Lingual Location Extraction

The Multi-Lingual and Multi-Modal Location Information Extraction project aims to use both image and textual data to validate sources of online news and social media posts. Cross validation of locational information extracted from textual data and associated images may help to identify inconsistencies between reported accounts, potentially highlighting sources of online misinformation. In this regard, reliable and accurate geolocation of text and image data will play a crucial role in combatting misinformation, and in aiding counter terrorism efforts.

This project consists of two components – a multi-lingual text-based geoparser and an image-based geotagger. The text-based geoparser aims to use modern natural language processing techniques to identify place names within text and classify those places in terms of their semantic meaning within a sentence. By doing this, we can identify the geographical subject of a news article or social media post, and use the other mentioned place names to help disambiguate potential matches for the identified subject.

The image based geotagger uses points of interest and spatial features to associate an image with a pair of geographic coordinates. The model uses features such as text and signage, foliage types, and building styles to geolocate images to within a broad region. A more fine-scale geographic prediction can then be made by using spatial features which are identifiable and projectable onto a map, such as building orientations and road layouts.