Universidade Nova de Lisboa, ICNOVA, iNOVA Media Lab
Fellow at CAIS from March to August 2021
The potentialities of computer vision for digital networks and online images studies are valuable assets for digital research but still unknown or little explored. Against this background, this project interrogates vision APIs for social and medium research, while developing forms of technical-methodological creation with digital methods. To that end, methods recipes and research software will be created to facilitate the study of online images through networks. Based on situated case studies, the project proposes ways of building and reading networks of images and correspondent descriptions or sites of circulation across the Web. Crucial here, I argue, is a technical understanding on Vision APIs infrastructure combined with the practice of digital methods and visual network exploration. The project attempts to minimise the difficulties found in the use of computer vision for research purposes. The expected results include replicable research models for using this approach to different objects of study.
Forthcoming presentations:
The potentialities of computer vision APIs are valuable assets for digital research but still little explored for social and medium studies. To date, script files are the main tools available to get an advantage of computer vision APIs, while graphical user interface-based tools are rare, serving only a computer vision API at a time. Against this background, the objective of this collaborative project is first to expand the features of Memespector Graphical user interface (GUI)* (Chao, 2020), an open-source research software tool originally created to repurpose Google Cloud Vision API to the study of collections of images. The new version of Memespector GUI will support more proprietary computer vision APIs such as Clarifai and Microsoft’s Cognitive Services. Moreover, Memespector GUI will support a newly-created open-source API that classifies images using multiple open-source pre-trained deep learning models. The second objective is to test, try and validate the new version of the Memespector GUI while proposing a step-by-step protocol for researchers to make use of the tool for querying proprietary and or open-source computer vision APIs. To that end, the project will compare the outputs of different machine learning models using the same collection of images. Therefore, the project attempts to analyse and compare the range, modes and granularity of image labelling and also its lack of precision relying on computing and visual methods. With robust empirical evidence, the expected results include the provision of a profile description of the labelling and semantic capacities of the proprietary and open-source computer vision APIs, expanding the analytical horizon of such algorithmic systems (see Rieder & Hofmann, 2020). Once stabilised, the new version of Memespector GUI combined with the publicly available step-by-step protocol for researchers would facilitate both the use and study of computer vision APIs for social and medium research.
Memespector Graphical User Interface (GUI): research software that supports multiple computer vision APIs developed by Jason Chao
Chao, T. H. J. (2021). Memespector GUI: Graphical User Interface Client for Computer Vision APIs (Version 0.2) [Software]. Available from https://github.com/jason-chao/memespector-gui.
Offline Image Query and Extraction Tool: a command-line tool that locates image files scattered in nested and sparse directories by filename, copies them to a new location and then inserts labels as prefixes to the image filenames. This tool serves the study of visual content for social or media research but it is not limited to this purpose, allowing researchers to explore and analyse specific collections of images on demand.
Chao, T. H. J. & Omena, J. J. (2021). Offline Image Query and Extraction Tool (Version 0.1) [Software]. Available from https://github.com/jason-chao/offline-image-query.
