COMPUTER VISION

Tutorials

Chollet, F. (2017). Deep learning with Python.

Rosebrock, A. (2017). Deep learning for computer vision with Python.

Solem, J. E. (2012). Programming computer vision with Python: Tools and algorithms for analyzing images.

Williams, N. W., Casas, A., & Wilkerson, J. D. Images as data for social science research: An introduction to Convolutional Neural Nets for image classification.

Stanford: CS231n: Convolutional Neural Networks for Visual Recognition

Neural Networks and Deep Learning

A Neural Network Playground

Libraries and packages

Keras: a library for building neural networks.

Pillow (or Python Imaging Library): a library for opening, manipulating, and saving many different image file formats.

scikit-image: A collection of algorithms for image processing, providing functions like image segmentation, edge detection, feature detection, geometrical transformations, etc.

OpenCV: a library for computer vision, image processing, and machine learning.

OpenFace: a facial recognition and analysis library. It provides the detection of facial action units and is particularly useful for emotional analysis.

OpenPose: a library for body detection.

VGG Image Annotator: An offline image annotation tool. It creates an easy-to-use interface for coders to annotate images or videos on their computers.

Computer vision APIs

Microsoft Azure: object recognition, facial detection/recognition/analysis, customized image analysis, image captioning

Face++: facial detection/recognition/analysis, body detection, gesture analysis

Clarifai: object recognition, customized image analysis.

CloudSight: object recognition, image captioning

Google Vision: object recognition

Amazon Rekognition: facial detection/recognition/analysis, object recognition, video analysis

IBM Watson: object recognition

Sighthound: facial detection/recognition/analysis, vehicle analysis

Academic papers

Here is a list of academic papers using computer vision methods to answer questions that are relevant to social sciences (e.g., communication, psychology, political science, economics, etc.)

  • Lee, B., Seo, M. K., Kim, D., Shin, I. S., Schich, M., Jeong, H., & Han, S. K. (2020). Dissecting landscape art history with information theory. Proceedings of the National Academy of Sciences.
  • Peng, Y. (2020). What makes politicians’ Instagram posts popular? Analyzing social media strategies of candidates and office holders with computer vision. The International Journal of Press/Politics.
  • Xi, N., Ma, D., Liou, M., Steinert-Threlkeld, Z. C., Anastasopoulos, J., & Joo, J. (2020, May). Understanding the Political Ideology of Legislators from Social Media Images. In Proceedings of the International AAAI Conference on Web and Social Media (pp. 726-737).
  • Haim, M., & Jungblut, M. (2020). Politicians’ self-depiction and their news portrayal: Evidence from 28 countries using visual computational analysis. Political Communication.
  • Araujo, T., Lock, I., & van de Velde, B. (2020). Automated visual content analysis (AVCA) in communication research: A protocol for large scale image classification with pre-trained computer vision models. Communication Methods and Measures.
  • Joo, J., Bucy, E. P., & Seidel, C. (2019). Automated coding of televised leader displays: Detecting nonverbal political behavior with computer vision and deep learning. International Journal of Communication.
  • Fridkin, K. L., Gershon, S. A., Courey, J., & LaPlant, K. (2019). Gender differences in emotional reactions to the first 2016 presidential debate. Political Behavior, 1–31.
  • Guntuku, S. C., Preotiuc-Pietro, D., Eichstaedt, J. C., & Ungar, L. H. (2019, July). What twitter profile and posted images reveal about depression and anxiety. In Proceedings of the International AAAI Conference on Web and Social Media (pp. 236–246).
  • Zhang, H., & Pan, J. (2019). Casm: A deep-learning approach for identifying collective action events with text and image data from social media. Sociological Methodology, 49(1), 1-57.
  • Campbell, K., Carpenter, K. L., Hashemi, J., Espinosa, S., Marsan, S., Borg, J. S., … & Tepper, M. (2019). Computer vision analysis captures atypical attention in toddlers with autism. Autism, 23(3), 619–628.
  • Matz, S. C., Segalin, C., Stillwell, D., Müller, S. R., & Bos, M. W. (2019). Predicting the personal appeal of marketing images using computational methods. Journal of Consumer Psychology, 29(3), 370-390.
  • Mancosu, M., & Bobba, G. (2019). Using deep-learning algorithms to derive basic characteristics of social media users: The Brexit campaign as a case study. PloS one, 14(1), e0211013.
  • Peng, Y. (2018). Same candidates, different faces: Uncovering media bias in visual portrayals of presidential candidates with computer vision. Journal of Communication, 68(5), 920–941.
  • Peng, Y. & Jemmott III, J. (2018). Feast for the eyes: Effects of food perceptions and computer vision features on food photo popularity. International Journal of Communication, 12, 313–336.
  • Wang, Y., & Kosinski, M. (2018). Deep neural networks are more accurate than humans at detecting sexual orientation from facial images. Journal of Personality and Social Psychology, 114(2), 246–257.
  • Kim, Y., & Kim, J. H. (2018). Using computer vision techniques on Instagram to link users’ personalities and genders to the features of their photos: An exploratory study. Information Processing & Management, 54(6), 1101–1114.
  • Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., Aiden, E. L., & Fei-Fei, L. (2017). Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States. Proceedings of the National Academy of Sciences, 114(50), 13108–13113.
  • Reece, A. G., & Danforth, C. M. (2017). Instagram photos reveal predictive markers of depression. EPJ Data Science, 6(1), 15.
  • Wei, X., & Stillwell, D. (2017, February). How smart does your profile image look? Estimating intelligence from social network profile images. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (pp. 33–40). New York, NY: ACM.
  • Naik, N., Kominers, S. D., Raskar, R., Glaeser, E. L., & Hidalgo, C. A. (2017). Computer vision uncovers predictors of physical urban change. Proceedings of the National Academy of Sciences, 114(29), 7571–7576.
  • Manikonda, L., & De Choudhury, M. (2017, May). Modeling and understanding visual attributes of mental health disclosures in social media. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 170–181). New York, NY: ACM.
  • Gygli, M., & Soleymani, M. (2016, October). Analyzing and predicting GIF interestingness. In Proceedings of the 2016 ACM on Multimedia Conference (pp. 122–126). New York, NY: ACM.
  • Bakhshi, S., Shamma, D. A., Kennedy, L., Song, Y., de Juan, P., & Kaye, J. J. (2016, May). Fast, cheap, and good: Why animated GIFs engage us. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 575–586). New York, NY: ACM.
  • Liu, L., Preotiuc-Pietro, D., Samani, Z. R., Moghaddam, M. E., & Ungar, L. (2016). Analyzing personality through social media profile picture choice. In Proceedings of the Tenth International AAAI Conference on Web and Social Media. (pp. 211–220). Cologne, Germany: AAAI.
  • Joo, J., Steen, F. F., & Zhu, S. C. (2015). Automated facial trait judgment and election outcome prediction: Social dimensions of face. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3712-3720). IEEE.
  • Abdullah, S., Murnane, E. L., Costa, J. M., & Choudhury, T. (2015, February). Collective smile: Measuring societal happiness from geolocated images. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 361–374). New York, NY: ACM.
  • Bakhshi, S., & Gilbert, E. (2015). Red, purple and pink: The colors of diffusion on Pinterest. PLOS ONE, 10(2), e0117148.
  • Deza, A., & Parikh, D. (2015). Understanding image virality. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1818–1826). Boston, MA: IEEE.
  • Gelli, F., Uricchio, T., Bertini, M., Del Bimbo, A., & Chang, S. F. (2015). Image popularity prediction in social media using sentiment and context features. In Proceedings of the ACM Conference on Multimedia (pp. 907–910). New York, NY: ACM.
  • Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S. (2016). Combining satellite imagery and machine learning to predict poverty. Science, 353(6301), 790–794.
  • Joo, J., Li, W., Steen, F. F., & Zhu, S. C. (2014). Visual persuasion: Inferring communicative intents of images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 216-223). IEEE.
  • Khosla, A., Das Sarma, A., & Hamid, R. (2014, April). What makes an image popular?. In Proceedings of the 23rd International Conference on World Wide Web (pp. 867–876). New York, NY: ACM.
  • Totti, L. C., Costa, F. A., Avila, S., Valle, E., Meira Jr, W., & Almeida, V. (2014). The impact of visual attributes on online image diffusion. In Proceedings of the ACM Conference on Web Science (pp. 42–51). New York, NY: ACM.
  • Bakhshi, S., Shamma, D. A., & Gilbert, E. (2014, April). Faces engage us: Photos with faces attract more likes and comments on Instagram. In Proceedings of the 32nd ACM Conference on Human Factors in Computing Systems (pp. 965–974). New York, NY: ACM.
  • Horiuchi, Y., Komatsu, T., & Nakaya, F. (2012). Should candidates smile to win elections? An application of automated face recognition technology. Political Psychology, 33(6), 925–933.
  • Machajdik, J., & Hanbury, A. (2010). Affective image classification using features inspired by psychology and art theory. In Proceedings of the ACM International Conference on Multimedia (pp. 83–92). New York, NY: ACM.
  • Datta, R., Joshi, D., Li, J., & Wang, J. Z. (2006, May). Studying aesthetics in photographic images using a computational approach. In Proceedings of the European Conference on Computer Vision (pp. 288–301). Berlin, Germany: Springer.
  • Ke, Y., Tang, X., & Jing, F. (2006, June). The design of high-level features for photo quality assessment. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 419–426). New York, NY: IEEE.