I’ve compiled a list of resources for social scientists who are interested in computer vision methods. Please feel free to contact me if you think some things need to be added to this list.

Libraries and packages

Pillow (or Python Imaging Library): a library for Python that adds support for opening, manipulating, and saving many different image file formats.

scikit-image: A collection of algorithms for image processing, providing functions like image segmentation, edge detection, feature detection, geometrical transformations, and so on.

OpenCV: a leading open source library for computer vision, image processing, and machine learning. It has a strong focus on real-time applications.

Keras: a Python library for building neural networks. It is capable of running on top of TensorFlow, Microsoft Cognitive Toolkit or Theano.

Computer vision APIs

Microsoft Azure: object recognition, facial detection/recognition/analysis, customized image analysis, image captioning

Face++: facial detection/recognition/analysis, body detection, gesture analysis

Clarifai: object recognition, customized image analysis.

CloudSight: object recognition, image captioning

Google Vision: object recognition

Amazon Rekognition: facial detection/recognition/analysis, object recognition, video analysis

IBM Watson: object recognition

Sighthound: facial detection/recognition/analysis, vehicle analysis


Neural Networks and Deep Learning
A Neural Network Playground

Academic papers

Here is a list of academic papers using computer vision methods to answer questions that are relevant to social science.

Social science (including communication, psychology, political science, economics, etc.)

  • Peng, Y. (2018). Same candidates, different faces: Uncovering media bias in visual portrayals of presidential candidates with computer vision. Journal of Communication, Advance Online Publication.
  • Peng, Y. & Jemmott III, J. (2018). Feast for the eyes: Effects of food perceptions and computer vision features on food photo popularity. International Journal of Communication, 12, 313–336.
  • Wang, Y., & Kosinski, M. (2018). Deep neural networks are more accurate than humans at detecting sexual orientation from facial images. Journal of Personality and Social Psychology, 114(2), 246–257.
  • Naik, N., Kominers, S. D., Raskar, R., Glaeser, E. L., & Hidalgo, C. A. (2017). Computer vision uncovers predictors of physical urban change. Proceedings of the National Academy of Sciences, 114(29), 7571–7576.
  • Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S. (2016). Combining satellite imagery and machine learning to predict poverty. Science, 353(6301), 790–794.
  • Horiuchi, Y., Komatsu, T., & Nakaya, F. (2012). Should candidates smile to win elections? An application of automated face recognition technology. Political Psychology, 33(6), 925–933.

Computer science (including human-computer interaction, data science, etc.)

  • Kim, Y., & Kim, J. H. (2018). Using computer vision techniques on Instagram to link users’ personalities and genders to the features of their photos: An exploratory study. Information Processing & Management, 54(6), 1101–1114.
  • Reece, A. G., & Danforth, C. M. (2017). Instagram photos reveal predictive markers of depression. EPJ Data Science, 6(1), 15.
  • Wei, X., & Stillwell, D. (2017, February). How smart does your profile image look? Estimating intelligence from social network profile images. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (pp. 33–40). New York, NY: ACM.
  • Manikonda, L., & De Choudhury, M. (2017, May). Modeling and understanding visual attributes of mental health disclosures in social media. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 170–181). New York, NY: ACM.
  • Gygli, M., & Soleymani, M. (2016, October). Analyzing and predicting GIF interestingness. In Proceedings of the 2016 ACM on Multimedia Conference (pp. 122–126). New York, NY: ACM.
  • Bakhshi, S., Shamma, D. A., Kennedy, L., Song, Y., de Juan, P., & Kaye, J. J. (2016, May). Fast, cheap, and good: Why animated GIFs engage us. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 575–586). New York, NY: ACM.
  • Liu, L., Preotiuc-Pietro, D., Samani, Z. R., Moghaddam, M. E., & Ungar, L. (2016). Analyzing personality through social media profile picture choice. In Proceedings of the Tenth International AAAI Conference on Web and Social Media. (pp. 211–220). Cologne, Germany: AAAI.
  • Joo, J., Steen, F. F., & Zhu, S. C. (2015). Automated facial trait judgment and election outcome prediction: Social dimensions of face. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3712-3720). IEEE.
  • Abdullah, S., Murnane, E. L., Costa, J. M., & Choudhury, T. (2015, February). Collective smile: Measuring societal happiness from geolocated images. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 361–374). New York, NY: ACM.
  • Bakhshi, S., & Gilbert, E. (2015). Red, purple and pink: The colors of diffusion on Pinterest. PLOS ONE, 10(2), e0117148.
  • Deza, A., & Parikh, D. (2015). Understanding image virality. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1818–1826). Boston, MA: IEEE.
  • Gelli, F., Uricchio, T., Bertini, M., Del Bimbo, A., & Chang, S. F. (2015). Image popularity prediction in social media using sentiment and context features. In Proceedings of the ACM Conference on Multimedia (pp. 907–910). New York, NY: ACM.
  • Joo, J., Li, W., Steen, F. F., & Zhu, S. C. (2014). Visual persuasion: Inferring communicative intents of images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 216-223). IEEE.
  • Khosla, A., Das Sarma, A., & Hamid, R. (2014, April). What makes an image popular?. In Proceedings of the 23rd International Conference on World Wide Web (pp. 867–876). New York, NY: ACM.
  • Totti, L. C., Costa, F. A., Avila, S., Valle, E., Meira Jr, W., & Almeida, V. (2014). The impact of visual attributes on online image diffusion. In Proceedings of the ACM Conference on Web Science (pp. 42–51). New York, NY: ACM.
  • Bakhshi, S., Shamma, D. A., & Gilbert, E. (2014, April). Faces engage us: Photos with faces attract more likes and comments on Instagram. In Proceedings of the 32nd ACM Conference on Human Factors in Computing Systems (pp. 965–974). New York, NY: ACM.
  • Machajdik, J., & Hanbury, A. (2010). Affective image classification using features inspired by psychology and art theory. In Proceedings of the ACM International Conference on Multimedia (pp. 83–92). New York, NY: ACM.
  • Datta, R., Joshi, D., Li, J., & Wang, J. Z. (2006, May). Studying aesthetics in photographic images using a computational approach. In Proceedings of the European Conference on Computer Vision (pp. 288–301). Berlin, Germany: Springer.
  • Ke, Y., Tang, X., & Jing, F. (2006, June). The design of high-level features for photo quality assessment. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 419–426). New York, NY: IEEE.