Content Mining
Introduction and Terminology
Topics on this page include content and text mining as well as topic modeling.
Content Mining is the overall concept of pulling together into one place a large corpus of text, data, or images from various sources and running scripts on them to answer a research question.
- Text Mining is a research technique using computational analysis to uncover patterns in large text-based data sets. (Wikipedia)
- Data Mining is to use machine-learning and statistical models to uncover clandestine or hidden patterns in a large volume of data (Wikipedia)
- Webscraping, web harvesting or web data extraction is data scraping used for extracting data from web sites. The term typically refers to automated processes implemented using a bot or web crawler.
- Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to perform a specific task without using explicit instructions, relying on patterns and inference instead. (Wikipedia)
- Topic Modeling: A form of text mining to identify patterns or topics in a large corpus of text. (Blei, 2012)
- Natural Language Processing (NLP) is s process of programming computers to process and analyze large amounts of natural language texts.
Tools for Content Mining
- AntConcA freeware corpus analysis toolkit for concordancing and text analysis.
- DataFile ConverterData conversion software that helps data handlers convert data file from one format to another. Can use the free trial temporarily.
- Google ColabSpace in Google Drive to run a Jupyter Notebook on Google servers using Python.
- jsLDA:In-browser topic modelingDavid Mimno of Cornell created this topic modeler that works in your browser.
- Tesseract OCR softwareOpen source software for OCRing images to create machine readable text.
- VoyantWeb based tool for performing text analysis. It can analyze online or user-uploaded texts. Voyant is open source and free to use.
Freely Available Corpora for Mining
- Biodiversity Heritage Library APIsDownload, remix and reuse BHL content.
- Chronicling America Newspapers APIImport and study texts from newspapers around the country from 1789-1963
- Digital Public Library of America (DPLA)Import and study digitized collections and their metadata from around the country.
- Europeana Digital LIbrary LabsExplore and re-use Europe's cultural heritage.
- General IndexGeneral Index consists of 3 tables derived from 107,233,728 journal articles.
- Internet Archive eBooks and TextsExplore over 20,000,000 freely downloadable books and texts
- New York Times APIsAccess the New York Times data using 10 APIs for the Archive, Article Search, Books, Community, Geographic, Most Popular, Times Newswire, TimesTags, and Top Stories
- Project GutenbergAn online library of free ebooks.
- Public Library of Science (PLOS)Query the content of PLOS journals or access article-use data
- University of Oxford Text ArchiveImport and study thousands of texts in more than 25 languages, all full-text literary and linguistic resources.
USC Licensed Content for Mining
Costs are incurred for working with some of this data. Please contact a USC librarian if you are interested.
- Adams MathewAdam Matthew makes available for USC LIbraries American History: Settlement, Commerce, Revolution and Reform, 1493-1859 American History: Civil War, Reconstruction and the Modern Era, 1860-1945 Slavery, Abolition and Social Justice If you are interested in data mining any of these collections, contact USC Libraries to initiate the process.
- Cambridge University PressCost negoiated per request. Contact USC Libraries to initiate the process.
- ConstellateJSTOR and Portico content can be mined through this product using Jupyter Notebooks
- EBSCOhost APICosts may be incurred. This site consists of various methods that deliver XML-formatted data to the caller who can then format and use the data as desired. Customers can call EBSCOhost API for a wide variety of applications using the HTTP/SOAP or HTTP/REST protocols.
- Elsevier APIAccess journal text from Elsevier products.
- Gale Primary ResourcesSome free: downloading large datasets incurs costs. Contact USC Libraries to initiate the process. Gale Artemis: Primary Sources searches across 23 Gale Primary Source databases, covering 1500-2012 and include Term Frequency search option and Term Cluster viewier. To download large datasets, USC LIbraries will have to request data on your behalf from our Gale sales representative. It can take up to 3 weeks to process this request.
- HathiTrust Digital Library datasets:HathiTrust makes the text of public domain works in its corpus available for research purposes. As a member of the USC Community, see also: HathiTrust Research Center
- IEEECosts negotiated per request. Contact USC Libraries to initiate process. Through a negotiation of the vendor license, the library facilitates on a case-by-case basis.
- NewsbankCosts incurred. Contact USC Libraries to initiate the process. Restrictions in place. Costs for TDM research are between $6-$8,000 and can take up to 6-8 weeks to process.
- Oxford University PressCosts incurred. Contact USC Libraries to initiate the process. Researchers may use resources for non-commercial text mining. However, OUP offers consultation services with technical project managers to assist in planning projects, including "avoidance of any technical safeguard triggers OUP has in place to protect stability and security of website."
- ProQuestCost negotiated per request and available TDM Studio platform. Contact USC Libraries to initiate the process. Proquest does allow free text mining for the newspapers to which USC Libraries have purchased perpetual access licenses. USC Libraries will have to request this data on your behalf. TDM Studio offers (for pay) select researchers to use Proquest resources including newspapers for research.
- Sage JournalsReview their policy at the link. Use the articles for non-commercial use only, respect the limits on downloading, and agree to their terms and conditions.
- SpringerLinkFree(with subscription). Users can download subscribed and open access content for TDM purposes directly from the SpringLink Platform. Content can be downloaded via a web browser or with an HTTP GET requeust using a scripting tool such as curl,wget, and Python's urllib, among others. No API key or other authentication is required. TDM researchers are requested to be considerate and limit their downloading speed to a reasonable rate.
- Taylor and FrancisCosts may be incurred. Contact a USC Librarian to coordinate with the Taylor & Francis representative.
Tutorials and Information
- Gavin, Michael. (2023) Literary MathematicsLearn key concepts for the statitical analysis of corpus data, with examples and background discussion. Explore sample results, analysis, and demonstrations of computational literary analyses.
- Programming Historian TutorialsProvides basic, peer-reviewed tutorials on a wide range of digital tools for humanists
- Sage Text Mining Tutorials for Social ScientistsOne hour long video discussing the fundamentals of text mining, including research design, ethics, NLP, and tips on teaching to social science students
- The TDM Today Show! Episode 01Series of videos explaining text and data mining (TDM)
Example Projects
- Emerging Technology and arXiv.(July 2016) "Data Mining Reveals Six Basic Emotional Arcs of Storytelling," MIT ReviewScientists at the Computational Story Laboratory at the University of Vermont have used sentiment analysis to map emotional arcs of over 1700 stories.
- King, Lindsey and Peter Leonard. (2019) "Robots Reading Vogue." Yale University.Vogue is completely digitized and marked up as both text and images. See examples of work Yale has done on this corpus.
- National Science Foundation. (2016) "Rescued History: Using Big Data." Press ReleaseMassive data analysis helps uncover black women's experiences.
- Newspapers as Data Project"Using Newspapers as Data for Collaborative Pedogogy: A Multidisciplinary Interrogation of the Borderlands in Undergraduate Classrooms" brings together a group of library faculty and disciplinary scholars to introduce students to data literacy and computational analysis using digitized historical newspapers from Arizona.
- On the Books: Jim Crow and Algorithms of ResistanceCan text mining and machine learning identify racist language in legal documents?
- Sperr, Edwin. News In Proportion.Searches keywords across all of the Library of Congress' Chronicling America newspapers and gives a map visualizing where those words appeared most
Further Resources
- BigDIVABig Data Infrastructure Visualization Application is a dynamic environment for browsing, searching, and interacting with the ARC (Advanced Research Consortium) Catalog.