Visualizing Global News

Fig. 1: Elizabeth Losh and members of Software Studies Initiative explore media visualization of 113 video public addresses by U.S. President Barack Obama. Visualizations are available at http://lab.softwarestudies.com/2011/09/digging-into-global-news.html
Television news often serves as the first, most vivid draft of history and shapes the conventions of political speech and civic participation around the world, but undertaking systematic analysis of large digitized corpora of broadcast news video archives and even smaller corpora of government information videos in the public record presents a number of technical, methodological, and institutional challenges. Although a deeper understanding of how the news is represented in moving images promises to improve access to historical records, participation in public dialogue, and education at all levels, large corpora of video news programs are considerably more difficult to catalog, mine, and visualize than text news collections. Furthermore, new forms of dissemination, annotation, and commentary made possible by the Internet are rapidly transforming video news consumption on a global scale. At the same time, traditional newspapers of record are incorporating more multimedia content, so new computational methods such as video search will become increasingly necessary for researchers working with all news collections. By collaborating with archivists and computer scientists, it is possible to analyze visual rhetoric in very large video collections using media visualization
“Visualizing Global News” studies how excerpts from large collections of political speech videos with historically significant personages are remixed into even larger collections of news broadcasts. It applies new techniques in search and visualization to aid both humanities researchers and the greater public interested in exploring the visual details, aesthetic features, and narrative context of source footage that is reused in news broadcasts. The project focuses on materials from four political figures who appear in video news from 2001 to 2011 and tags and visualizes a rich set of intersecting corpora: newscasts from the US program Democracy Now!, the Qatar news channel Al Jazeera English, the top-rated Dutch news show NOS Journaal. This video corpus contains over 30,000 separate news programs, and over 2,000 videos of political speeches.
As participants in the Digital Media Analysis, Search and Management (DMASM) international workshops, we are well aware of the technical problems plaguing automatic systems and that even distinguishing foreground objects from backgrounds can be challenging. Nonetheless, the MediaMill technology has performed well in the yearly TRECVID benchmark competition, and ImagePlot can create compelling visualizations working with key frames. Our presentation aims to provide a policy overview of the opportunities and challenges of DH with global video news archives. Currently the “big data” problem of identifying heterogeneous sources in moving image archives has been largely funded by corporations interested in protecting intellectual property and state entities interested in surveillance. Without digital humanities scholars playing a larger role and generating research questions that merit publication, the most vivid aspects of the historical record are likely to remain under-theorized and analyzed without large- scale comparative study across nations and periods.

Visual rhetoric has a long tradition in the humanities that includes analysis of symbolic objects in portraits of world leaders or the choreography of their oratorical performances. However, contemporary news broadcasts present historical figures in the public record in increasingly visually complex ways that use massive collections of heterogeneous and frequently unsourced footage, motion graphics, and digital effects.
In this project we use new computational and visualization techniques to foster new forms of humanities scholarship and public access to historical records. Political figures produce memoirs, letters, editorials, and other forms of written discourse, but the speeches they compose are also performed and thus can be analyzed as much more than written texts. Now that the speeches of contemporary political leaders are recorded and archived, humanists have a rich record of public rhetoric to analyze that includes facial expression, bodily gesture, vocal performance, and frequently the use of sets and props. In the era of digital video new editing and compositing techniques are also part of the “official version” of a given speech, and portions of these speeches may be further edited and composited when they appear as part of news broadcasts. Broadcasters may adopt (or reject) signature visual styles that brand their programs and even signal their political orientation, and these markers of visual rhetoric can be mapped over time or represented comparatively (Manovich, 2011b). Scholars in the fields of rhetoric, performance studies, history, journalism, film and media studies, and civic education can compare and contrast visual and verbal political messages and changes over time in large video collections.
While a transcript of a given speech may give humanities scholars information about word use and specific references to people, places, and events, much of the rhetoric of the news is actually nonverbal. For example, the use of slow motion, freeze frame, or replay techniques may change the meaning of a given rhetorical moment. Visual arguments in news programs are now often advanced by compelling editing, dazzling information graphics, or aesthetic choices that privilege aspects such as warm lighting or patriotic color schemes. At the same time, the size of digital collections of video news is growing rapidly in response to a number of trends: 1) cable television and satellite television, which spawned the twenty-four-hour news cycle, have become international phenomena, 2) television stations now create web-only content to draw Internet viewers to their programs online, 3) newspapers are incorporating multimedia content and interactive features in archives that once only indexed print news, and 4) bloggers and vloggers involved in citizen journalism have participated in a dramatic expansion of the scope and scale of independent media.
The difficulty of humanities news analysis is exacerbated by the fact that a typical television news program is actually “database cinema” (Manovich 2005) composed of many different kinds of source footage, including studio shots of anchors, field reporting, tock footage, video news releases, government public relations materials, and witness journalism from cell phones and mobile devices. Source clips may not be attributed much less tagged with date, location, individuals shown, or person/organization behind the camera (Losh 2008, Gregory 2010), and the meaning of objects or gestures in the frame, as evidence of the intention or motivation of particular political actors, can be controversial (Losh 2011).
The UCSD project team created visualizations with the official digital video archive from the Obama administration at WhiteHouse.gov that focused on the collection of “Weekly Addresses” directly addressing an imagined American Internet viewer that includes extensive discussions about current events that range from repeated occurrences (economic boosterism, holiday celebrations, etc.) to one-time disasters and tragedies (the Deepwater Horizon oil spill, the shootings at Fort Hood, etc.). UCSD has also examined a significant subset of videos in the WhiteHouse.gov archive that are addressed to audiences abroad as part of U.S. public diplomacy campaigns, which are sometimes also remixed into global news broadcasts. Even a country with a relatively small population or one that is not usually considered critical to U.S. interests, such as Côte d'Ivoire, may have a designated Obama direct address. This dataset allows those studying the visual rhetoric of international relations to work with a particularly rich and dense corpus that includes acknowledgement of cultural exchanges and national holidays. For example, historians studying U.S. Iranian relations in the twenty first century could look closely at three separate annual addresses recorded by Barack Obama that were intended to go directly to the people of Iran on the occasion of Nowrūz, the Persian new year. Each address is fundamentally different in tone and diction, as the U.S.-Iranian diplomatic relationship seems to deteriorate, as well as in shot composition and White House location.
Media created by a sitting U.S. president are always historically significant, but media created by Barack Obama -- the first test case area in this project -- are especially interesting to humanities researchers. In addition to scholarship done by rhetoricians, historians, political scientists, and performance studies and communication scholars, Barack Obama has also generated significant scholarly attention worldwide from those who study American popular culture, race relations, religion, gender, class, civic participation, digital culture, and globalization. The volume of scholarly publication of peer-reviewed books and articles about Obama and the large number of conferences, panels, and talks reflect the potential importance of this collection to scholarly discourse.
The Obama official video corpus has also been significant for the international press, which often uses content from the weekly addresses and other WhiteHouse.gov speeches and public statements in news broadcasts shown worldwide. Researchers at UCSD have noted the presence of material from the official Obama corpus in global broadcasts that include shows from the BBC, PressTV in Iran, Al Arabiya in the United Arab Emirates, ABS-CBN in the Philippines, and many others. The expense of maintaining news bureaus abroad is an obvious reason to rebroadcast free HD footage, but stations often add their own visuals and editorial commentary. For example, the Indian television network NDTV cropped official U.S. government footage from “President Obama’s Statement on Credit Downgrade” and added its own corporate branding along with a skeptical digital banner that read “Obama Assures But Dow Plunges.”

Fig. 3: UCSD visualization of White House government records footage of an Obama speech to the Iranian people (top) and visualizations of how the same speech appears in Democracy Now! (middle) and Al Jazeera English (bottom). To create these visualizations, the UCSD team first used open source software to automatically detect shot boundaries in the video, and then applied media visualization tools to create a grid of images where each image is the first frame of each UCSD visualization of White House government records footage of an Obama speech to the Iranian people (top) and visualizations of how the same speech appears in Democracy Now! (middle) and Al Jazeera English (bottom). To create these visualizations, the UCSD team first used open source software to automatically detect shot boundaries in the video, and then applied media visualization tools to create a grid of images where each image is the first frame of each shot.
Working with a large set of news archives from different sources will also allow us to develop possible answers to a number of critical research questions for archivists: How can we build systems that allow users to compare historical world events from different national or political perspectives? How can we connect one nation’s collections to the collections of others, and how will those connected collections benefit users? Given the capabilities of new search tools, how can we best improve cataloging processes, which currently depend on costly manual input, to make collections broadly usable and accessible? How can we begin to trace the re-use and re-contextualization of primary materials, in particular, news materials?

Fig. 4: President Obama's 2009 Nowruz address to the Iranian people as excerpted by the TV news program Democracy Now! Each frame represents one shot of the program. The frames are arranged in the order of shots (left to right, top to bottom). Shot 19 (third column, third row) is the excerpt from President Obama'a video address.

Fig. 5: President Obama's 2009 Nowruz address excerpted by Al Jazeera TV broadcast. Each frame represents a sequential shot of the program. Shots 5 and 6 (first row) are excerpts from President Obama's video address.
References

American Council of Learned Societies (2006). Our cultural commonwealth : The report of the American Council of Learned Societies Commission on Cyberinfrastructure for the Humanities and Social Sciences. New York, NY: American Council of Learned Societies.
Atkinson, Joshua (2010). Alternative media and politics of resistance: A communication perspective. New York, NY: Peter Lang.
Becker, Konrad (2009). Deep search:The politics of search beyond Google. Innsbruck; Piscataway, N.J. Studien Verlag.
Boler, Megan (2008). Digital media and democracy tactics in hard times. Cambridge, MA.: MIT Press.
Borgman, Christine L. (2010, May). The digital archive: The data deluge arrives in the humanities. Time Will Tell, But Epistemology Won't. In Memory of Richard Rorty. Presentation at A Celebration of Richard Rorty's Archive, University of California, Irvine.
Dayan, Daniel (1992). Media events: the live broadcasting of history. Cambridge, MA: Harvard University Press.
Douglass, Jeremy (2009, January). Computer visions of computer games: Analysis and visualization of play recordings. Workshop at Media Arts, Science, and Technology (MAST) 2009: The Future of Interactive Media. University of California, Santa Barbara.
Douglass, Jeremy, Huber, William & Manovich, Lev (2011). Understanding scanlation: How to read one million fan-translated manga pages. Image and Narrative. Brussels, Belgium.
Edelman, Murray (1988). Constructing the Political Spectacle. Chicago: University Of Chicago Press.
Entman, Robert (1989). Democracy without citizens: Media and the decay of American politics. New York, NY: Oxford University Press.
Etzioni, A. (1972). Minerva: An Electronic Town Hall. Policy Sciences, 3(4), 457-474.
Gregory, S. (2010). Cameras Everywhere: Ubiquitous Video Documentation of Human Rights, New Forms of Video Advocacy, and Considerations of Safety, Security, Dignity and Consent. Journal of Human Rights Practice, 2(2), 191-207. doi:10.1093/jhuman/huq002
Hill, C. A., & Helmers, M. H. (2004). Defining visual rhetorics. Mahwah, NJ: Lawrence Erlbaum.
Huurnink B. et al. (2010). Today's and tomorrow's retrieval practice in the audiovisual archive. Proceedings from CIVR '10: ACM International Conference on Image and Video Retrieval (pp.18-25). Xi'an, China.
Ito, Mizuko (2009). Hanging Out, Messing Around, and Geeking Out. Cambridge, MA: MIT Press.
Jamieson, Kathleen (1988). Eloquence in an Electronic Age: the Transformation of Political Speechmaking. New York: Oxford University Press.
Jarrett, Susan; Losh, Elizabeth & Puente, David (2006). Transnational identifications: Biliterate writers in a first-year humanities course. Journal of Second Language Writing 15, 24-48.
Jenkins, Henry (2006). Convergence culture: Where old and new media collide. New York, NY: New York University Press.
Juhasz, Alexandra (2011). Learning from YouTube. Cambridge, MA: MIT Press.
Kumar, Belhumeur & Nayar (2008). FaceTracer: A Search Engine for Large Collections of Images with Faces. Proceedings from The European Conference on Computer Vision, 340-353.
Lévy, Maurice & European Commission (2011). The New Renaissance. Brussels: European Commission.
Losh, Elizabeth (2008). Government YouTube: Bureaucracy, surveillance, and legalism in state sanctioned online video channels. In Geert Lovink & Sabine Niederer (Eds.), Video vortex reader (pp. 111-124). Amsterdam, Netherlands: Institute of Network Cultures.
Losh, Elizabeth (2009). Virtualpolitik: An electronic history of government media-making in a time of war, scandal, disaster, miscommunication, and mistakes. Cambridge, MA: MIT Press.
Losh, Elizabeth & Alexander, Jonathan (2010). 'A YouTube of one's Own?': 'Coming out' narratives as rhetorical action. In Christopher Pullen & Margaret Cooper (Eds.), LGBT identity and online new media (pp. 23-36). New York, NY: Routledge.
Losh, Elizabeth (2011) Shooting for the public: YouTube, Flickr, and the Mavi Marmara shootings. In Geert Lovink & Rachel Somers Miles (Eds.), Video vortex reader II (pp. 283-292). Amsterdam, Netherlands: Institute of Network Cultures.
Losh, Elizabeth (2012). Channeling Obama: YouTube, Flickr, and the social media president. Comparative American Studies 10(2-3) (forthcoming).
Loviglio, J. (2005). Radio's intimate public: network broadcasting and mass-mediated democracy. Minneapolis: University of Minnesota Press.
Lubin, D. (2003). Shooting Kennedy: JFK and the culture of images. Berkeley: University of California Press.
Lynch, Marc (2006). Voices of the new Arab public: Iraq, Al-Jazeera, and Middle East politics today. New York, NY: Columbia University Press.
Manovich, Lev (2001). The Language of New Media. Cambridge, Mass.: The MIT Press.
Manovich, Lev (2007). Cultural analytics. [White paper]. Retrieved from http://lab.softwarestudies.com/2008/09/cultural-analytics.html.
Manovich, Lev (2009a). Cultural analytics: Visualizing cultural patterns in the era of 'more media.' DOMUS (Spring 2009). Milan.
Manovich, Lev (2009b). The practice of everyday (media) life: From mass consumption to mass cultural production? Critical inquiry 35(2), 319-331.
Manovich, Lev (2010). Software Takes Command. Italian translation: Milan: Edizioni Olivares, published as Software Culture. Revised version Continuum, 2013.
Manovich, Lev (2011a). From reading to pattern recognition. In Mieke Gerritzen, Geert Lovink, & Minke Kampman (Eds.), I read where I am: Exploring the new information cultures. Amsterdam, Netherlands: Valiz.
Manovich, Lev (2011b). "Media Visualization: Visual Techniques for Exploring Large Media Collections." in Kelly Gates (Ed.), Media Studies Futures. Blackwell, forthcoming 2012.
Manovich, Lev(2011c). "What is Visualization?" Visual Studies, 26(1), 36-49.
Manovich, Lev and Jeremy Douglass (2011d). "Visualizing Change," Oliver Grau with Thomas Veigl (Eds.), Imagery in the 21st Century (pp. 315-338). Cambridge, MA: The MIT Press.
Manovich, Lev(2012). Trending: The promises and the challenges of big social data. In Matthew K. Gold (Ed.), Debates in the digital humanities. The University of Minnesota Press.
McCarty, William (2007). Beyond retrieval? Computer science and the humanities. Retrieved from http://www.mccarty.org.uk/essays/McCarty,%20Beyond%20 retrieval.pdf
McKinsey Global Institute (2011). Big data: The next frontier for innovation, competition, and productivity. New York, NY: McKinsey Global Institute.
Navas, Eduardo (2012). "Modular Complexity: The Collapse of Time and Space into Search." AnthroVision journal. France: Lyon, forthcoming Spring/Summer 2012.
Olson, L. C., Finnegan, C. A., & Hope, D. S. (2008). Visual rhetoric: a reader in communication and American culture. Los Angeles: Sage
Oomen, Johan, et al.(2009). Images for the future: Unlocking the value of audiovisual heritage. In J. Trant & D. Bearman (Eds.), Museums and the Web 2009: Proceedings. Toronto: Archives & Museum Informatics.
Pantti, Mervi (2009). Misfortunes, memories and sunsets: Non-professional images in Dutch news media. International Journal of Cultural Studies 12(5), 471-489.
Pasma, Trijntje (2011). Metaphor and register variation: The personalization of Dutch news discourse. Amsterdam: Vrije Universiteit.
Postman, Neil (1992). How to watch TV news. New York, NY: Penguin Books.
Rodrigues, Isabel Galhano (2010). Gesture space and gesture choreography in European Portuguese and African Portuguese interactions: A pilot study of two cases. Gesture in embodied communication and human-computer interaction. Berlin: Springer.
Sarkar, Sudeep. et al. (2005). The humanID gait challenge problem: Data sets, performance, and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(2), 162-177.
Seinstra, Frank J. et al. (2007). High-performance distributed image and video content analysis with parallel-Horus. IEEE Multimedia 14(4), 64-75.
Snoek, Cees and Smeulders (2010). Visual-concept search solved? IEEE Computer 43(6), 76- 78.
Snoek, Cees et al. (2004-2010). The MediaMill TRECVID 2004-2010 semantic video search engine. In Proceedings from TRECVID Workshops. Gaithersburg, MD.
Ubois, Jeff (2005). New approaches to television archiving. First Monday 10(3). Retrieved from http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/1210/1130.
UCLA Film & Television Archive, UCLA MA Program in Moving Image Archive Studies and INA (Institut National de l'Audiovisuel, France) and Ina SUP European Centre for Research, Training and Education on Digital Media (2010). Reimaging the Archive: Remapping and Remixing Traditional Models in the Digital Era [Conference]. Retrieved from http://polaris.gseis.ucla.edu/reimagining/statement.htm.
van de Sande, Koen E. A., Gevers, Theo & Snoek, Cees (2010). Evaluating color descriptors for object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(9), 1582-1596.
van de Sande, Koen E. A., Gevers, Theo & Snoek, Cees (2011). Empowering visual categorization with the GPU. IEEE Transactions on Multimedia 13(1), 60-70.
Yamaoka, S., L. Manovich, J. Douglass, F. Kuester (2011). "Cultural Analytics in Large- Scale Visualization Environments." IEEE Computer, December 2011.
Zepel, Tara (2011). Culture as data as culture : Re-presenting the videostyle of the 2008 presidential campaign ads. In Geert Lovink & Rachel Somers Miles (Eds.), Video vortex reader II (pp. 234-239). Amsterdam, Netherlands: Institute of Network Cultures