Big data and national security: A guide for Australian policymakers (2024)

Main image: The ubiquitous use of technology enables near complete coverage of human lives across the planet often in real time (Gilles Lambert/Unsplash)

[1] Rob Kitchin, "Big Data, New Epistemologies and Paradigm Shifts," Big Data & Society, Vol 1, Iss 1 (2014); The Data Revolution: Big Data, Open Data, Data Infrastructures & Their Consequences (London: SAGE, 2014); Dennis Murphy and Daniel Kuehl, "The Case for a National Information Strategy," Military Review, September–October (2015), https://www.armyupress.army.mil/Portals/7/military-review/Archives/English/MilitaryReview_20151031_art013.pdf.

[2] Kitchin, "Big Data, New Epistemologies and Paradigm Shifts,”; Danah Boyd and Kate Crawford, “Critical Questions for Big Data," Information, Communication & Society, Vol 15, No 5 (2012), 662–79; and Miriam J. Metzger and Andrew J. Flanagin, "Credibility and Trust of Information in Online Environments: The Use of Cognitive Heuristics," Journal of Pragmatics, No 59, Part A (2013), 210–20.

[3] David Rubin et al., "Harnessing Data for National Security," SAIS Review 34, No 1 (2014).

[4] There are many different conceptualisations of national security. The notion of values is highlighted as “the absence of threats to acquired values and subjectively, the absence of fear that such values will be attacked” by Arnold Wolfers, Discord and Collaboration; Essays on International Politics (Baltimore, Johns Hopkins Press, 1962), 485. >David A. Baldwin, "The Concept of Security," Review of International Studies 23 (1997), subsequently refined “the absence of threat” as “a low probability of damage to acquired values”.

[5] Kitchin, The Data Revolution; Boyd and Crawford, "Critical Questions for Big Data".

[6] This paper draws on data collected as part of a larger research project that examined the impact of big data on intelligence production and national security in Australia. The research was conducted at Deakin University 2017–2021 and included ethics approval. It involved semi-structured interviews with 47 senior and operational decision-makers as well as technologists working in Australia’s national intelligence community (NIC) agencies. The author gained access to some of Australia’s most highly regarded intelligence leaders and practitioners to provide insight and analysis into the challenges and opportunities of big data. Intelligence practitioners are well positioned to provide insight into the impact of big data on national security, however this paper extends beyond those conversations.

[7] Department of Defence., Defence Strategic Update, Department of Defence (Commonwealth of Australia, 2020), https://www.defence.gov.au/about/publications/2020-defence-strategic-update.

[8] Jacob Metcalf, Emily F. Keller, and Danah boyd, "Perspectives on Big Data, Ethics, and Society," (2016), 2.

[9] Boyd and Crawford, "Critical Questions for Big Data," 663.

[10] Viktor Mayer-Schönberger and Kenneth Cukier, Big Data : A Revolution That Will Transform How We Live, Work, and Think, First Mariner Books edition. ed. (Boston: Mariner Books, Houghton Mifflin Harcourt, 2014).

[11] Andrea De Mauro, Marco Greco, and Michele Grimaldi, "What Is Big Data? A Consensual Definition and a Review of Key Research Topics," American Institute of Physics Proceedings, (2015), 1644.

[12] Doug Laney, "3D Data Management Controlling Data Volume Velocity and Variety," META Delta File 949, 6 Febraruy 2001, https://studylib.net/doc/8647594/3d-data-management--controlling-data-volume--velocity--an...; Kitchin, The Data Revolution: Big Data, Open Data, Data Infrastructures & Their Consequences, 68.

[13] The Data Revolution: Big Data, Open Data, Data Infrastructures & Their Consequences, 68.

[14] Gregory B. Saathoff Babak Akhgar, Hamid R. Arabnia, Richard Hill, Andrew Staniforth, Petra Saskia Bayerl, Application of Big Data for National Security: A Practitioner's Guide to Emerging Technologies (Amsterdam; Waltham Elsevier, 2015); Bart van der Sloot, Dennis Broeders and Erik Schrijvers, ed. Exploring the Boundaries of Big Data (The Hague: The Netherlands Scientific Council for Government Policy, 2016); and Rob Kitchin and Tracey P. Lauriault, "Small Data in the Era of Big Data," GeoJournal 80, No 4 (2014); Boyd and Crawford, "Critical Questions for Big Data"; Kitchin, "Big Data, New Epistemologies and Paradigm Shifts."

[15] The grouping of terms was first used by US intelligence leader Sue Gordon and in subsequent communication with the author she confirmed that to the best of her knowledge this grouping was her own construction. The same themes emerged in my research as features of big data most relevant to intelligence and my PhD offers empirical evidence to deepen understanding and define these terms.

[16] Kenneth Cukier, "Data, Data Everywhere," The Economist, 27 February 2010, https://www.economist.com/special-report/2010/02/27/data-data-everywhere.

[17] Jeff Desjardins, "How Much Data Is Generated Each Day?," World Economic Forum, 17 April 2019, https://www.weforum.org/agenda/2019/04/how-much-data-is-generated-each-day-cf4bddf29f/.

[18] David Reinsel, John Gantz, and John Rydning, "The Digitization of the World from Edge to Core," Data Age 2025 (IDC White Paper, 2018), https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf. IDC predicts that the Global Datasphere will grow from 33 Zettabytes (ZB) in 2018 to 175 ZB by 2025.

[19] Mayer-Schönberger and Cukier, Big Data, 73–97.

[20] Kristene Unsworth, "The Social Contract and Big Data," Journal of lnformation Ethics 25, Spring (2016).

[21] Miah Hammond-Errey, "The Transformative Potential of Big Data," The Interpreter, 24 June 2019, https://www.lowyinstitute.org/the-interpreter/transformative-potential-big-data.

[22] David Omand and Mark Phythian, "Digital Intelligence and Cyberspace," ed. David Omand and Mark Phythian, Principled Spying: The Ethics of Secret Intelligence (Oxford University Press, 2018), 145.

[24] Hammond-Errey, “The Transformative Potential of Big Data.”

[25] Customer Loyalty Schemes Review, Qantas Submission in Response to the ACCC's Draft Report, 3 October 2019, https://www.accc.gov.au/system/files/Qantas%20-%20October%202019.pdf; Andrew Curran, “Qantas Ready to Honor $3 Billion Worth of Frequent Flyer Miles”, Simply Flying, 3 March 2021, https://simpleflying.com/qantas-3-billion-frequent-flyer-miles/.

[26] Qantas, "Privacy and Security," https://www.qantas.com/au/en/support/privacy-and-security.html.

[27] Shoshana Zuboff, "Creating Value in the Age of Distributed Capitalism," McKinsey Quarterly, 1 September 2010, https://www.mckinsey.com/business-functions/strategy-and-corporate-finance/our-insights/creating-value-in-the-age-of-distributed-capitalism; Unsworth, "The Social Contract and Big Data."; Tim Harford, "Big Data: Are We Making a Big Mistake?," Financial Times, 28 March 2014, https://www.ft.com/content/21a6e7d8-b479-11e3-a09a-00144feabdc0; Michael J. Mazarr et al., "The Emerging Risk of Virtual Societal Warfare: Social Manipulation in a Changing Information Environment," (Santa Monica, CA: RAND Corporation, 2019), https://www.rand.org/pubs/research_reports/RR2714.html.

[28] Kate Crawford and Jason Schultz, "Big Data and Due Process: Toward a Framework to Redress Predictive Privacy Harms," Boston College Law Review 55, No 1 (2014), https://lawdigitalcommons.bc.edu/bclr/vol55/iss1/4/; Australian Government, "What Is Personal Information?," (Canberra, Australia: Office of the Australian Information Commissioner, 2017), https://www.oaic.gov.au/privacy/your-privacy-rights/your-personal-information.

[29] See for example, L. Rocher, J. M. Hendrickx, and Y. A. de Montjoye, "Estimating the Success of Re-Identifications in Incomplete Datasets Using Generative Models," Nature Communications 10, No 1 (2019), https://www.nature.com/articles/s41467-019-10933-3; Ira S. Rubinstein and Woodrow Hartzog, "Anonymization and Risk," Washington Law Review 91 (2016), https://digitalcommons.law.uw.edu/wlr/vol91/iss2/18/; Y. A. de Montjoye et al., "Unique in the Crowd: The Privacy Bounds of Human Mobility," Nature Communications Rep 3 (2013), https://www.nature.com/articles/srep01376.pdf.

[30] Henrik Twetman and Gundars Bergmanis-Korats, "Data Brokers and Security," Risks and Vulnerabilities Related to Commercially Available Data (NATO StratCom COE, 2021), https://stratcomcoe.org/publications/data-brokers-and-security/17; Dymples Leong and Teo Yi-Ling, "Data Brokers: A Weak Link in National Security," The Diplomat, 21 August 2020, https://thediplomat.com/2020/08/data-brokers-a-weak-link-in-national-security/; Matthew Crain, "The Limits of Transparency: Data Brokers and Commodification," New Media & Society 20, No 1 (2016); Kitchin, The Data Revolution.

[31] Crain, "The Limits of Transparency: Data Brokers and Commodification."

[32] Twetman and Bergmanis-Korats, "Data Brokers and Security."; Leong and Yi-Ling, "Data Brokers: A Weak Link in National Security."; Crain, "The Limits of Transparency: Data Brokers and Commodification."; Kitchin, The Data Revolution.

[33] “What Are Data Brokers — And What is Your Data Worth?”, WebFX, 16 March 2020, https://www.webfx.com/blog/internet/what-are-data-brokers-and-what-is-your-data-worth-infographic/.

[35] Gareth Mitchell, "How Much Data Is on the Internet?," BBC Science Focus Magazine, 2019, https://www.sciencefocus.com/future-technology/how-much-data-is-on-the-internet/.

[36] Twetman and Bergmanis-Korats, "Data Brokers and Security."

[39] "Bridging the World through Digital Connectivity," StoryWorks, BBC (2018), https://www.bbc.com/storyworks/specials/digital-connectivity/.

[40] Klaus Schwab, The Fourth Industrial Revolution (Penguin Books Limited, 2017).

[42] Australian Government Productivity Commission, "Digital Disruption: What Do Governments Need to Do?," Productivity Commission Research Paper (Canberra: Australian Government Productivity Commission, 2016), https://www.pc.gov.au/research/completed/digital-disruption.

[43] Surjit Singh and Rajeev Mohan Sharma, Handbook of Research on the Iot, Cloud Computing, and Wireless Network Optimization (IGI Global, 2019); Intel, "A Guide to the Internet of Things," https://www.intel.com/content/www/us/en/internet-of-things/infographics/guide-to-iot.html.

[44] Murphy and Kuehl, "The Case for a National Information Strategy," 72.

[45] Intel, "A Guide to the Internet of Things".

[46] Unsworth, "The Social Contract and Big Data."

[47] There is extensive research revealing that many of the categorisation methods and inferences and assumptions made about humans and human behaviour using big data and AI are not technically sophisticated, are unsubstantiated, and in some cases simply untrue. See for examples, Kate Crawford and Jason Schultz, “AI Systems as State Actors,” Columbia Law Review, Vol 119, No 7, https://columbialawreview.org/content/ai-systems-as-state-actors/; Kate Crawford, Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence, (Yale University Press 2021); and Huon Curtis, “Finding Australia’s Asymmetric Advantage in Big Data”, The Strategist, ASPI, 30 June 2021, https://www.aspistrategist.org.au/finding-australias-asymmetric-advantage-in-big-data/.

[48] Genevieve Bell, "The Character of Future Indo-Pacific Land Forces," Australian Army Journal XIV, No 3 (2018), 175, https://search.informit.org/doi/pdf/10.3316/ielapa.377081144168780.

[49] José van Dijck, Thomas Poell, and Martijn de Waal, The Platform Society: Public Values in a Collective World (Oxford Scholarship Online, 2018).

[51] Mazarr et al., "The Emerging Risk of Virtual Societal Warfare."; Shosana Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power: Barack Obama's Books of 2019 (Profile, 2019); Dale Neef, "Big Data Big Bang," ed. Dale Neef, Digital Exhaust: What Everyone Should Know About Big Data, Digitization, and Digitally Driven Innovation (PH Professional Business, 2014); Meredith Whittaker, "The Steep Cost of Capture," Interactions XXVIII, November–December 2021 (2021).

[52] van Dijck, Poell, and de Waal, The Platform Society; Rodrigo Fernandez, Tobias J. Klinge, Reijer Hendrikse, and Ilke Adriaans, "How Big Tech Is Becoming the Government," Tribune Magazine, 5 February 2021, https://tribunemag.co.uk/2021/02/how-big-tech-became-the-government.

[53] Mazarr et al., "The Emerging Risk of Virtual Societal Warfare."

[54] Omand and Phythian, "Digital Intelligence and Cyberspace," 145.

[55] Jayshree Pandya, "The Dual-Use Dilemma of Artificial Intelligence," Forbes, 28 January 2019, https://www.forbes.com/sites/cognitiveworld/2019/01/07/the-dual-use-dilemma-of-artificial-intelligence/?sh=748ef99d6cf0.

[56] Carissa Véliz, "Privacy and Digital Ethics after the Pandemic," Nature Electronics 4, No 1 (2021).

[59] John Lee, "Big Data for Liberal Democracy," The Australian, 5 March 2021, https://www.theaustralian.com.au/inquirer/big-data-for-liberal-democracy/news-story/10ed6d4d8b01677955fc468d3751a4b7.

[61] Cristian Santesteban and Shayne Longpre, "How Big Data Confers Market Power to Big Tech: Leveraging the Perspective of Data Science," The Antitrust Bulletin 65, No 3 (2020).

[62] Joanna Redden, "Six Ways (and Counting) That Big Data Systems Are Harming Society," The Conversation, 7 December 2017, https://theconversation.com/six-ways-and-counting-that-big-data-systems-are-harming-society-88660.

[63] Department of Defence, "Defence Strategic Update."

[65] Véliz, "Privacy and Digital Ethics after the Pandemic."

[66] Anja Prummer, "Micro-Targeting and Polarization," Journal of Public Economics 188, August, (2020). Julie E. Cohen, "Law for the Platform Economy," University of California, Davis Law Review 51 (2017).

[67] ASIO, Asio Annual Report 2019–20, Commonwealth of Australia, 2020, 3-4, https://www.asio.gov.au/sites/default/files/ASIO%20Annual%20Report%202019-20.pdf.

[71] ASIO, Asio Annual Report 2019–20, 4.

[73] Danielle Cave, "Data Driven: How Covid-19 and Cyberspace Are Changing Spycraft," Australian Foreign Affairs, Spy vs Spy The New Age of Espionage, No 9 (2020), https://www.australianforeignaffairs.com/articles/extract/2020/07/data-driven.

[74] ASIO, Director-General’s Annual Threat Assessment, 2021.

[75] Martin C. Libicki, "Information Dominance," Strategic Forum Institute for National Strategic Studies and the National Defense University 132 (1979).

[78] Entities are added to the US Entity List when it is determined they are acting contrary to the national security or foreign policy interests of the United States.

[79] Dahm, "Exploring China's Unmanned Ocean Network."

[80] Elsa B. Kania, "Chinese Military Innovation in Artificial Intelligence," Testimony before the US–China Economic and Security Review Commission Hearing on Trade, Technology, and Military–Civil Fusion, Center for New American Security, 7 June 2019, https://www.cnas.org/publications/congressional-testimony/chinese-military-innovation-in-artificial-intelligence.

[81] J. Michael Dahm, "Introduction to South China Sea Military Capabilities Series," South China Sea Military Capabilities Series: A Survey of Technologies and Capabilities on China’s Military Outposts in the South China Sea (The Johns Hopkins University Applied Physics Laboratory, 2020), https://www.jhuapl.edu/Content/documents/IntroductiontoSCSMILCAPStudies.pdf.

[82] H. I. Sutton, "China Builds Surveillance Network in South China Sea," Forbes, 5 August 2020, https://www.forbes.com/sites/hisutton/2020/08/05/china-builds-surveillance-network-in-international-waters-of-south-china-sea/?sh=5f3d462174f3.

[83] John Gray, "Surveillance Capitalism Vs. The Surveillance State," Noema, 17 June 2020, https://www.noemamag.com/surveillance-capitalism-vs-the-surveillance-state/.

[85] ACCC, "Digital Advertising Services Inquiry: Interim Report 2020, 56.

[86] Mayer-Schönberger and Cukier, Big Data, 73–97.

[88] Dana Priest, Craig Timberg, and Souad Mekhennet, "Private Israeli Spyware Used to Hack Cellphones of Journalists, Activists Worldwide," The Washington Post, 18 July 2021, https://www.washingtonpost.com/investigations/interactive/2021/nso-spyware-pegasus-cellphones/.

[89] Kitchin, The Data Revolution; Crain, "The Limits of Transparency".

[92] David Omand, Jamie Bartlett, and Carl Miller, "Introducing Social Media Intelligence (Socmint)," Intelligence & National Security 27, No 6 (2012).

[94] Ibid.; Miah Hammond-Errey, "Understanding and Assessing Information Influence and Foreign Interference," Journal of Information Warfare 18, Winter (2019); Richmond, "A Day in the Life of Data."

[95] Prummer, "Micro-Targeting and Polarization."

[96] "Digital Microtargeting Political Party Innovation Primer 1," International Institute for Democracy and Electoral Assistance IDEA, (Stockholm,: International Institute for Democracy and Electoral Assistance, 2018); Mazarr et al., "Hostile Social Manipulation: Present Realities and Emerging Trends."

[97] Mazarr et al., "Hostile Social Manipulation"; and Mazarr et al., "The Emerging Risk of Virtual Societal Warfare."

[98] Barbara Trish, "Big Data under Obama and Trump: The Data-Fueled US Presidency," Politics and Governance 6, No 4 (2018).

[99] Mazarr et al., "The Emerging Risk of Virtual Societal Warfare.”

[100] A document leaked from Facebook’s Australian office in 2017 showed analysis of internal (non-public) Facebook data that attempted to identify emotionally vulnerable and insecure young people to “give them a confidence boost”. The document was to be shared with advertisers under a non-disclosure agreement and noted that Facebook had the power to target over 6.4 million Australian and New Zealander high schoolers, tertiary students, and young people in the workforce. Using its algorithms to monitor newsfeed posts and photos as well as a young user’s interaction with content through comments, likes, and shares, Facebook detailed in the report how it was able to ascertain a person’s emotional state — categorising them with tags such as “anxious”, “nervous”, “defeated”, “stressed” and “useless”. Nick Whigham, "Leaked Document Reveals Facebook Conducted Research to Target Emotionally Vulnerable and Insecure Youth," news.com.au, 1 May 2017, https://www.news.com.au/technology/online/social/leaked-document-reveals-facebook-conducted-research-to-target-emotionally-vulnerable-and-insecure-youth/news-story/d256f850be6b1c8a21aec6e32dae16fd; and Peter Griffin, "Facebook Won't Give up Its Insidious Practices without a Fight," Noted, 22 March 2019.

[101] For discussion of the complexities relating to political microtargeting, see for example, Brahim Zarouali, Tom Dobber, Guy De Pauw, and Claes De Vreese, “Using a Personality-Profiling Algorithm to Investigate Political Microtargeting: Assessing the Persuasion Effects of Personality-Tailored Ads on Social Media, Communication Research, 20 October 2020.