[Bracketing] the Black Swan (Part II)

In my last blog post I discussed the possibility of being able to bracket, or identify, a ‘black swan,’ an extremely rare event which has significant consequences. Trying to identify a black swan event is a pretty tall order since these events by definition, are highly unlikely. As I discussed in my blog entry last month, the challenge is to ‘reach out’ on to the statistical distribution towards the unlikely hypotheses.

Research on knowledge systems  suggests that the most commonly identified hypotheses among a group of experts are on the extreme left of the distribution. In most analytic tasks, the most instrumental hypothesis is probably here. For example, there are a few commonly discussed hypothesesfor the outcome of the Syrian Civil War (e.g. Assad regime wins, stalemate, etc.). In the graph below these hypotheses would fall in the green shaded region as H1, H2, and H3. But, in the case of black swan events, the hypothesis (or hypotheses) are less frequently suggested and are further out on the right. In the Syrian example, this might include Iran invading and achieving victory out in the yellow shaded region.

Steve Blog 2 Pic 1


Imaginative structured analytic techniques assist analysts in reaching out further on this distribution, but , some of the techniques have notable  limitations. For example, one such technique, brainstorming, assumes equal participation among diverse group members, which defies conventional experience. Further, most of these techniques cannot tell the analyst where they are on the distribution, and more importantly, when they have reached saturation and generated the bulk of plausible hypotheses. In a traditional brainstorming session, this is usually identified by a lull in the conversation and participants are satisfied they have captured the likely hypotheses.

Boundary analysis, developed by William N. Dunn, is another way to generate hypotheses. The technique requires analysts to sample documents containing hypotheses (e.g. news reports) and write down each hypothesis. As an analyst records more hypotheses he should observe the effect of Bradford’s Law: after a point the number of new hypotheses gathered from each document drops precipitously. Since the hypotheses come from the documents rather than the group itself, the technique may ameliorate some of the negative effects of group dynamics on hypothesis generation. Furthermore, one can simply expand the scope of the search for more documents to gain access to rarely cited hypotheses.

Stopping Point of Bradford’s Law

Steve Blog 2 Pic 2

For most analytic tasks, stopping at the “knee of the curve” (where the marginal frequency of each new hypothesis levels off) will likely include the correct hypothesis. But for “black swan” events, we have no such defined rule. By definition it would seem that a black swan should fall after the stopping rule, but it is also entirely possible that the black swan really was foreseeable.

We simply don’t know.

To address this question I teamed up with my colleague Jay Rickabaugh to apply boundary analysis retrospectively to a ‘real world’ intelligence analysis task: the 2012 University of Pittsburgh bomb threats.

The Pitt Bomb Threats

Over the course of ten weeks in the spring of 2012, the University of Pittsburgh received approximately 140 bomb threats. While the threats took a variety of forms, beginning with scrawled threats in campus restrooms, the most persistent and numerous threats came from emails sent through a remailer, which masked the location of the perpetrator. Further, confounding the investigation were copycat actions, false accusations and others seeking publicity by capitalizing on the chaos.  The swarming of these threats made this case different from a traditional bomb scare and thus the possibility of black swan explanations seems more possible.

During the multi-agency investigation, several leads were pursued but each led to a dead-end. Finally on April 19th, after weeks of threats causing the University of Pittsburgh to spend more than $300,000 in direct costs alone, the University met the demand of one of the threateners to rescind a $50,000 reward, and immediately thereafter, the emailed threats stopped.

In mid-August, after a months-long investigation, authorities held a press conference to announce that they were charging Adam Busby, a 64-year-old Scottish nationalist involved with the Scottish National Liberation Army (SNLA) in connection with the emailed threats. The result was stunning and best summed up by Andrew Fournaridis, administrator of a blog developed during the bomb threats who wrote:

“This is the mind-bending stuff intelligence analysts must deal with on a daily basis, especially in the 21st century cyber-crime era.”

To this day authorities have never divulged Busby’s motivation.

The question is: will boundary analysis find the black swan before the stopping rule?

Using Boundary Analysis & Findings

For our analysis we used open source documents from two local newspapers (the Pittsburgh Post-Gazette and Pittsburgh Tribune-Review) and blog postings from www.stopthepittbombthreats.blogspot.com, a major platform for crowd-sourcing during the threats. After compiling all the sources we had more 130 news articles and numerous blog posts ranging from January 1, 2012 to August 30, 2012.

Articles that did not contain useful information (e.g. articles about how students coped with threats) were omitted, leaving us with 73 articles that we coded by date in an Excel spreadsheet.  Next, each article was scrutinized for hypotheses, a process that took a single coder approximately 8-10 hours.

Our boundary analysis of the bomb threats yields two findings:

  • Boundary analysis identified the ‘usual suspects’ quickly

In conducting our retrospective boundary analysis we quickly found our stopping rule. In fact, within in a time span of roughly one month, from March to April, almost all of our hypotheses were identified in our documents (see graph). These original hypotheses included typical explanations such as students avoiding exams, students who have conflicts with university administration, pranksters, etc.

Steve Blog 2 Pic 3

The ability of boundary analysis to locate the main hypotheses quickly may also be helpful when combined with hypothesis testing techniques. For example, once the analyst extracts the most common hypotheses he can begin testing each one using a diagnostic technique (for example, alternative competing hypotheses) and move further out on the distribution as needed.

  • The normal stopping rule did not bracket the black swan hypothesis

After an examination of our three data sources, the correct hypothesis—a foreign national from the UK pranking the University—was not identified in the documents. However, we stopped our analysis at the stopping rule, or “knee of the curve.” We do not have enough information to suggest what a good limit to set would be, but applying these same principles to more black swan intelligence cases (the DC Sniper, Eric Rudolph, etc.) would give us a better indication. With more research, we can begin to identify how far past the knee one would need to research to be reasonably confident the black swans are identified. Thus, when unanticipated or abnormal events begin to occur, we do not use ordinary methods for unique circumstances.


While we were unable to bracket the black swan using traditional limits, the two findings have important implications for intelligence analysis. Probably the greatest benefit of boundary analysis could be to give analysts a list of ‘usual suspects’ hypotheses. Analysts can then use diagnostic techniques to whittle down the number of plausible hypotheses. If these usual hypotheses are not useful, the analyst can keep moving to the right of the distribution by extending the boundary analysis or employ an imaginative technique. As we note, an area of future research is conducting more research retrospectively to determine if there is a stopping rule that will catch most black swans.

Ridgway Syllabi

Here are syllabi from courses offered by Ridgway-affiliated professors.  We hope these give students an idea of what to expect. Please note, these are only examples of previous offerings and are subject to change.  Classes are constantly evolving and each professor retains the right to make any alterations.

Dennis Gormley:

PIA 2426: Special Intelligence Topics

PIA 2412-1150: Analyzing Critical International Security Challenges

Ryan Grauer (click here for Dr. Grauer’s personal website and updated content):

PIA 3303: Advanced Seminar in Security Studies

PIA 2434/3434: Civil-Military Relations

PIA 3019: Integrative Seminar in International Affairs

PIA 2303: Security and Intelligence Studies

Michael Kenney:

PIA 2096: Terrorism Capstone

PIA 2327: Terrorism

PIA 2429: War on Drugs

Forrest Morgan (RAND profile):

PIA 2340-1100: Space and National Security

PIA 2352-1140: Strategy and Policy

Jen Murtazashvili (click here for Dr. Murtazashvili’s  personal website):

PIA 2428/3090: State Building

PIA 2505: Post-Conflict Reconstruction

PIA 2492: Politics of Central Asia

PIA 2458: Political Islam




The Dragon of the East Goes Spear-Phishing


Are you sitting down?  Good, because I have some troubling news.  The Chinese are spying on us!  Yes, the sarcastic tone detracts from the gravity of the situation, but the media has been awhirl with the merely symbolic indictment of PLA Unit 61398.  The recent incident involving several Pittsburgh-based companies demonstrates that national security hits home.  The question that remains is: So what?  We spy on allies and companies of interest, and our allies certainly steal our trade secrets.  There is a reason people call espionage the second oldest profession.  But what concerns policymakers and intelligence analysts in the long-term is the shear alacrity and overtness of Chinese cyber espionage targeting American industry.

First, it is important to define terms as scholars, governments, and companies have various definitions.  While there are permutations of each definition, industrial espionage is simply one private company stealing from another; on the other hand, economic espionage involves foreign government-backed activity.  For instance, IBM stealing from Google would be industrial espionage, but China stealing U.S. trade secrets is economic espionage; this article focuses on the latter.   The issue becomes increasingly muddled with state-owned enterprises and private defense contractors, but the concepts remain the same.

Another key distinction is between cyber war and cyber exploitation or spying.  Although there is little differentiation in academia, cyber war is more about offensive penetration capabilities.  Stuxnet, the U.S./Israeli virus that attacked Iranian nuclear centrifuges is an example of cyber war.  Cyber espionage or exploitation is also penetrative, but involves the covert collection of data rather than attack or destruction of systems.  This is the type of operation PLA Unit 61398 is accused of committing.  The difference is subtle, but the policy and strategic implications, subsequently addressed, are significant.


In 2001, the FBI released a report claiming that 22% of the industrial “suspicious activity” was government-sponsored.  However, a deeper look at the pilfering of American technology illustrates a graver threat to national security; the cost is potentially staggering.  The Center for Strategic and International Studies produced this table in 2013 to illustrate the potential damages:

tableCSIS estimates that global and U.S. cyber-crime at most costs 1.4% and .8% of GDP or $1 trillion and $120 billion, respectively.   Conversely, the FBI testified during a House subcommittee on Counterterrorism and Intelligence that the loss to U.S. companies is approximately $13 billion.  Nevertheless, the disparate estimates illustrate a key point:  there are significant barriers to quantifying the costs of economic espionage.  In the end, loss of intellectual property is difficult to calculate.   Simply assessing the opportunity cost of research and development does not tell the whole story.

This problem is pertinent to national security as well.  Chinese military officers stealing Alcoa’s aluminum trade secrets has the potential to damage the economy and U.S. companies.  But what if China stole secrets that can enhance nuclear weapons?  Well it did, albeit over fifteen years ago.  Defense contractors and military technology is on the top of the list for foreign governments.  A recent commission by defense contractor Northrop Grumman to assess China’s capacity to conduct cyber warfare and espionage labeled it as the “…single greatest threat to U.S. technology…”  Countless examples of Chinese cyber espionage operations and subsequent investigations, cat and mouse games, and code-names such as Ghost Net, Aurora, and Shady Rat epitomize Sino-U.S. cyber relations.  There are myriad sources detailing Chinese industrial and economic espionage in books, reports, and house review, yet this is nothing new.

Some researchers argue that the Chinese ethos embodies intelligence collection. Military and strategic intelligence is ingrained in Chinese culture, originating with Sun Tzu’s Art of War (sunzi bingfa).  This is not an ethnocentric accusation, but merely an observation held by scholars and intelligence analysts.  Intelligence and the Art of War is applicable in various facets of Chinese culture.  Four years ago I studied Chinese business culture at Fudan University in Shanghai.  We were required to memorize lines from Sun Tzu and apply it to contemporary business models.  Yet the claim that Chinese steal more aggressively due to cultural propensities is unfounded.  In the past, some of the greatest perpetrators of economic espionage against the United States were the French and Israelis.  The shift is simply because China is a rising power, no different from the expansion of Soviet espionage during the Cold War.  In this case, however, the high-tech advancements in information and communication technologies (ICTs) create attribution problems and permit government agents to operate thousands of miles away. In sum, as long as there is a technological gap or economic advantage, countries will spy and steal, and China is no different.


This does that change the fact that China is presently the consummate pilferer of U.S. technology and something must be done.  First, we need to know how much economic espionage is truly hurting the U.S. economy.  There is no doubt that stealing military secrets from a defense contractor will weaken the U.S. relative to other nations, but the jury is still out on the impact of cyber spying on U.S. companies.  The House Committee on Counterterrorism and Intelligence claims that foreign government-backed corporate espionage is costing U.S. jobs and billions of dollars.  There are many examples of companies going under after Chinese, French, and Israeli stole secrets, but, as previously mentioned, the estimates range greatly.  The U.S. needs to enhance its ability to articulate, quantify, and communicate losses due to economic espionage.

American security policy also includes the enhancement of cyber capabilities, but the investment in resources is misplaced.  The establishment of U.S. CYBERCOM is one way the U.S. beefs up its attack and defense capabilities.  But the enlistment of cyber-warriors, as necessary as it is, is the incorrect response to cyber espionage.  Cyber-warriors are for cyber-war, not necessarily cyber exploitation or espionage. The Chinese are not looking to engage the U.S. in a cyber-war, or a war on any battlefield in the near future.  American military prowess and technological capabilities are unequalled. So the Chinese (and other nations) will continue to steal in an attempt to level the playing field.  The U.S. needs cyber spies, not warriors, and the FBI is falling behind.  In fact, the agency has begun to recognize this after the recent indictment of the PLA Unit.  Several days after the announcement, the FBI is considering changing its drug policy to encourage young hackers and computer programmers to apply. Explicit cyber-war is a long way away, but cyber spying has been here for quite some time and there is no indication it is going anywhere. China has become increasingly aggressive and more overt it its tactics, but, in the end, art of war has not changed.