Securing the Information Highway

20 October 2009

The November/December issue of Foreign Affairs (unfortunately not yet available online as of this writing) has an eponymous piece by Wesley Clark and Peter Levin: in it, they write that

[The limited success of the July 4 cyberattacks principally affecting the US and Korea] may embolden future hackers to attack critical infrastructure, such as power generators or air-traffic-control systems, with devastating consequences for the U.S. economy and national security…There is no form of military combat more irregular than an electronic attack: it is extremely cheap, is very fast, and can disrupt or deny critical services at the moment of maximum peril…Disturbingly, [other nations] seem to understand the vulnerabilities of the United States’ network infrastructure better than many Americans do…The longer the U.S government waits [to confront the real threats in cybersecurity], the more devastating the eventual assault is likely to be…the consequences of a major breach would be catastrophic.

Clark and Levin recount William Safire‘s claim that a 3-kiloton explosion of a Siberian natural gas pipeline in 1982—”the most monumental non-nuclear explosion and fire ever seen from space”—was the direct consequence of a Trojan inserted into Canadian SCADA software that the CIA allowed the KGB to steal. They recirculate the rumor that the Israeli destruction of a purported Syrian nuclear facility in 2007 was facilitated by a cyberattack targeting Syrian air defense systems. (This blog has linked to other reports of capabilities along similar lines, such as this one.)

But their real focus is (not surprisingly, given Levin’s history as founder of a hardware outfit specializing in the area) is on the problem of validating hardware. DoD has been very concerned with the idea of hardware Trojans the last few years. Nobody in the military/intelligence-industrial complex wants to take it on faith that chips that are manufactured in China or Taiwan don’t have backdoors. There are apparent precedents for the hardware Trojan such as old reports involving Crypto AG. So NSA started up a trusted foundry and DARPA started the TRUST program (whose PM funded some of my research some years back, so I applaud his taste on both counts). But that leaves the vast majority of chips in network components still unaccounted for, including a large number of counterfeit chips.

Clark and Levin propose an emphasis on reconfigurable hardware (such as FPGAs) and the sort of immunological paradigm started by the Forrest group at UNM as an example of a sound defensive strategy. While the practical utility (as compared to the undeniable conceptual elegance) of the paradigm for network defense is not clear to me (but then again I’m obviously a partisan when it comes to the best scientific principles for designing network defense infrastructure), the ideas of using reconfigurable hardware and avoiding a computational and network monoculture that goes hand-in-hand with immunological principles are sound ones that I’ve agreed with for some time. I gained an appreciation of the benefits of FPGAs from performing research on algorithms for reconfigurable computing architectures some years back, and at a conference last year I got into a brief argument on the security dangers of monocultures with a government sysadmin who lauded the monolithic computing infrastructure he maintained. So it’s not a stretch to say that I am extremely sympathetic to their point of view.

Clark and Levin close by highlighting the need for open infrastructure–both reconfigurable hardware and open source software, and (insofar as it can be implemented) this is the entirely correct approach to technological security of any form. As Reagan said: “trust, but verify”.

update 10/23: The article is available here (subscription required).

Leave a Comment » | Commentary, Communications security, Equilibrium Networks, Nonrandom bits, Security | Permalink
Posted by eqnets

VizSec09

12 October 2009

VizSec 2009 was yesterday; aside from Bill Cheswick’s keynote and participating in the poster session (the poster is available on our downloads page), I was pleasantly surprised by Joel Glanfield et al.’s OverFlow work. Like us, they have recognized that aggregating IPs (among other things) is a good thing for visualizing network traffic, particularly over time. One thing OverFlow does that we don’t is to explicitly show a representation of the aggregated connections as a graph drawing. When aggregation is done statically (even including layer 4) this seems like the sort of thing that can be very friendly to analysts, but there are occlusion issues that suggest focusing on one aggregated node at a time, especially for time series data. Anyway I look forward to seeing more of this sort of thing and fewer “yarn ball” visualizations and their ilk that too often convey little or no useful information because of a refusal to recognize one of the great lessons of physics: that successfully analyzing complex systems is largely dependent on identifying relevant spatial and time scales and then ignoring irrelevant details. When I heard people saying that analysts complain that visualizations frequently “get in the way of the data” I think I know what they meant.

One thing I was pleasantly not surprised by is that the afternoon panel seemed to repudiate the notional equation “Security + Visualization = Science”. As I’ve commented here (and there), there can be no truly scientific theory of security. Visualization doesn’t change this. The place where security and visualization can overlap with each other and with science is in the development of frameworks guided by scientific principles, both in architectural and cognitive aspects. For example, an immunology-based security visualization tool might seek to leverage some kind of corresponding visualization, like some sort of graph summarizing “antibodies” that draws from biologists’ experience.

But trying to compare different visualizations scientifically is almost surely doomed to failure outside of a “perturbative regime” where small elements of visualizations are altered and the cognitive effects are measured. For instance, comparing Wireshark and TNV might be done carefully and provide some insight, but it does not qualify as science. And it doesn’t need to. Engineering is a good thing, and so are usability studies. But while we certainly base our own framework on principles from physics, we haven’t bothered with trying to do formal usability studies, because people will make it known if or when they want minor improvements to an interface, and that’s precisely the sort of thing that falls into the “perturbative regime” anyway. I think the bottom line is that if you care about how users interact with your tool and what it can do for them, just let them have a say in the development process.

1 Comment | Commentary, Communications security, Equilibrium Networks, Networks, Nonrandom bits, Science, Security | Permalink
Posted by eqnets

Cyberterror: menace or myth?

8 October 2009

Today I went to a talk by Irving Lachow at the think tank where I used to work on whether cyberterrorism is a myth or a menace. The conclusion is probably obvious: it’s basically not a real menace. There’s lots for everyone to worry about from organized crime and certainly lots more for the government-industrial complex to worry about from nation-state threats, but cyberterrorism is a mirage, and basically everyone who doesn’t have skin in the game (and even some people who do or might–such as myself) agree. I’ve talked about similar themes here and here and here, among other entries on this blog.

Lachow mentioned that he did his research on this topic through unclassified sources, then went onto JWICS with crossed fingers–and ended up standing by his initial conclusions. Terrorists do organizational and support stuff and “influence operations” (I guess this is a subtler version of PSYOPS) using networks, but they don’t really engage in cyberterrorism. And the stuff that gets called cyberterrorism basically doesn’t deserve the name. Lachow believes, as I do, that crime, espionage, and state-level network attacks are what we really ought to be concerned with: cyberterror should be considered a “lesser included threat”–although the risks for all of these threats will only increase with time.

One theme that he brought up that I and many others have mulled over is cyberdeterrence. Basically, nobody knows how to do it. The nuclear analogies are false. But (as I’ve mentioned here) for a really big attack, one that’s worthy of a strategic offensive move by a nation-state or terrorist group, there will be a kinetic component. Attribution won’t be a problem. Old-fashioned deterrence with guns still works just fine.

3 Comments | Commentary, Communications security, Equilibrium Networks, Nonrandom bits, Security | Permalink
Posted by eqnets

Arquilla on the cyberoffensive

22 September 2009

From a Wired blurb covering John Arquilla’s ideas about cyberdeterrence:

Armies (even guerrilla armies) are so dependent on digital communications these days that a well-placed network hit could hobble their forces. Do these cyberattacks right—and openly—and the belligerents will think twice before starting trouble. Arquilla calls his plan “a nonlethal way to deter lethal conflict.”

Sure, it’s risky. A misinterpreted or misattributed attack could inflame tensions. Or you might fritz the good guys and civilians by mistake. But Arquilla says this “kinder, gentler deterrence” is better than threatening to strangle an adversary’s economy or reduce its cities to radioactive cinders.

Over the past decade I have maintained the hope and at times even a tentative belief that cyberwar might be “kinder, gentler” war. I still hold that hope, but not the belief.

History and common sense have shown over time that militaries seek to make war more controllable for themselves and more chaotic for their opponents. But the nature of opponents has changed, and has typically broadened unless or until combatants are able or willing to sacrifice an advantage in raw killing power. The concept of total war has evolved with the Grande Armée, the March to the Sea, unrestricted submarine warfare in World War I, strategic bombing in World War II, and the nuclear hostage-taking of the Cold War. I suspect that the next total war will be organized around cyber. And make no mistake, the cyberwar will be unpleasant and sometimes lethal to noncombatants. But it will not be the only aspect of that war.

No military is going to forsake kinetic strikes for overt cyber strikes in the foreseeable future. Even assuming the effectiveness of a successful cyber strike, the reliability usually isn’t there. And if something is worth an overt strike without a reliable offensive cyber capability, then it’s worth a kinetic strike. Since there is no reason to believe that cyber can be anything but a complement to kinetic anytime soon, the idea of cyberdeterrence is meaningless without the more traditional forms of deterrence.

It’s worth noting that of the three scenarios mentioned in the Wired blurb, two are about the US interfering in a conflict between other states (India v. Pakistan and Russia v. Georgia) and one is about setting up honeypots for terrorists à la Dark Market. Nowhere does it mention an instance where the US is really a combatant. There’s a good reason for this: the US can deter other actors precisely because of its unparalleled strength. If the US launches an overt cyber strike, you can bet that it will be prepared to precipitate kinetic consequences and to get into a “real” fight.

Would we really attack India’s and Pakistan’s nuclear C2 in order to keep them from using nukes on each other? Not likely. It would be hard if not impossible to do effectively, might backfire in any number of ways, and the threat of US interference certainly wouldn’t deter them any more than nuclear war would.

Would we really deploy a “cyberdeterrent squad to disrupt the Russian military’s communication networks” over Georgia? Not likely. Georgia isn’t worth an essentially strategic strike against Russia in any event, especially if we couldn’t commit to a larger conflict that might emerge.

The “kinder, gentler deterrence” Arquilla talks about is little more than a proverbial shot over the bow. And it only works because of our real guns.

Leave a Comment » | Commentary, Communications security, Networks, Nonrandom bits, Security | Permalink
Posted by eqnets

Graphene

17 September 2009

Whenever I browse the cond-mat archive these days I’m constantly amazed at the number of papers dealing with graphene. And a lot of them look really interesting, for example this one from yesterday. There’s a lot of interest among theorists and experimentalists in graphene as an experimentally accessible substrate for two-dimensional quantum field theory, and there’s a lot of ways in which this material might be applied, especially in electronics.

A quick lookup shows the growth of papers in cond-mat with “graphene” in the title took over a year to take off since the first big paper dealing with it, but boy did it take off:

graphene

The number for 2009 is a projection that assumes a constant rate of papers per month in a given year (i.e., 458 papers so far should lead to about 643 for 2009).

The interesting thing to me about this graph is that it looks like a logistic. This is what you’d expect the dissemination of information to behave like, and my eighth-grade science project was on modeling the spread of rumors this way. On the other hand, the graph also looks like part of a Gaussian (though the eyeball fit doesn’t look as good, I won’t bother with numerics in either case since there’s not very good data), but graphene research is almost certainly not a flash in the pan–there are too many potential applications. But since the inflection point has been passed I think it may be time to consider graphene as a distinct discipline within condensed matter theory.

1 Comment | Commentary, Nonrandom bits, Science | Permalink
Posted by eqnets

MIL-OSS

13 August 2009

Yesterday I spoke at MIL-OSS at the Georgia Tech Research Institute (thanks to the audience and the organizers) and talked about our technology and open source strategy, but the highlight of the day for me was when my former colleague David Wheeler explained Jim Stogdill’s phrase “code is maneuver” and its implications for open source: because the effectiveness of militaries will be increasingly dependent on their software, the ability to modify, patch, and improve that software (and of course to secure the systems and networks running that software) will be increasingly decisive in conflicts.

When open source gets pitched to the Hill and AT&L, a strong case can be made for it on national security grounds: government purpose rights mean less in practice than they do in theory, and Uncle Sam should have the four freedoms for (e.g.) the F-22 avionics code, and not just for webservers. (More generally, anyone who pays for custom code should either press for these freedoms or get a deep discount.) It’s important here to note that open source does not need to mean “released to the public”: it’s really not that hard to license even highly classified code under an open-source license and still deal with classification and ITAR restrictions appropriately.

There will be a lot of inertia against this. Procurement and management execs care more about how much money they control than almost anything else, and the easiest way to spend a lot of money (as someone commented during a panel session) is to write the same software over and over again. But it’s important to note that open source software generally creates wealth and markets for the public, even if it adversely affects the bottom line of pure-proprietary software companies. Their business models are not an excuse to needlessly duplicate time, money, and effort on projects–and certainly not to artificially impede the performance of our military and government.

2 Comments | Commentary, Equilibrium Networks, Nonrandom bits, Security | Permalink
Posted by eqnets

Common data sets and the illusion of scientific security testing

12 August 2009

In my book there is nothing as good as real data produced by a red team, except captured data produced by a red team from NSA (even if it’s not really annotated or labeled well and their MO isn’t quite what it would be in practice). When I was involved with a past IDS research effort in the early 2000s there was a great deal of emphasis on the DARPA/Lincoln Labs datasets, which were old even back then. They were a lot better than nothing, but one thing that concerns me and most everyone else is the lack of good common data, let alone reproducible testbeds like the National Cyber Range is supposed to provide. So it is nice to see that the DARPA/LL datasets are dead–long live the 2009 CDX datasets.

(Wireshark opened the small border data capture fine on my laptop, so don’t let the lack of .pcap extensions bother you.)

But one often-implied corollary of having common or reproducible input data troubles me. Some folks have got the idea that it is possible to scientifically evaluate computer security systems. Even with good input data, I don’t believe such a thing is really possible except in an extremely narrow sense. Let me explain by way of analogy.

Suppose someone came to you with a box of padlocks of the same model and asked you to scientifically evaluate the security of that padlock model. There are a few things you could do that would be obvious. You could test mechanical properties scientifically, asking questions like: How much force does it take applied in such-and-such a way to produce a mechanical failure of the lock? What is the dominant failure mode that results? but it is very implausible to imagine that you could evaluate all the possible failure modes–and hence the actual security of the lock–scientifically.

Sticking with the mechanical failure modes: what if someone decides to use acid to dissolve the lock? or liquid nitrogen to make it brittle? or heats it with an acetylene torch? And maybe a cold, brittle lock is easier to pick; or a hot, ductile lock is…you get the picture. And this doesn’t even begin to address lockpicking in all its forms, which is equal parts art and science.

(BTW/FWIW: one of my favorite episodes from college involves breaking into a room [that I was allowed to be in] that was secured with a fancy keypad lock system using nothing more than a piece of string from an interoffice envelope. It was after a power outage, and the keypad was inoperative, but the folks that installed the lock didn’t think about a very simple mechanical failure mode.)

In the real world you can usually expect a combinatorial explosion of possible failure modes that would have to be tested to assure security. Even in quantum cryptography people rightly worry about things that aren’t in the formal protocols, like efficiencies and TEMPEST-type issues with photon detectors. One of the reasons people are so excited about quantum crypto in the first place is that it is, among other things, a truly credible attempt to use physical theory to reduce the number of failure modes in a security protocol. And one of the reasons I don’t bother to pay attention to formal security proofs outside of cryptography is that their assumptions are never credible to a degree comparable to the Bell inequalities.

This is not to say that security systems shouldn’t be tested–of course they should (especially if there is a “proof” of security)–but it doesn’t make sense to read too much into the results if they’re good. (If your results from evaluating a security system are bad, then that security system is not for you, regardless of why.) In science a hypothesis can never be proved, only disproved. And in security evaluation a system can never be proven secure, only broken. The difference is that in science the hypotheses can be deductively identified and tailored to test good theories that seek to reflect a underlying objective truth of big-n Nature; in security evaluation the system can only be used to test attacks that seek to reflect the ingenuity of one particular set of red team tactics, for which there is often no underlying objective validity, just a common-sense notion of what ought to be done. The domain of applicability of any security evaluation is fundamentally limited because there is no way to come up with a scientific theory of security. Science typically deals with establishing and understanding regularities in phenomena, while security evaluation typically deals with the opposite.

I was hoping to be able to (but can’t) make it to a meeting in Seattle at the end of the month that is trying to produce

progress in the area of Quantifiable Scientific Evaluation of CyberSecurity research. Currently, there is no well understood scientific standard used to guage [sic] the quality of research results in this area. Instead, decisions are made by program committees and journal editors.  Also, experimental results are often not repeatable, sometimes due to the proprietary nature of the code or the privacy of the data. This meeting seeks to establish the beginnings of an agreed-upon set of scientific standards whereby progress can be measured, and identify barriers to such standards.

Since I can’t be there, I will just say this: Concentrate on getting good, normalized inputs and outputs for comparative security evaluations. That is plenty hard enough, even though it is not science except in a trivial sense. If a goal is to use nontrivial science in security research, try applying ideas from science (like immunology or my favorite, statistical physics) and mathematics in the development of engineering principles for security systems–where it can be of some benefit–rather than in the evaluation of systems, where anything nontrivial that can be done might be valid and statistically significant and of practical engineering value, but is still probably not scientific.

Anyway, my hat is off to the CDX guys for putting those pcap files and logs up.

1 Comment | Commentary, Communications security, Equilibrium Networks, Nonrandom bits, Science, Security | Permalink
Posted by eqnets

Scaling up ion trap quantum computing

9 August 2009

Science has just published a paper by Dave Wineland’s group at NIST that describes an experimental implementation of

a combination of all the fundamental elements required to perform scalable quantum computing using qubits stored in the internal states of trapped atomic ions.

(Technology Review has a layman’s overview here.)

It has been eight years since I really kept up with quantum computing, but this result is clearly very important. I recall hearing about the idea of a “quantum bus” to shuttle ions back and forth in order to scale up ion trap quantum computing at a dinner that I was at along with Wineland in 2000, and the idea appears to date from 1998 or even earlier. There were no obvious showstoppers, his group (among others) was good and certain to be adequately funded for as long as it would take, and so it is not that big a surprise to see this working now.

But even though the transport of ions in traps had been done before, this current result has built on of over a decade of work. An idea apparently developed over the last few years that I don’t recall from those early days (it seems to date from a 2003 paper) involves using Mg+ “refrigerator” ions to sympathetically cool qubit-carrying Be+ ions by

using a combination of Doppler cooling and resolved sideband cooling on the 24Mg+ ions. Importantly, the cooling light only interacts with 24Mg+, leaving the qubits stored in 9Be+ intact.

Although the fact that it took so long for them to get here indicates just how difficult the physics and engineering challenges are, more results extending this work (probably starting with improving the fidelity, with longer-term goals of working with multidimensional trap arrays and finding good ways to scale up the entanglement distribution) are certain to follow.

In the last paragraph of a review paper on quantum algorithms and protocols I wrote in 2000, I said that “it is appropriate to say some words about whether a scalable quantum computer will in fact ever be built. [I believe] that such devices could well be built within 20 years.” The Wineland group’s latest result makes me feel better about that statement (and worse about using RSA for anything long-term) than I have in the last couple of years.

Update: Speaking of “certain to be adequately funded for as long as it would take”, I just noticed a Nature article from June talking about IARPA cutting funding to Wineland’s group. From the article:

In 2007, the newly created IARPA took over funding for quantum information science from the National Security Agency. The following year, IARPA stopped funding the NIST researchers because, it says, it did not want to fund other government agencies…the NIST funds ran out while [IARPA] managers were reviewing the programme and deciding how NIST might be involved.

While IARPA’s stance is understandable, this sort of disruption is not conducive to National Medal of Science-level work, even if as NIST said

It’s ultimately the responsibility of NIST to make sure these programmes receive the resources they need, and we are committed to ensuring they remain adequately funded.

2 Comments | Commentary, Nonrandom bits, Science | Permalink
Posted by eqnets

Hacking nuclear command and control

24 July 2009

A recent paper by Jason Fritz available at the International Commission on Nuclear Nonproliferation and Disarmament notionally discusses hacking nuclear command and control. The paper is mostly a cursory overview of nuclear C2 with speculation added, e.g. “such systems might depend on X which hackers might be able to exploit [in some unspecified way Y]“. Some choice quotes:

If access to command and control centres is obtained, terrorists could fake or actually cause one nuclear-armed state to attack another, thus provoking a nuclear response from another nuclear power.  This may be an easier alternative for terrorist groups than building or acquiring a nuclear weapon or dirty bomb themselves. …

Efforts by militaries to place increasing reliance on computer networks, including  experimental technology such as autonomous systems, and their desire to have multiple launch options, such as nuclear triad capability, enables multiple entry points for terrorists.  For example, if a terrestrial command centre is impenetrable, perhaps isolating one nuclear armed submarine would prove an easier task.  There is evidence to suggest multiple attempts have been made by hackers to compromise the extremely low radio frequency once used by the US Navy to send nuclear launch approval to submerged submarines. …

A sophisticated and all encompassing combination of traditional terrorism and cyber terrorism could be enough to launch nuclear weapons on its own, without the need for compromising command and control centres directly. …

It may take years to prepare an attack against advanced networks, including the identification of exploits, development of tools, and the implementation of a plan, yet technology is rapidly advancing and networks continually updating, possibly disrupting those plans.  Terrorist organisations may not be able to keep up with the massive financial backing of nation states.  State-sponsored hackers have this problem themselves. Despite the possibility of exaggerated claims, a threat remains…

Cyber terrorists [seeking to provoke a US nuclear launch through spoofing] would not need deception that could stand up over time; they would only need to be believable for the first 15 minutes or so. …

Some reports have noted a Pentagon review, which showed a potential “electronic back door into the US Navy’s system for broadcasting nuclear launch orders to Trident submarines”.  The investigation showed that cyber terrorists could potentially infiltrate this network and insert false orders for launch.  The investigation led to “elaborate new instructions for validating launch orders”. …

Nuclear command and control structures are vulnerable to cyber terrorism…Inherent flaws in current nuclear postures provide increasing opportunities for computer exploitation. Despite claims that nuclear launch orders can only come from the highest authorities, numerous examples point towards an ability to sidestep the chain of command and insert orders at lower levels.  Cyber terrorists could also provoke a nuclear launch by spoofing early warning and identification systems or by degrading communication networks.

The juicy-looking bit about hacking SSBNs is from this reference, and the relevant quote is below:

The sobering results of the still- classified work by a Pentagon “Commission on Nuclear Fail-Safe” – to which [Bruce] Blair testified about Soviet nuclear safeguards, inside a vault at the Pentagon around 1992 – point to US vulnerabilities that could also apply to Russian systems today. Investigators found an “electronic back door” into the US Navy’s system for broadcasting nuclear launch orders to Trident submarines.

“This deficiency allowed unauthorized hackers, which could be terrorists or high school mischief makers, to potentially insert a launch order and transmit it to the Trident,” Blair says. The gap was so serious that Navy launch order verifications had to be revised.

Notice the bit about 1992.

After reading this paper and having studied nuclear weapons policy both in coursework and informally over the years (to illustrate, a review I wrote in 2007 for a book on Chinese nuclear policy is here), the paper struck me as highly speculative. Despite this, it may get a lot of sensational attention, which would be bad.

The folks at STRATCOM have all seen WarGames (just a few years ago, I shared an office with a STRATCOM O-5 who loved to talk about things nuclear), and despite the USAF’s well-publicized rccent gaffes, the US military does not take nuclear C2 lightly. Having met quite a few workers in the Russian nuclear establishment visiting a US nonproliferation institute, I feel justified in saying that Russia does not take nuclear C2 lightly either. More generally, anyone that is willing to “eat grass” to develop nuclear weapons is going to safeguard those weapons as carefully as they can, and initiatives like Nunn-Lugar go further by providing US help in securing nuclear materials.

What’s more, the dependence of most states’ nuclear C2 systems on networks is far from clear, and the corresponding vulnerability to information warfare even less so. Mr Fritz can’t be blamed for relying on OSINT, but the result is a work that does not begin to answer any of the by-now familiar questions it raises.

4 Comments | Commentary, Communications security, Nonrandom bits, Security | Permalink
Posted by eqnets

Why Poissonian traffic models matter more now than ever, part 3

21 July 2009

A paper by Karagiannis et al. that appeared around the same time as Veitch et al. and called “A Nonstationary Poisson View of Internet Traffic” provided concrete evidence to support the convergence to Poissonian packet interarrivals. Mindful of the same short-timescale issues as Veitch et al., Karagiannis et al. remarked that “the packet interarrival time distribution may deviate from the Poisson model for very small values because of multiple-packet deterministic sequences” due primarily to buffered packets in the upstream router. However, this should not be a factor except as the link becomes saturated, which is precisely the caveat of Cao et al. detailed earlier.

Karagiannis et al. examined traces from multiple links, including an OC-48 backbone link and a 100 Mbps trans-Pacific backbone link in 2003, and concluded that (formatting as in the original)

•    Packet arrivals appear Poisson at sub-second time scales: The packet interarrivals follow an exponential distribution. In addition, packet sizes and interarrival times appear uncorrelated…
•    Internet traffic is nonstationary at multi-second time scales: We demonstrate that traffic oscillates around a global mean, in a piecewise linear manner.
•    Internet traffic exhibits long-range dependence (LRD) at large time-sacles: In agreement with previous findings, we observe that Internet traffic is LRD at scales of seconds and above.

They went on to note that

…Our work suggests that Poisson models should not be abandoned especially in the Internet backbone with high speeds, and huge levels of traffic multiplexing.

With respect to the OC-48 data, Karagiannis et al. also showed that

For interarrival times, independence holds for 20,000 consecutive packet arrivals…

Moreover,

To stress-test the claim for the memoryless properties of Poisson arrivals and independence, we studied bursts of packets…We find that the distributions of the duration of the busy/idle period…are well approximated by exponential distributions. This is irrespective of the interarrival time that is used as the boundary for distinguishing between idle and busy periods.

Karagiannis et al. concluded by noting that

this type of traffic model (i.e., Poisson with nonstationarity at multi-second scales) is consistent with the kind of long-range dependence that is commonly observed in network data over larger time scales…we expect the traffic characteristics for the Internet backbone to continue to grow even better behaved in the future.

The coup de grâce in the fifteen-year saga of packet interarrival behavior may have been delivered in early 2009, when Gÿorgy Terdik and Tibor Gyires delivered a paper called “Does the Internet Still Demonstrate Fractal Nature?” In this paper Terdik and Gyires noted the work of Veitch et al. and remarked that

a Poisson cluster process could model the aggregate traffic where the packet interarrivals within individual clusters of each flow could be characterized by an overdispersed Gamma distribution.

Analyzing data from an OC-192 link in 2008 with this in mind, Terdik and Gyires found that

the burstiness of the interarrival times decreased significantly compared to earlier traces…Furthermore, we found that in many traces the distribution was Poisson deviating from previous observations. Therefore in answering our original question, we can conclude that based on the sample traces, the Internet is losing its self-similar nature that was so prevalent for years.

And there you have it. It is a subtle picture of evolving behavior for network packet interarrival times, but the central point is clear: network traffic is becoming increasingly (and at high speeds already appears to be) Poissonian. This simplification means that a lot of elegant and powerful mathematical techniques that were not ever considered for profiling network traffic because of earlier results in the nineties are actually increasingly likely to be the appropriate way to handle things now and in the future. It may also mean that the debates you hear now about network capacity are not going to continue for many more years, because the telcos will be able to plan better. (Or they may continue simply because it’s to the telcos’ advantage to have such debates.)

In the next post in this series, I’ll discuss some mathematically oriented ideas that seek to exploit the emergence of Poisson traffic to provide a lasting foundation for scalable network profiling algorithms.

1 Comment | Commentary, Equilibrium Networks, Mathematics, Networks, Nonrandom bits | Permalink
Posted by eqnets

« Previous Entries
Next Entries »
Follow

Get every new post delivered to your Inbox.