Happy Thanksgiving

26 November 2009

I'm thankful for seeing truth presented with beauty.

This is a picture to help understand an Anosov flow obtained from the cat map. It’s part of research on a technique we’ve used to analyze network traffic.


DIMACS workshop on designing networks for manageability

14 November 2009

The highlight of the DIMACS workshop on designing networks for manageability for me was Nick Duffield‘s talk on characterizing IP flows at network scale. The basic idea is to use machine learning to identify the flow predicates that best reproduce packet-level classifications. By sampling flows according to a simple dynamical weighting, Duffield et al. demonstrate that this sort of flow classification is accurate (to a few percent, with the misclassifications largely due to overloading of HTTP, e.g., with media over web), scalable (i.e., faster than real-time), versatile (i.e., independent of the particular ML classifier), and stable (over space and time, with a deployment on a separate but similar network producing essentially equivalent results over several months). This work is more recent than related research we’ve cited in our whitepaper “Scalable visual traffic analysis” (on our downloads page) detailing the rationale behind our own traffic aggregation methods.

Much of the workshop (especially its first day) was more focused on current deployment and engineering issues than I would have thought for an overarching focus on “algorithmic foundations of the internet”. Both another mathematician that came with me and I expected to see some work on (or at least suggesting the use of) sparse linear algebra to deal with traffic matrices. I was surprised not to see anyone talk about some kind of agent-based configuration methods for networks–this sort of approach has been used to great effect on hosts.

But there were a number of other talks I found interesting: Aditya Akella from Wisconsin talked about an entropy characterization of “reachability sets” describing packets that can be sent between pairs of routers based on their configurations, and used this to construct a routing complexity measure for networks. Dan Rubenstein from Columbia talked about a “canonical graph” method for efficiently detecting misconfigurations for routing protocols. Iraj Saniee talked about why networks are globally hyperbolic (using a result of Gromov’s well-known work on groups), a conclusion that seems intuitively obvious to me if the existence of a global curvature (bound) is assumed. (Basically a network spreads out if it’s drawn in any reasonable way, and hyperbolic geometry amounts to expansion.)

Mung Chiang from Princeton talked about the results in “Link-State Routing with Hop-by-Hop Forwarding Can Achieve Optimal Traffic Engineering” first presented at INFOCOM 2008. He and coworkers perturb assumptions behind routing protocols to obviate the need for hard optimization problems (i.e., computation of optimal link weights to input to OSPF is NP-hard, but changing OSPF can make the corresponding optimization problem easier). From what I could tell OSPF corresponds to a “zero-temperature” protocol, whereas the improved protocol corresponds to a “finite-temperature” one.

Michael Schapira from Yale and Berkeley talked about game-theoretic and economic perspectives on routing. It is a happy “accident” that the internet is BGP stable (usually, although a notable event where a Pakistani ISP set all its hop counts to 1 some time ago created a routing “black hole”). Although ISPs are selfish, economic considerations tend to result in stability. But that’s not a guarantee. So Schapira and coworkers analyzed the situation and found that “interdomain routing with BGP is a game” in which the ASes are the players, the BGP stable states are pure Nash equilibria, and BGP is the “best response“. I mentioned to him that the “accidental” nature of this stablity is likely due to reciprocity, in that an ISP that discovers one of its neighbors engaging in predatory routing is likely to retaliate in the future. I think the use of economic and game theory is generally a good idea. An emphasis of the economics of cybercrime has developed recently, and understanding the market forces at play here and elsewhere is likely to lead to improvements in the reliability and security of networks.


Random bits

10 November 2009

“Brazilian government officials disputed the report [that cyberattacks had taken down power grids in Brazil] over the weekend, and [the] director of [Brazil's] Homeland Security Information and Communication Directorate…found no evidence of hacker attacks, adding that Brazil’s electric control systems are not directly connected to the internet.”

A purported proof of the four-color theorem…without using computers

http://www.wired.com/threatlevel/2009/11/brazil_blackout/

Solution of second-order matrix difference equations

9 November 2009

While browsing through my notes recently I came across this cute and not-quite trivial, but entirely elementary result. Since it doesn’t seem to be available anywhere else, I present it as blog fodder. (As usual, I will offer $.50 and a Sprite to the first person who provides a usable reference in the event that this result is already known.)

Consider the second-order matrix difference equation

x_{k+1} = Ax_k + \tilde Ax_{k-1}

with x_0, x_1 given. Some algebra shows that (e.g.)

x_2 = (A)x_1 + \tilde A x_0,

x_3 = (A^2 + \tilde A)x_1 + (A) \tilde A x_0,

x_4 = (A^3 + bsp_{1,1}[A, \tilde A])x_1 + (A^2 + \tilde A) \tilde A x_0, etc.

where the bosonic symmetrized product bsp_{j,k}[A, \tilde A] for two “species” of order (j,k) is defined as the sum of all distinguishable products with j occurences of the first species and k of the second (if j or k = 0 then the BSP is a pure power of the nontrivial species).

For example, bsp_{3,2}[A,\tilde A] equals

A A A\tilde A \tilde A + A A\tilde A A \tilde A + A A \tilde A \tilde A A + A\tilde A A A \tilde A + A\tilde A A \tilde A A

+ A \tilde A \tilde A A A + \tilde A A A A \tilde A + \tilde A A A \tilde A A + \tilde A A \tilde A A A + \tilde A \tilde A A A A.

It is not hard to verify that

bsp_{j,k}[A, \tilde A] = A \cdot bsp_{j-1,k} + \tilde A \cdot bsp_{j,k-1}[A, \tilde A]

and more generally for \ell = 0,\dots \min(j,k) that

bsp_{j,k}[A, \tilde A] = \sum_{m=0}^\ell bsp_{\ell-m,m}[A, \tilde A] \cdot bsp_{j-(\ell-m),k-m}[A, \tilde A].

The same algebra that facilitated writing down the first few terms for the difference equation quickly leads to the general solution, viz.

x_k = \left( A^{k-1} + \sum_{j=1}^{(k-1)/2 - 1} bsp_{k-2j-1,j}[A,\tilde A] + \tilde A^{(k-1)/2} \right) x_1

+ \left( A^{k-2} + \sum_{j=1}^{(k-1)/2-1} bsp_{k-2j-2,j}[A, \tilde A] \right) \tilde A x_0

for k odd, and

x_k = \left( A^{k-1} + \sum_{j=1}^{k/2 - 1} bsp_{k-2j-1,j}[A,\tilde A] \right) x_1

+ \left( A^{k-2} + \sum_{j=1}^{k/2-2} bsp_{k-2j-2,j}[A, \tilde A] + \tilde A^{k/2-1} \right) \tilde A x_0

for k even.

I used this to understand what happens when one of the internal states of a classical Bose gas becomes “more fermionic” in the sense that fewer particles can occupy that state, but the microscopic transition probabilities are otherwise unchanged. (The underlying motivation was to apply this to get a handle on the effects of some complex configurations for an older mathematically-oriented network monitoring framework in a tractable traffic regime.) It turns out that this is completely uninteresting unless the remaining “bosonic” states are close to equally likely to occur: in this event you get a slight but also quite counter-intuitive effect on the state distributions. Pseudocolor figures of the energy functions corresponding to these distributions will illustrate:

3 totally bosonic interal states

3 totally bosonic interal states. The distribution is multigeometric, and the energy is affine.

Capping the occupancy of one internal state

Capping the occupancy of one internal state

A more restrictive cap

A more restrictive cap

A "fermionic" internal state

A "fermionic" internal state

Note that the color scales differ for each figure. I think the pictures indicate how surprising the behavior of this (completely classical) “poor man’s exclusion principle” is: the effect is as unexpected as it is slight.

Hopefully posting the main result will save some other folks a head-scratch or two. I’d be interested to know if it’s applied elsewhere.


Random bits

2 November 2009

“[The Department of State] spent $133 million over the past six years on certification and accreditation (C&A) reports”

“If you [continually] scare your customers or your management into taking your product or your security agenda seriously, they are almost guaranteed to stop listening to you at some point.”

Doug Natelson on universality of disordered materials and toy models

This Week in Mathematical Physics is covering algebras and operads from the ground up


Random bits

28 October 2009

“By the time the research project [spearheaded by the University of Illinois 'to make certain that the smart meters and other devices implemented by power companies can resist hackers and other attackers'] is completed, most of the nation will have already adopted untested and unsecured technologies.” (extra super bonus: Richard Clarke identifies Brazil as a location where hackers took down a power grid recently)

“a single [31 GeV] photon produced by the gamma-ray burst GRB 090510″ indicates that “if there is a quantum influence on the speed of light, it can only operate at distances of less than 1.2 times the Planck length”

Relating the asymptotic bound simultaneously optimizing transmission rate and relative minimum distance of error-correcting codes to phase diagrams of quantum statistical mechanical systems


A graded lexicographic index, part 3

26 October 2009

In an earlier post we covered some more sophisticated ways of implementing a graded lexicographic index, a/k/a the state index. One of these dealt with producing a superset of the state indices of the neighbors of a given state (here X_{B,b} is abusively interpreted as a graph embedded in \mathbb{R}^{B+1} where edges join any two vertices that are a distance \sqrt{2} apart). A careful exploitation of this technique leads to a very efficient way of updating the state index for random walks on X_{B,b}. Basically, the idea is to just determine which element of the neighborhood is the correct one, and restrict the neighborhood algorithm to a single vertex. In practice this means a lot of careful indexing and addition of appropriate binomial coefficients, but the basic idea is really just to omit the extra work in the neighborhood algorithm.

Here’s some MATLAB code that does the trick:

function y = stateindexupdate(a);

% a is an array of states (assumed to be) in a bucketspace
% this illustrates efficient stateindex updating

T = size(a,1);
B = size(a,2);
b = sum(a(1,:));

% Pascal matrix
mb = max(b+1,B+1);
P1 = pascal(mb);
P = P1(1:B-1,:);
PA = [zeros(1,mb);P];

ia(1) = stateindex(a(1,:));

for j = 2:T
 % compute tau:
 tau = 1 + cumsum(fliplr(a(j,:)));
 % compute the ball move that got us here:
 move = a(j,:) - a(j-1,:);
 n = 0; % init the "slot" index
 % compute the corresponding slot index...
 % first, ID the first nonzero element of the move a la Java/C:
 for k = 1:B
  if move(k)
   % now find the second nonzero element m:
   for i = k+1:B
    if move(i)
     m = i;
     break;
    end
   end
   % the slot n is as follows:
   n = move(k) * ( (B-k)*((B-k)+1)/2 - B + m);   % move(i) = sign(move(i))
   break;
  end
 end

 % compute the first (signed) triangular number further from zero:
 for k = 0:B
  n2 = k * (k+1)/2;
  if n2 >= abs(n)
   n2 = sign(n) * n2;
   break;
  end
 end

 d = abs(n2-n);
 l = ( -1 + sqrt(1+8*abs(n2)) )/2;  % l*(l+1)/2 = abs(n2);

 % the sum will go from d+2 to l+1...
 ind = (d+2):(l+1);
 % finally, perform the sum
 lind = length(ind);
 temp = zeros(1,lind);
 if n2 < 0
  for i = 1:lind
   temp(i) = PA(ind(i),tau(ind(i)-1)-1);
  end
  delta = -sum(temp);
 elseif n2 > 0
  for i = 1:lind
   temp(i) = PA(ind(i),tau(ind(i)-1));
  end
  delta = sum(temp);
 else
  delta = 0;
 end

 % finally, update the stateindex:
 ia(j) = ia(j-1) + delta;
end

y=ia;

This runs really fast. A Java version of this code that we produced performs many millions of updates per second on a decent processor core.

One of the nicest things about “grlex” ordering and indexing is that it can be used as a primitive to tackle lots of otherwise difficult problems. For instance, I’ve used it to help (among other things) emulate an older network security monitoring platform, analyze generalized multivariate de Moivre martingales, perform multivariate Lagrange interpolation, study energy functions on equivalence classes of directed graphs, and to construct propagation operators implementing periodic hexagonal, truncated-octahedral, and more generally permutohedral boundary conditions for lattice models (such as spin or lattice Boltzmann models) with surprising ease.

Hopefully some of these more advanced techniques for grlex stuff will be of use elsewhere. If you’d like to use some of this code, just ask: we’ll provide it under an open-source license.


Random bits

23 October 2009

“The Chinese government is ratcheting up its cyberspying operations against the U.S., a congressional advisory panel found”

Helping Wall Street cheat

Using AdS/CFT to explain why some metals have resistance that scales linearly with temperature


Random bits

21 October 2009

A crowdsourced proof of the density Hales-Jewett theorem hits the arxiv

“A pessimist might say that combining string theory and loop quantum gravity is like combining epicycles and aether”


A graded lexicographic index, part 2

19 October 2009

The previous post on this topic covered basic code for producing a graded lexicographic index (a/k/a the state index) and its inverse. This post will cover some more advanced techniques. The first one of these is simple: as de Boor pointed out (and it took me quite a while to catch on) the state index I(\alpha) of \alpha \in X_{B,b} can be given by a simple formula. Let PA denote the matrix given by the following MATLAB code:

mb=max(b+1,B+1);  P1=pascal(mb);  P=P1(1:B-1,:);  PA=[zeros(1,mb);P];

For example, with B = 4, b = 8 this is

PA = \left( \begin{smallmatrix} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 \\ 1 & 3 & 6 & 10 & 15 & 21 & 28 & 36 & 45 \end{smallmatrix} \right).

(Identifying the elements of the Pascal matrix with binomial coefficients would introduce some minor technical complications w/r/t the top row, and so we avoid this.) Now let

\alpha \equiv \sum_{k=1}^K \alpha_{j_k} \delta_{j_k}, \ \ \alpha_{j_k} > 0 \ \ \forall k.

Then

I(\alpha) = 1 + \sum_{k=1}^K \sum_{m=1}^{\alpha_{j_k}} PA \left( B + 1 - j_k, y_{km}(\alpha)\right)

where

y_{km}(\alpha) = b + 2 - m - \sum_{\ell = 0}^{k-1} \alpha_{j_\ell}

and an empty sum is evaluated as zero. This is embodied in the following MATLAB code:

J = find(alpha);
K = length(J);

for k = 1:K
   temp1 = sum(alpha(J(1:(k-1))));
   for m = 1:alpha(J(k))
      stateindex = stateindex + PA(B+1-J(k),b+2-temp1-m);
   end
end

stateindex = stateindex + 1;

This is hard to beat in general. However, for some applications it is sufficient to compute the state index at a neighboring site. And this can actually be done much more effectively in practice.

A deceptively simple algorithm always produces a superset of the neighborhood of a state in bucketspace. Write

\tau_k (\alpha) := 1+ \sum_{j=1}^k \alpha_{B-j+1}

and for a finite subset Y of \mathbb{R}, let \mu_k(Y) and \mu^k(Y) respectively denote the k least and greatest elements of Y. Now set I_1 := \{I(\alpha)\} and given I_k for k > 0 form

I_{k+1} := I_{k+1}^- \cup \{I_k\} \cup I_{k+1}^+

where

I_{k+1}^- := \{ \left( \mu_{k-1}(I_k) \cup I_1 \right) - PA(k, \tau_k) \}

and

I_{k+1}^+ := \{ \left( \mu^{k-1}(I_k) \cup I_1 \right) - PA(k, \tau_k - 1) \}.

Then the state indices of the nearest neighbors of \alpha are contained in the set I_B. It is not hard to see that this algorithm requires \sim 3B^2 simple operations (e.g., lookup, addition/subtraction, min/max) if we store the (fairly small) Pascal matrix. On “interior” states (i.e., states with no zero entries), this algorithm produces the exact neighborhood. However, on “boundary” states (i.e., states with at least one zero entry), some spurious indices are returned. It is not obvious (especially given its structure) how the algorithm might be refined to deal with this problem directly.

However, the spurious indices can themselves be checked efficiently. Given B and b, the following MATLAB code illustrates how to invert the state index:

PA2 = flipud(fliplr(PA));
alpha = zeros(1,B);
ind0 = 0; % init
for j = 1:B
   temp1 = cumsum(PA2(j,(ind0+1):(mb-1)));
   ind1 = max(find(temp1<s));
   if ind1
      temp2 = temp1(ind1);
      s = s - temp2;
      alpha(j) = ind1;
      ind0 = ind0 + ind1;
   end
end

The inversion essentially takes O(b) operations. With this tool in hand, we can trivially establish which state indices are spurious and which are not.

In the next post on this topic, we’ll produce a technique which is actually much faster than any of these for updating state indices in both MATLAB and Java, even after accounting for considerable care in the optimization (such as accounting for the order in which bitshifts are executed).


Follow

Get every new post delivered to your Inbox.