[AIRG] Higher-arity mutual information

[Lucas Morton <lamorton@xxxxxxxx>]

Hi all,

Here are the references I've found on extending measures of redundancy/information beyond two variables using the concept of the maximum-entropy distribution subject to N-variable marginal distributions (N>1). Interestingly, this wheel has been invented several times. [1]Â

There are two kinds of papers: those (by Amari and by Schniedman et al) that constrained all the marginals up to order N, and those that constrained proper subsets of the marginals up to a given order. Of the latter group, Bertschinger et al and Griffith & Koch approached the issue by considering two-variable marginals, with one variable always being the designated 'target' variable. Cavallo & Pittarelli, while presenting a more general framework, did not make connections outside the realm of databases.

• Cavallo, R.; Pittarelli, M. "The Theory of Probabilistic Databases" Proceedings of the 13th VLDB Conference, Brighton (1987)
• www.vldb.org/conf/1987/P071.PDF
• The probabilistic database join is equivalent to finding the maximum-entropy distribution subject to a set of marginal distributions.
• Shun-ichi Amari, "Information Geometry on Hierarchy of Probability
  Distributions," IEEE Trans. Info. Theory 47, 1701 (2001)
• https://people.csail.mit.edu/jrennie/trg/papers/amari-ig-hierarchy-01.pdf
• From the founder of information geometry (very heavy notation/jargon)
• Includes all marginals up to a given order in constraint set.
• Schneidman et al, "Network Information and Connected Correlations," Phys. Rev. Let. 91, 238701 (2003).
• Âhttps://www.princeton.edu/~wbialek/our_papers/schneidman+al_03b.pdf
• Focuses on measures that incorporate all marginal distributions up to a given order.
• Griffith, V.; Koch, C. Quantifying synergistic mutual information. arXiv:1205.4265 (2013).
• https://arxiv.org/pdf/1205.4265v6
• Similar to Betschinger, motivated by cryptographic key distribution
• Considers partial set of marginals of order 2
• Bertschinger et al,"Quantifying unique information," Entropy 16, 2161 (2014).
• https://doi.org/10.3390/e16042161
• Focuses on game-theoretic motivation / intuition
  
• Considers use of partial set of marginal distribution constraints of order 2
  

Many of the references within are worth reading for background. The Beer-Williams "Partial Information Decomposition" and McGill's (non-positive) "interaction information" are other attempts to construct a multivariate generalization of mutual information.

[1] (...whereas Newton could say, "If I have seen a little farther than others, it is because I have stood on the shoulders of giants," I am forced to say, "Today we stand on each other's feet." - Richard Hamming)

---------------------------------------------------

Lucas A. Morton

Research Associate

Department of Engineering Physics

University of Wisconsin - Madison

https://fusionlab.ep.wisc.edu/