解释“信息论”的实用方法
信息论在编码和信息处理中发挥作用。解码存在。例如：压缩（多媒体）、密码学。
在信息论中，我们遇到“熵”、“自信息”、“互信息”等术语，整个主题都基于这些术语。这听起来只不过是抽象的。坦率地说，它们确实没有任何意义。
有没有任何书籍/材料/解释（如果可以的话）可以以实用的方式解释这些事情？
编辑：
信息论简介：符号、信号和信息论约翰·罗宾逊·皮尔斯（John Robinson Pierce）的《噪音》是这本书，它以我想要的方式（实际上）解释了它。实在是太好了。我开始阅读它。
Information theory comes into play where ever encoding & decoding is present. For example: compression(multimedia), cryptography.
In Information Theory we encounter terms like "Entropy", "Self Information", "Mutual Information" and entire subject is based on these terms. Which just sound nothing more than abstract. Frankly, they don't really make any sense.
Is there any book/material/explanation (if you can) which explains these things in a practical way?
EDIT:
An Introduction to Information Theory: symbols, signals & noise by John Robinson Pierce is The Book that explains it the way I want (practically). Its too good. I started reading it.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（8）
Shanon 的原创论文“通信的数学理论”是研究这一理论的一个非常非常重要的资源。任何人都不应该错过它。
通过阅读它，您将了解香农如何得出该理论，该理论应该可以消除大部分疑虑。
研究霍夫曼压缩算法的工作原理也会非常有帮助。
编辑：
根据亚马逊的评论，《
似乎不错（我还没有尝试过）。 [通过谷歌搜索“信息论外行”]
Shanon's original paper "A mathematical theory of communication" is one very very important resource for studying this theory. Nobody NOBODY should miss it.
By reading it you will understand how Shanon arrived at the theory which should clear most of the doubts.
Also studying workings of Huffman compression algorithm will be very helpful.
EDIT:
seems good according to the amazon reviews (I haven't tried it).
[by Googleing "information theory layman" ]
我自己对“信息论”的看法是，它本质上只是应用数学/统计学，但因为它应用于通信/信号，所以被称为“信息论”。
开始理解这些概念的最好方法是给自己设定一个真正的任务。例如，将您最喜欢的博客的几页保存为文本文件，然后尝试减小文件的大小，同时确保您仍然可以完全重建文件（即无损压缩）。例如，您将开始用 1 替换 和 的所有实例......
我始终认为边做边学将是最好的方法
My own view on "Information Theory" is that it's essentially just applied math / statistics but because it's being applied to communications / signals it's been called "Information Theory".
The best way to start understanding the concepts is to set yourself a real task. Say for example take a few pages of your favourite blog save it as a text file and then attempt to reduce the size of the file whilst ensuring you can still reconstruct the file completely (I.e. lossless compression). You'll start for example replacing all the instances of and with a 1 for example....
I'm always of the opinion learning by doing will be the best approach
我本来打算推荐费曼用于流行科学目的，但经过反思，我认为对于轻松进行严肃的研究来说，它也可能是一个不错的选择。如果不了解数学，你就无法真正了解这些东西，但费曼是如此令人回味，以至于他在不吓到马的情况下偷偷地引入了数学。
费曼计算讲座 http://ecx.images-amazon.com/images/I/ 51BKJV58A9L._SL500_AA240_.jpg
涵盖的内容不仅仅是信息论，而是好东西，读起来很愉快。 （此外，我有义务为团队物理而努力。啊！啊！Rhee！）
I was going to recommend Feynman for pop-sci purposes, but on reflection I think it might be a good choice for easing into a serious study as well. You can't really know this stuff without getting the math, but Feynman is so evocative that he sneaks the math into without scaring the horses.
Feynman Lectures on Computation http://ecx.images-amazon.com/images/I/51BKJV58A9L._SL500_AA240_.jpg
Covers rather more ground than just information theory, but good stuff and pleasant to read. (Besides, I am obligated to pull for Team Physics. Rah! Rah! Rhee!)
我记得《个人计算机世界》中的文章提出了用于识别硬币的 ID3 版本，尽管它使用了对数公式的启发式替代方案。我认为它最小化了平方和而不是最大化了熵——但那是很久以前的事了。 （我认为）Byte 中还有另一篇文章使用对数公式来获取类似事物的信息（而不是熵）。诸如此类的事情给了我一个让我更容易理解这个理论的把柄。
编辑 - “不是熵”我的意思是我认为它使用了信息值的加权平均值，但没有使用“熵”这个名称。
我认为从决策表构建简单的决策树是理解概率和信息之间关系的一个很好的方法。它使从概率到信息的联系更加直观，并且提供了加权平均值的示例来说明平衡概率的熵最大化效应。非常好的一天一课。
同样好的一点是，您可以用霍夫曼解码树（毕竟，这是一个“我正在解码哪个标记？”决策树）替换该决策树，并将该链接与编码联系起来。
顺便说一句 - 看看这个链接...
Mackay 有一本免费下载的教科书（并且有印刷版本），虽然我还没有读完全部内容，但我读过的部分似乎非常好。尤其是第 293 页开始的贝叶斯中对“解释消失”的解释。
CiteSeerX 是信息论论文（除其他外）非常有用的资源。两篇有趣的论文是...
CN2 可能不是第一天的材料。
I remember articles in, I think, Personal Computer World that presented a version of ID3 for identifying coins, though it used a heuristic alternative to the log formula. I think it minimised sums of squares rather than maximising entropy - but it was a long time ago. There was another article in (I think) Byte that used the log formula for information (not entropy) for similar things. Things like that gave me a handle that made the theory easier to cope with.
EDIT - by "not entropy" I mean I think it used weighted averages of information values, but didn't use the name "entropy".
I think construction of simple decision trees from decision tables is a very good way to understand the relationship between probability and information. It makes the link from probability to information more intuitive, and it provides examples of the weighted average to illustrate the entropy-maximizing effect of balanced probabilities. A very good day-one kind of lesson.
And what's also nice is you can then replace that decision tree with a Huffman decoding tree (which is, after all, a "which token am I decoding?" decision tree) and make that link to coding.
BTW - take a look at this link...
Mackay has a free downloadable textbook (and available in print), and while I haven't read it all, the parts I have read seemed very good. The explanation of "explaining away" in Bayes, starting page 293, in particular, sticks in mind.
CiteSeerX is a very useful resource for information theory papers (among other things).Two interesting papers are...
Though CN2 probably isn't day one material.
尽管这些概念可能很抽象，但它们最近在机器学习/人工智能中得到了很好的用途。
这可能是对这些理论概念的实际需要的良好动机。总之，您想要估计您的模型（例如 LSTM、CNN）在逼近目标输出方面的表现（例如使用信息论中的交叉熵或 Kullback-Leibler 散度）。 （检查信息瓶颈和深度学习与信息瓶颈原理通过信息论解释深度学习的观点）
此外，您不会建立有用的通信或网络系统，无需对通道容量和属性进行一些分析。
从本质上讲，它可能看起来很理论，但它是当今通信时代的核心。
为了更详细地理解我的意思，我邀请您观看 ISIT 讲座：信息论精神，作者：David TSe 教授。
另请参阅 Claude Channon 本人的论文Bandwagon，他解释了信息论何时可能有用以及不适合使用时。
这篇论文可帮助您入门，有关全面的详细信息，请阅读信息论要素。
Though, the concepts may be abstract, they find good use in recent times in Machine learning/Artificial intelligence.
This might serve as a good motivation on practical need for these theoretic concepts. In summary, you want to estimate how well your model (LSTM, CNN for example) does in approximating the target output ( using for example cross entropy or Kullback-Leibler Divergence from information theory). (check on information bottleneck and deep learning and information Bottleneck principle for perspectives on explaining deep learning through information theory)
In addition, you won't build a useful communication or networked system without some analysis of the channel capacity and properties.
In essence, it might look theoretic but it is at the heart of the present communication age.
To get a more elaborate view on what I mean, I invite you to watch this ISIT lecture: The Spirit of Information Theory by Prof David TSe.
Check also the paper Bandwagon by Claude Channon himself explaining when information theory might be useful and when it is not appropriate for use.
This paper helps you get you started and for comprehensive details read Elements of Information theory.
信息论在机器学习和数据挖掘等领域具有非常有效的应用。特别是数据可视化、变量选择、数据转换和预测、信息论标准是最流行的方法之一。
参见例如
http://citeseerx .ist.psu.edu/viewdoc/download?doi=10.1.1.87.825&rep=rep1&type=pdf
或者
http://www.mdpi.com/1424-8220/11/6/5695信息论
使我们能够以正式的方式实现最佳数据压缩，例如根据后验分布和马尔可夫毯子：
http://www.mdpi.com/1099-4300/13/7/1403
它允许我们检索变量选择中错误概率的上限和下限：
< a href="http://www.mdpi.com/1099-4300/12/10/2144" rel="nofollow">http://www.mdpi.com/1099-4300/12/10/2144< /a>
与统计学相比，使用信息论的优点之一是不一定需要建立概率分布。人们可以计算信息、冗余、熵、传递熵，而根本不需要尝试估计概率分布。没有信息丢失的变量消除是根据条件后验概率的保留来定义的，使用信息论可以找到类似的公式......而不需要计算概率密度。计算是根据变量之间的互信息进行的，文献为这些提供了许多有效的估计器和低维近似值。看：
http://citeseerx.ist .psu.edu/viewdoc/download?doi=10.1.1.87.825&rep=rep1&type=pdf
http://www.mdpi.com/1424-8220/11/6/5695
Information theory has very efficient applications in e.g. machine learning and data mining. in particular data visualization, variable selection, data transformation and projections, information theoretic criteria are among the most popular approaches.
See e.g.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.87.825&rep=rep1&type=pdf
or
http://www.mdpi.com/1424-8220/11/6/5695
Information theory allows us to approach optimal data compaction in a formal way e.g. in terms of posterior distributions and Markov Blankets:
http://www.mdpi.com/1099-4300/13/7/1403
It allows us to retrieve upper and lower bounds on the probability of error in variable selection:
http://www.mdpi.com/1099-4300/12/10/2144
One of the advantages of using information theory compared to statistics is that one doesn't necessarily need to set up probability distributions. One can compute information, redundancy, entropy, transfer entropy without trying to estimate probability distributions at all. Variable elimination without information loss is defined in terms of preservation of conditional posterior probabilities, using information theory one can find similar formulations...without the need to compute probability densities. Caculations are rather in terms of mutual information between variables and the litterature has provided a lot of efficient estimators and lower dimensional approximations for these. See:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.87.825&rep=rep1&type=pdf
http://www.mdpi.com/1424-8220/11/6/5695
我可以推荐Glynn Winskel的这本书。它在我的大学用于信息论课程。
它从逻辑理论出发，定义了一种简单的命令式语言，称为IMP，并遵循许多关于语言形式语义的概念。
编程语言的形式语义
http://mitpress.mit.edu/books/formal -语义编程语言
I could suggest this book by Glynn Winskel. It was used in my university for the Information Theory course.
It starts from Logic Theory then defines a simple imperative language, called IMP, and it follows with many concepts about formal semantics in language.
The Formal Semantics of Programming Languages
http://mitpress.mit.edu/books/formal-semantics-programming-languages
信息论是数学和电气工程的一个分支，研究信息的传输、处理和存储。它最初由 Claude Shannon 在 1948 年提出，用于处理电话线中的噪声。
信息论的核心是量化信号中包含的信息量。这可以通过多种方式来完成，但一种常见的度量是熵。熵是对信号内容的不确定性程度的度量。熵越高，信号的可预测性就越差。
信息论很重要，因为它使我们能够量化和测量信息。这很重要，因为它使我们能够更好地理解和优化通信系统。此外，信息论可用于测量可压缩到给定空间的数据量。
Information theory is a branch of mathematics and electrical engineering that deals with the transmission, processing, and storage of information. It was originally proposed by Claude Shannon in 1948 to deal with the noise in telephone lines.
At its core, information theory is all about quantifying the amount of information contained in a signal. This can be done in a variety of ways, but one common measure is entropy. Entropy is a measure of how much uncertainty there is about the contents of a signal. The higher the entropy, the less predictable the signal is.
Information theory is important because it allows us to quantify and measure information. This is significant because it allows us to better understand and optimize communication systems. Additionally, information theory can be used to measure the amount of data that can be compressed into a given space.