将 HTML 实体转换为字符的 Bash 脚本
我正在寻找一种方法将其转变
hello < world
为:
hello < world
我可以使用 sed,但是如何在不使用神秘的正则表达式的情况下完成此操作?
I'm looking for a way to turn this:
hello < world
to this:
hello < world
I could use sed, but how can this be accomplished without using cryptic regex?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(14)
尝试重新编码 (存档页面; GitHub 镜像; rel="nofollow noreferrer">Debian 页面):
在 Linux 和类似的 Unix-y 系统上安装:
在 Mac OS 上安装,使用:
Try recode (archived page; GitHub mirror; Debian page):
Install on Linux and similar Unix-y systems:
Install on Mac OS using:
使用 perl:
使用命令行中的 php:
With perl:
With php from the command line:
另一种方法是通过网络浏览器进行管道传输,例如 w3m:
echo '&# 33;' | w3m -dump -T text/html
这对我在 Cygwin 中非常有用,因为在 Cygwin 中下载和安装发行版很困难。
这个答案是在此评论中找到的。
An alternative is to pipe through a web browser like w3m:
echo '!' | w3m -dump -T text/html
This worked great for me in Cygwin, where downloading and installing distributions are difficult.
This answer was found in this comment.
使用 xmlstarlet:
Using xmlstarlet:
python 3.2+版本:
A python 3.2+ version:
这个答案基于:在 Bash 中转义 HTML 的快捷方式?它非常适合在 Stack Exchange 上获取答案(使用
wget
)并将 HTML 转换为常规 ASCII 字符:编辑 1: 2017 年 4 月 7 日 - 添加左双引号和右双引号引用 转换。这是 bash 脚本的一部分,它通过网络抓取 SE 答案并将其与本地代码文件进行比较:询问 Ubuntu -
本地文件和 Ask Ubuntu 答案之间的代码版本控制
编辑 2017 年 6 月 26 日
使用
sed
将来自 Ask Ubuntu / Stack Exchange 的 1K 行文件上的 HTML 转换为 ASCII 需要大约 3 秒的时间。因此,我被迫使用 Bash 内置搜索和替换,响应时间约为 1 秒。这是函数:
This answer is based on: Short way to escape HTML in Bash? which works fine for grabbing answers (using
wget
) on Stack Exchange and converting HTML to regular ASCII characters:Edit 1: April 7, 2017 - Added left double quote and right double quote conversion. This is part of bash script that web-scrapes SE answers and compares them to local code files here: Ask Ubuntu -
Code Version Control between local files and Ask Ubuntu answers
Edit June 26, 2017
Using
sed
was taking ~3 seconds to convert HTML to ASCII on a 1K line file from Ask Ubuntu / Stack Exchange. As such I was forced to use Bash built-in search and replace for ~1 second response time.Here's the function:
在 macOS 上,您可以使用内置命令
textutil
(一般来说这是一个方便的实用程序):输出:
On macOS, you can use the built-in command
textutil
(which is a handy utility in general):outputs:
我喜欢 https://stackoverflow.com/a/13161719/1506477 中给出的 Perl 答案。
但是,它在纯文本文件上生成的行数不相等。 (而且我对 perl 的了解不足以调试它。)
我喜欢 https://stackoverflow.com/a/42672936 中给出的 python 答案/1506477 -
但它会在内存中创建一个列表
[ ... for l in sys.stdin]
,这对于大文件是禁止的。这是另一种无需在内存中缓冲的简单 Python 方法:使用
awkg
。awkg
是一个基于 Python 的类似 awk 的行处理器。您可以使用 pip https://pypi.org/project/awkg/ 安装它:-b
是 awk 的BEGIN{}
块,在开头运行一次。这里我们刚刚做了
from html import unescape
。每行记录都在
R0
变量中,我们为此做了print(unescape(R0))
免责声明:
我是
awkg
的维护者I like the Perl answer given in https://stackoverflow.com/a/13161719/1506477.
But, it produced an unequal number of lines on plain text files. (and I dont know perl enough to debug it.)
I like the python answer given in https://stackoverflow.com/a/42672936/1506477 --
but it creates a list
[ ... for l in sys.stdin]
in memory, that is forbidden for large files.Here is another easy pythonic way without buffering in memory: using
awkg
.awkg
is a python based awk-like line processor. You may install it using pip https://pypi.org/project/awkg/:-b
is awk'sBEGIN{}
block that runs once in the beginning.Here we just did
from html import unescape
.Each line record is in
R0
variable, for which we didprint(unescape(R0))
Disclaimer:
I am the maintainer of
awkg
仅通过 sed 替换来支持所有 HTML 实体的转义将需要太长的命令列表而不实用,因为每个 Unicode 代码点至少有两个相应的 HTML 实体。
但仅使用 sed、grep、Bourne shell 和基本 UNIX 实用程序(GNU coreutils 或等效工具)即可完成此操作:
但是请注意,支持
\uHHHH
和\UHHHHHHHH 的 printf 实现
序列是必需的,例如 GNU 实用程序的序列。要进行测试,请检查printf "\u00A7\n"
是否打印§
。要调用该实用程序而不是内置 shell,请将出现的printf
替换为env printf
。该脚本使用一个附加文件
named_entities.sed
来支持命名实体。它可以使用以下 HTML 页面根据规范生成:只需在现代浏览器中打开它,并将生成的页面另存为
named_entities.sed
文本。如果只需要命名实体,这个 sed 脚本也可以单独使用;这种情况下给它可执行权限就可以方便的直接调用了。现在,上面的 shell 脚本可以用作
./html_unescape.sh foo.html
,或者在从标准输入读取的管道中使用。例如,如果由于某种原因需要分块处理数据(如果
printf
不是内置 shell 并且要处理的数据很大,可能会出现这种情况),可以使用它是:脚本的解释如下。
需要支持三种类型的转义序列:
&#D;
其中D
是转义字符的 Unicode 代码点的十进制值;< /p>&#xH;
其中H
是转义字符的 Unicode 代码点的十六进制值;&N;
其中N
是转义字符的命名实体之一的名称。生成的
named_entities.sed
脚本支持&N;
转义,该脚本仅执行替换列表。此支持代码点转义的方法的核心部分是
printf
实用程序,它能够:以十六进制格式打印数字,以及
从代码点的十六进制值打印字符(使用转义符
\uHHHH
或\UHHHHHHHH
)。第一个功能在 sed 和 grep 的帮助下,用于将
&#D;
转义减少为&#xH;
转义。 shell 函数htmlEscDec2Hex
可以做到这一点。函数
htmlHexUnescape
使用 sed 将&#xH;
转义符转换为 printf 的\u
/\U
转义符,然后使用第二个功能来打印未转义的字符。To support the unescaping of all HTML entities only with sed substitutions would require too long a list of commands to be practical, because every Unicode code point has at least two corresponding HTML entities.
But it can be done using only sed, grep, the Bourne shell and basic UNIX utilities (the GNU coreutils or equivalent):
Note, however, that a printf implementation supporting
\uHHHH
and\UHHHHHHHH
sequences is required, such as the GNU utility’s. To test, check for example thatprintf "\u00A7\n"
prints§
. To call the utility instead of the shell built-in, replace the occurrences ofprintf
withenv printf
.This script uses an additional file,
named_entities.sed
, in order to support the named entities. It can be generated from the specification using the following HTML page:Simply open it in a modern browser, and save the resulting page as text as
named_entities.sed
. This sed script can also be used alone if only named entities are required; in this case it is convenient to give it executable permission so that it can be called directly.Now the above shell script can be used as
./html_unescape.sh foo.html
, or inside a pipeline reading from standard input.For example, if for some reason it is needed to process the data by chunks (it might be the case if
printf
is not a shell built-in and the data to process is large), one could use it as:Explanation of the script follows.
There are three types of escape sequences that need to be supported:
&#D;
whereD
is the decimal value of the escaped character’s Unicode code point;&#xH;
whereH
is the hexadecimal value of the escaped character’s Unicode code point;&N;
whereN
is the name of one of the named entities for the escaped character.The
&N;
escapes are supported by the generatednamed_entities.sed
script which simply performs the list of substitutions.The central piece of this method for supporting the code point escapes is the
printf
utility, which is able to:print numbers in hexadecimal format, and
print characters from their code point’s hexadecimal value (using the escapes
\uHHHH
or\UHHHHHHHH
).The first feature, with some help from sed and grep, is used to reduce the
&#D;
escapes into&#xH;
escapes. The shell functionhtmlEscDec2Hex
does that.The function
htmlHexUnescape
uses sed to transform the&#xH;
escapes into printf’s\u
/\U
escapes, then uses the second feature to print the unescaped characters.我创建了一个基于 实体列表所以它必须处理大多数实体。
I have created a
sed
script based on the list of entities so it must handle most of the entities.我原来的答案得到了一些评论,即
recode
不适用于 UTF-8 编码的 HTML 文件。这是正确的。recode
仅支持 HTML 4。编码HTML
是HTML_4.0
的别名:HTML 4 的默认编码是 Latin-1。这在 HTML 5 中发生了变化。HTML 5 的默认编码是 UTF-8。这就是
recode
不适用于 HTML 5 文件的原因。HTML 5 在此处定义实体列表:
该定义包括 JSON 格式的机器可读规范:
JSON 文件可用于执行简单的文本替换。以下示例是一个自修改 Perl 脚本,它将 JSON 规范缓存在其 DATA 块中。
注意:出于一些模糊的兼容性原因,规范允许实体没有终止分号。因此,实体按长度以相反的顺序排序,以确保首先替换正确的实体,这样它们就不会被没有结束分号的实体破坏。
用法示例:
My original answer got some comments, that
recode
does not work for UTF-8 encoded HTML files. This is correct.recode
supports only HTML 4. The encodingHTML
is an alias forHTML_4.0
:The default encoding for HTML 4 is Latin-1. This has changed in HTML 5. The default encoding for HTML 5 is UTF-8. This is the reason, why
recode
does not work for HTML 5 files.HTML 5 defines the list of entities here:
The definition includes a machine readable specification in JSON format:
The JSON file can be used to perform a simple text replacement. The following example is a self modifying Perl script, which caches the JSON specification in its DATA chunk.
Note: For some obscure compatibility reasons, the specification allows entities without a terminating semicolon. Because of that the entities are sorted by length in reverse order to make sure, that the correct entities are replaced first so that they do not get destroyed by entities without the ending semicolon.
Example usage:
我还找到了这个解决方案:
Also I found this one solution:
这是纯 bash 中的一个:
享受 bash 的乐趣:-)
Here is one in pure bash:
Have some bash fun :-)
使用 Xidel:
With Xidel: