如何将 sys.stdout 复制到日志文件?
编辑:因为看起来要么没有解决方案,要么我正在做一些非常不标准的事情以至于没有人知道 - 我将修改我的问题以询问:当 python 应用程序正在制作时完成日志记录的最佳方法是什么很多系统调用?
我的应用程序有两种模式。 在交互模式下,我希望所有输出都转到屏幕以及日志文件,包括任何系统调用的输出。 在守护进程模式下,所有输出都会写入日志。 守护进程模式使用 os.dup2() 效果很好。 我找不到一种方法可以在交互模式下将所有输出“tee”到日志,而无需修改每个系统调用。
换句话说,我想要命令行“tee”的功能用于 python 应用程序生成的任何输出,包括系统调用输出。
澄清一下:
为了重定向所有输出,我做了类似的事情,并且效果很好:
# open our log file
so = se = open("%s.log" % self.name, 'w', 0)
# re-open stdout without buffering
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
# redirect stdout and stderr to the log file opened above
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())
这样做的好处是它不需要来自代码其余部分的特殊打印调用。 该代码还运行一些 shell 命令,因此不必单独处理每个输出也很好。
简而言之,我想做同样的事情,除了复制而不是重定向。
起初,我认为简单地反转 dup2 就可以了。 为什么不呢? 这是我的测试:
import os, sys
### my broken solution:
so = se = open("a.log", 'w', 0)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
os.dup2(sys.stdout.fileno(), so.fileno())
os.dup2(sys.stderr.fileno(), se.fileno())
###
print("foo bar")
os.spawnve("P_WAIT", "/bin/ls", ["/bin/ls"], {})
os.execve("/bin/ls", ["/bin/ls"], os.environ)
文件“a.log”应该与屏幕上显示的内容相同。
Edit: Since it appears that there's either no solution, or I'm doing something so non-standard that nobody knows - I'll revise my question to also ask: What is the best way to accomplish logging when a python app is making a lot of system calls?
My app has two modes. In interactive mode, I want all output to go to the screen as well as to a log file, including output from any system calls. In daemon mode, all output goes to the log. Daemon mode works great using os.dup2()
. I can't find a way to "tee" all output to a log in interactive mode, without modifying each and every system call.
In other words, I want the functionality of the command line 'tee' for any output generated by a python app, including system call output.
To clarify:
To redirect all output I do something like this, and it works great:
# open our log file
so = se = open("%s.log" % self.name, 'w', 0)
# re-open stdout without buffering
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
# redirect stdout and stderr to the log file opened above
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())
The nice thing about this is that it requires no special print calls from the rest of the code. The code also runs some shell commands, so it's nice not having to deal with each of their output individually as well.
Simply, I want to do the same, except duplicating instead of redirecting.
At first thought, I thought that simply reversing the dup2
's should work. Why doesn't it? Here's my test:
import os, sys
### my broken solution:
so = se = open("a.log", 'w', 0)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
os.dup2(sys.stdout.fileno(), so.fileno())
os.dup2(sys.stderr.fileno(), se.fileno())
###
print("foo bar")
os.spawnve("P_WAIT", "/bin/ls", ["/bin/ls"], {})
os.execve("/bin/ls", ["/bin/ls"], os.environ)
The file "a.log" should be identical to what was displayed on the screen.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(18)
我知道这个问题已经被反复回答,但为此我从 John T 的 答案中获取了主要答案并对其进行了修改,以便它包含建议的刷新并遵循其链接的修订版本。 我还添加了 cladmi's 答案中提到的进入和退出,以便与 with 语句一起使用。 此外,文档提到使用
刷新文件os.fsync()
所以我也添加了它。 我不知道您是否真的需要它,但它就在那里。然后您可以使用它
或
I know this question has been answered repeatedly, but for this I've taken the main answer from John T's answer and modified it so it contains the suggested flush and followed its linked revised version. I've also added the enter and exit as mentioned in cladmi's answer for use with the with statement. In addition, the documentation mentions to flush files using
os.fsync()
so I've added that as well. I don't know if you really need that but its there.You can then use it
or
正如其他地方所述,也许最好的解决方案是直接使用日志记录模块:
但是,在某些(罕见)情况下,您确实希望重定向 stdout。 当我扩展 django 的 runserver 命令时,我遇到了这种情况,该命令使用 print:我不想破解 django 源代码,但需要 print 语句来转到文件。
这是使用日志记录模块将 stdout 和 stderr 重定向到 shell 的一种方法:
如果确实无法直接使用日志记录模块,则应该仅使用此 LogFile 实现。
As described elsewhere, perhaps the best solution is to use the logging module directly:
However, there are some (rare) occasions where you really want to redirect stdout. I had this situation when I was extending django's runserver command which uses print: I didn't want to hack the django source but needed the print statements to go to a file.
This is a way of redirecting stdout and stderr away from the shell using the logging module:
You should only use this LogFile implementation if you really cannot use the logging module directly.
要完成 John T 的回答: https://stackoverflow.com/a/616686/395687
我添加了
__enter__< /code> 和
__exit__
方法将其用作带有with
关键字的上下文管理器,这使得此代码可以用作
To complete John T answer: https://stackoverflow.com/a/616686/395687
I added
__enter__
and__exit__
methods to use it as a context manager with thewith
keyword, which gives this codeIt can then be used as
我用 Python 编写了一个
tee()
实现,它应该适用于大多数情况,而且它也适用于 Windows。https://github.com/pycontribs/tendo
另外,您可以将其与
结合使用如果需要,可以使用 Python 的logging
模块。I wrote a
tee()
implementation in Python that should work for most cases, and it works on Windows also.https://github.com/pycontribs/tendo
Also, you can use it in combination with
logging
module from Python if you want.(啊,只需重新阅读您的问题,就会发现这并不完全适用。)
这是一个示例程序,它使用 Python 日志模块。 从 2.3 开始,所有版本都有这个日志模块。 在此示例中,日志记录可通过命令行选项进行配置。
在安静模式下,它只会记录到文件,在正常模式下,它会记录到文件和控制台。
(Ah, just re-read your question and see that this doesn't quite apply.)
Here is a sample program that makes uses the python logging module. This logging module has been in all versions since 2.3. In this sample the logging is configurable by command line options.
In quite mode it will only log to a file, in normal mode it will log to both a file and the console.
我已经使用 Jacob Gabrielson 公认的解决方案大约一年了,但现在不可避免的事情发生了,我的一位用户希望在 Windows 上使用此解决方案。 看看其他提出的答案,我认为大多数答案都未能捕获生成进程的输出(如原始海报中粗体所示); 我认为做到这一点的唯一方法是执行 os.dup2() 。 我想我已经弄清楚如何回答原始发帖人的准确问题,并且无需使用 Unix 特定工具
tee
:我现在可以捕获 Python 程序的所有输出,包括任何生成的 shell 命令。 这适用于 Windows、Mac 和 Linux。 代码如下:你像这样使用它(注意除了 Jacob Gabrielson 之外的其他解决方案无法捕获的困难情况):
这不是一个简短而甜蜜的答案,但我试图保持简洁,并且这是我能做到的最简单的事情。 由于以下原因,它很复杂:
由于我无法使用
tee
,我必须以某种方式从我的 Python 进程中执行tee
的任务。 我不清楚是否有一种可移植的方式来fork()
和os.pipe()
(这说明在 Windows 中很难与分叉进程共享文件描述符),因此我决定使用线程
。在 Windows 中,
sys.stdout
和sys.stderr
确实不喜欢它们的底层fileno()
通过 <代码>os.pipe()通过os.dup2()
。 Python 解释器在执行第一个print(...)
命令后立即崩溃。仅在 Windows 上,为了解决解释器崩溃问题,我将
sys.stdout.write = ...
设置为一个新函数,该函数只需调用os.write(. ..)
。 默认情况下,我仅在检测到 Windows 时执行此操作。 因为我使用了monkeypatch,所以我希望这能够到达对sys.stdout
的所有缓存引用。 我选择了这种猴子修补方法,而不是分配一个全新的流,例如 sys.stdout=...,因为我担心旧的 sys.stdout 的副本会保留缓存在解释器的各个部分,但我猜测 sys.stdout.write 不太可能被直接缓存。如果您守护进程处理管道输出的线程,那么主线程完成后该线程就会被终止,但这并不能保证所有输出都已写入日志文件。 实际上有必要不守护这些辅助线程,并让它们在管道关闭时优雅地终止自身。
实际上,我并不完全确定我是否正确理解了所有极端情况——与微妙的操作系统功能交互的线程代码很难编写。 尽管如此,到目前为止它还是通过了我的测试。 因为它有点麻烦,所以我制作了一个 PyPI 包:
Github 位于这里。
I've been using Jacob Gabrielson's accepted solution for about 1 year, but now the inevitable has happened and one of my users wants this on Windows. Looking at the other proposed answers, I think most of these answers fail at capturing the outputs of spawned processes (as bolded by the original poster); I think the only way to do this is to do
os.dup2()
. I think I've figured out how to answer the original poster's exact question, and without using the Unix-specific tooltee
: I can now capture all outputs of my Python program, including any spawned shell commands. This works on Windows, Mac and Linux. The code is as follows:You use it like this (notice the hard case that other solutions, apart from Jacob Gabrielson's, fail to capture):
This is not a short-and-sweet answer, but I tried to keep it succinct, and it's as simple as I could make it. It's complicated for the following reasons:
Since I cannot use
tee
, I have to somehow perform the task oftee
from within my Python process. It was not clear to me that there was a portable way offork()
ing and communicating with anos.pipe()
(this states it's hard to share filedescriptors with forked processes in Windows) so I decided to usethreading
.In Windows,
sys.stdout
andsys.stderr
really don't appreciate when their underlyingfileno()
get rerouted through anos.pipe()
viaos.dup2()
. The Python interpreter crashes immediately after the firstprint(...)
command.On Windows only, to solve the interpreter crashes, I monkeypatch
sys.stdout.write = ...
by setting it to a new function that simply calls toos.write(...)
. By default, I only do this when Windows is detected. Because I monkeypatch, I'm hoping this will reach all cached references tosys.stdout
. I chose this monkeypatching approach instead of allocating a brand new stream, e.g.sys.stdout=...
, because I was concerned that copies of the oldsys.stdout
would remain cached in various parts of the interpreter, but I guessed thatsys.stdout.write
was less likely to have been directly cached.If you daemonize the thread that processes the output of the pipe, then that thread gets killed as soon as the main thread completes, but this does not guarantee that all outputs have been written to the log file. It's actually necessary to not daemonize those helper threads and to let them gracefully terminate themselves when the pipes are closed.
I'm actually not entirely sure that I got all the corner cases right -- threaded code that interacts with delicate OS features is scary to write. Nevertheless, it passed my tests thus far. Because it's kind of hairy, I've made a PyPI package:
The Github is here.
使用日志记录模块的另一个解决方案:
another solution using logging module:
上面的答案似乎都没有真正回答所提出的问题。 我知道这是一个旧线程,但我认为这个问题比每个人所做的要简单得多:
现在这将向正常的 sys.stderr 处理程序和您的文件重复所有内容。 为
sys.stdout
创建另一个类tee_out
。None of the answers above really seems to answer the problem posed. I know this is an old thread, but I think this problem is a lot simpler than everyone is making it:
Now this will repeat everything to the normal sys.stderr handler and your file. Create another class
tee_out
forsys.stdout
.根据@user5359531在@John T的答案下的评论中的请求,这里是修订后的引用帖子的副本该答案中链接讨论的版本:
As per a request by @user5359531 in the comments under @John T's answer, here's a copy of the referenced post to the revised version of the linked discussion in that answer:
我正在编写一个脚本来运行命令行脚本。 (因为在某些情况下,Linux 命令没有可行的替代品——例如 rsync 的情况。)
我真正想要的是在每种可能的情况下都使用默认的 python 日志记录机制,但是当发生意外的错误时仍然捕获任何错误。
这段代码似乎可以解决问题。 它可能不是特别优雅或高效(尽管它不使用 string+=string,所以至少它没有那种特定的潜在瓶子-
脖子 )。 我将其发布,以防它给其他人任何有用的想法。
显然,如果您不像我一样受奇思妙想的影响,请将 LOG_IDENTIFIER 替换为您不希望看到有人写入日志的另一个字符串。
I'm writing a script to run cmd-line scripts. ( Because in some cases, there just is no viable substitute for a Linux command -- such as the case of rsync. )
What I really wanted was to use the default python logging mechanism in every case where it was possible to do so, but to still capture any error when something went wrong that was unanticipated.
This code seems to do the trick. It may not be particularly elegant or efficient ( although it doesn't use string+=string, so at least it doesn't have that particular potential bottle-
neck ). I'm posting it in case it gives someone else any useful ideas.
Obviously, if you're not as subject to whimsy as I am, replace LOG_IDENTIFIER with another string that you're not like to ever see someone write to a log.
您还可以根据上面的 shx2 的答案,使用
class multifile 添加
stderr
:You can also add
stderr
as well, based on shx2's answer above usingclass multifile
:这是一个上下文管理器,它暂时将标准输出复制到文件中。 在我看来这是一个改进,因为它重置了 sys.stdout 和 即使发生异常,也会关闭文件,并且语法表明后台发生了不可见的更改。 扩展了 John T 的解决方案。
用法示例:
Here's a context manager that temporarily duplicates stdout to a file. It's an improvement in my view because it resets
sys.stdout
& closes the file even when exceptions occur, and the syntax is indicative of an invisible change in the background. Expaneded on John T's solution.Example usage:
我编写了
sys.stderr
的完整替代品,只是复制了将stderr
重命名为stdout
的代码,以使其也可用于替换sys .stdout
。为此,我创建与当前
stderr
和stdout
相同的对象类型,并将所有方法转发到原始系统stderr
和stdout
:要使用它,您只需调用
StdErrReplament::lock(logger)
和StdOutReplament::lock(logger)
传递您想要用来发送输出文本的记录器。 例如:
运行此代码,您将在屏幕上看到:
文件内容:
如果您还想查看
的内容log.debug
在屏幕上调用,您将需要向记录器添加一个流处理程序。 在这种情况下,它会像这样:运行时会输出如下:
虽然它仍会将其保存到文件
my_log_file.txt
:< /a>
当使用
StdErrReplament:unlock()
禁用此功能时,它只会恢复stderr
流的标准行为,因为附加的记录器永远不会因为其他人而分离可以参考其旧版本。 这就是为什么它是一个永远不会消亡的全局单例。 因此,如果使用imp
或其他内容重新加载此模块,它将永远不会重新捕获当前的sys.stderr
,因为它已经被注入其中并在内部保存。I wrote a full replacement for
sys.stderr
and just duplicated the code renamingstderr
tostdout
to make it also available to replacesys.stdout
.To do this I create the same object type as the current
stderr
andstdout
, and forward all methods to the original systemstderr
andstdout
:To use this you can just call
StdErrReplament::lock(logger)
andStdOutReplament::lock(logger)
passing the logger you want to use to send the output text. For example:
Running this code, you will see on the screen:
And on the file contents:
If you would like to also see the contents of the
log.debug
calls on the screen, you will need to add a stream handler to your logger. On this case it would be like this:Which would output like this when running:
While it would still saving this to the file
my_log_file.txt
:When disabling this with
StdErrReplament:unlock()
, it will only restore the standard behavior of thestderr
stream, as the attached logger cannot be never detached because someone else can have a reference to its older version. This is why it is a global singleton which can never dies. Therefore, in case of reloading this module withimp
or something else, it will never recapture the currentsys.stderr
as it was already injected on it and have it saved internally.我之前遇到过同样的问题,发现这个片段非常有用:
来自: http://mail.python.org/pipermail/python-list/2007-May/438106.html
I had this same issue before and found this snippet very useful:
from: http://mail.python.org/pipermail/python-list/2007-May/438106.html
print 语句将调用您分配给 sys.stdout 的任何对象的
write()
方法。我会启动一个小类来编写一次到两个地方...
现在
print
语句将回显到屏幕并附加到您的日志文件中:这显然是快速而肮脏的。 一些注意事项:
如果您在程序执行期间不会进行记录。
这些都非常简单,我很乐意将它们作为练习留给读者。 这里的关键见解是
print
只是调用分配给sys.stdout
的“类文件对象”。The
print
statement will call thewrite()
method of any object you assign to sys.stdout.I would spin up a small class to write to two places at once...
Now the
print
statement will both echo to the screen and append to your log file:This is obviously quick-and-dirty. Some notes:
<stdout>
if youwon't be logging for the duration of the program.
These are all straightforward enough that I'm comfortable leaving them as exercises for the reader. The key insight here is that
print
just calls a "file-like object" that's assigned tosys.stdout
.由于您可以轻松地从代码中生成外部进程,因此您可以使用
tee
本身。 我不知道有任何 Unix 系统调用可以完全执行tee
的功能。您还可以使用 multiprocessing 包模拟
tee
(或者使用 processing(如果您使用的是 Python 2.5 或更早版本)。更新
这是 Python 3.3+ 兼容版本:
Since you're comfortable spawning external processes from your code, you could use
tee
itself. I don't know of any Unix system calls that do exactly whattee
does.You could also emulate
tee
using the multiprocessing package (or use processing if you're using Python 2.5 or earlier).Update
Here is a Python 3.3+-compatible version:
您真正想要的是标准库中的
logging
模块。 创建一个记录器并附加两个处理程序,一个将写入文件,另一个将写入 stdout 或 stderr。有关详细信息,请参阅记录到多个目标
What you really want is
logging
module from standard library. Create a logger and attach two handlers, one would be writing to a file and the other to stdout or stderr.See Logging to multiple destinations for details
这是另一个解决方案,它比其他解决方案更通用 - 它支持将输出(写入 sys.stdout)分割为任意数量的类似文件的对象。 不要求包含
__stdout__
本身。注意:这是一个概念验证。 这里的实现并不完整,因为它只包装了类文件对象的方法(例如
write
),省略了members/properties/setattr等。但是,它目前的情况对于大多数人来说可能已经足够好了。除了它的通用性之外,我喜欢它的一点是它很干净,因为它不会直接调用
write
、flush
、os .dup2
等Here is another solution, which is more general than the others -- it supports splitting output (written to
sys.stdout
) to any number of file-like objects. There's no requirement that__stdout__
itself is included.NOTE: This is a proof-of-concept. The implementation here is not complete, as it only wraps methods of the file-like objects (e.g.
write
), leaving out members/properties/setattr, etc. However, it is probably good enough for most people as it currently stands.What I like about it, other than its generality, is that it is clean in the sense it doesn't make any direct calls to
write
,flush
,os.dup2
, etc.