使用 python 记录文件和进程超时的输出

发布于 2024-10-16 21:37:00 字数 924 浏览 1 评论 0原文

我正在浏览一大堆文件并对所有文件运行命令。我想记录输出,如果该命令对一个文件花费的时间超过 5 分钟,我想停止该命令并转到下一个文件。

我有两个问题:

  1. 我想将文件名记录到输出文件,并记录输出消息。我使用 Popen 来记录消息并使用 communicate 来记录消息,但我使用 write() 写入的所有文件名都没有直到整个任务完成后才开始写作。

  2. 我不知道如何轮询该进程并在 5 分钟后退出并转到下一个文件。

下面是简化的代码:

import os, fnmatch
import subprocess
import sys
f=open('filenames','w')

'Locate all files matching supplied filename pattern in and below supplied root directory.'''

def locate(pattern, root=os.curdir):        
    for path, dirs, files in os.walk(os.path.abspath(root)):
        for filename in fnmatch.filter(files, pattern):
            yield os.path.join(path, filename)


for filename in locate("*.dll"):
    f.write(filename)   
    #cmd defintion is changed for simplicity 
    cmd='cat %s' %filename
    p=subprocess.Popen(cmd,stdout=f)   
    p.communicate()[0]

I am going through a big list of files and running a command on all of them. I want to log the output and if the command takes more than 5 minutes for a file, I would like to stop the command and go to the next file.

I have two problems:

  1. I want to record the file name to the output file and also record the output messages. I am using Popen to log the messages and using communicate so it gets logged, but all the filenames I am writing with write() don't get written till the whole task is done.

  2. I am not sure how I can poll the process and quit it after 5 minutes and go to the next file.

Below is simplified code:

import os, fnmatch
import subprocess
import sys
f=open('filenames','w')

'Locate all files matching supplied filename pattern in and below supplied root directory.'''

def locate(pattern, root=os.curdir):        
    for path, dirs, files in os.walk(os.path.abspath(root)):
        for filename in fnmatch.filter(files, pattern):
            yield os.path.join(path, filename)


for filename in locate("*.dll"):
    f.write(filename)   
    #cmd defintion is changed for simplicity 
    cmd='cat %s' %filename
    p=subprocess.Popen(cmd,stdout=f)   
    p.communicate()[0]

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

放我走吧 2024-10-23 21:37:00
  1. f.write(filename) 之后使用 f.flush()
  2. 请参阅使用超时模块“subprocess”
  1. Use f.flush() after f.write(filename).
  2. See Using module 'subprocess' with timeout
橪书 2024-10-23 21:37:00

我大致使用的是:

from datetime import datetime
from time import time

import subprocess
import sys
import threading
import os

class Execution(threading.Thread):
  comm = None
  proc = None
  returncode = None
  stdoutdata = None
  stderrdate = None

  def __init__(self, comm):
    threading.Thread.__init__(self)
    self.comm = comm

  def run(self):
    self.proc = subprocess.Popen(self.comm, bufsize=-1, stdin=None,  
                               stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    self.proc.wait()
    (self.stdoutdata, self.stderrdata) = self.proc.communicate()
    self.returncode = self.proc.poll()

for curcommand in somecommandlist:
  ex = Execution(curcommand)
  starttime = time()
  ex.start()
  ex.join(some_timeout_in_seconds)

  wastimeout = False
  if (ex.isAlive()):
    wastimeout = True
    print("TIMEOUT OCCURRED @ %s" % (datetime.today(),))
    ex.proc.kill()

  ex.join()
  endtime = time()
  duration = endtime - starttime
  print("DURATION %s" % (duration,))

  exitcode = ex.returncode
  print("EXIT CODE: %s" % (exitcode,))
  print somepostprocessing(ex.stdoutdata,ex.stderrdata)
  sys.stdout.flush()

基本上:在单独的线程内启动进程。这是必要的,因为 threading 模块会为您提供超时等待,而 subprocess 模块则不会。等待超时;检查线程是否还活着。如果线程处于活动状态,我们会通过终止进程来使其终止(这里,subprocess 为您提供一个 kill() 构造,其中 threading没有)并等待线程自行结束。

请记住,communicate() 会阻止并存储子级 stderr 和 stdout 的完整输出,因此如果输出非常大,这可能不起作用。

Here's what I use, roughly:

from datetime import datetime
from time import time

import subprocess
import sys
import threading
import os

class Execution(threading.Thread):
  comm = None
  proc = None
  returncode = None
  stdoutdata = None
  stderrdate = None

  def __init__(self, comm):
    threading.Thread.__init__(self)
    self.comm = comm

  def run(self):
    self.proc = subprocess.Popen(self.comm, bufsize=-1, stdin=None,  
                               stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    self.proc.wait()
    (self.stdoutdata, self.stderrdata) = self.proc.communicate()
    self.returncode = self.proc.poll()

for curcommand in somecommandlist:
  ex = Execution(curcommand)
  starttime = time()
  ex.start()
  ex.join(some_timeout_in_seconds)

  wastimeout = False
  if (ex.isAlive()):
    wastimeout = True
    print("TIMEOUT OCCURRED @ %s" % (datetime.today(),))
    ex.proc.kill()

  ex.join()
  endtime = time()
  duration = endtime - starttime
  print("DURATION %s" % (duration,))

  exitcode = ex.returncode
  print("EXIT CODE: %s" % (exitcode,))
  print somepostprocessing(ex.stdoutdata,ex.stderrdata)
  sys.stdout.flush()

Basically: start the process inside a separate thread. This is necessary because the threading module gives you timeout waiting, while the subprocess module does not. Wait until the timeout; check if the thread is still alive. If the thread is live, we cause it to terminate by killing the process (here, subprocess gives you a kill() construct, which threading does not) and waiting for the thread to end on its own accord.

Keep in mind that communicate() blocks and stores the full output from the child's stderr and stdout, so this may not work if the output is very large.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文