如何在 Twisted 中对 FTP 命令进行排队？
我正在使用 Twisted 编写一个 FTP 客户端，它可以下载大量文件，并且我正在尝试以非常智能的方式完成它。但是，我一直遇到一个问题，我会很快下载几个文件（有时每批约 20 个，有时约 250 个），然后下载会挂起，最终连接超时，然后下载并挂起开始一切重新来过。我使用 DeferredSemaphore 一次只下载 3 个文件，但我现在怀疑这可能不是避免限制服务器的正确方法。
以下是有问题的代码：
def downloadFiles(self, result, directory):
# make download directory if it doesn't already exist
if not os.path.exists(directory['filename']):
os.makedirs(directory['filename'])
log.msg("Downloading files in %r..." % directory['filename'])
files = filterFiles(None, self.fileListProtocol)
# from http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker/2862440#2862440
# use a DeferredSemaphore to limit the number of files downloaded simultaneously from the directory to 3
sem = DeferredSemaphore(3)
jobs = [sem.run(self.downloadFile, f, directory) for f in files]
d = gatherResults(jobs)
return d
def downloadFile(self, f, directory):
filename = os.path.join(directory['filename'], f['filename']).encode('ascii')
log.msg('Downloading %r...' % filename)
d = self.ftpClient.retrieveFile(filename, FTPFile(filename))
return d
您会注意到我正在重用 FTP 连接（顺便说一下，是活动的）并使用我自己的 FTPFile 实例来确保当文件下载连接“丢失”时本地文件对象关闭（即完成）。看着 FTPClient 我想知道我是否应该直接使用 queueCommand 。老实说，我迷失了 retrieveFile< /a> 命令 _openDataConnection 及其他，所以也许它已经被使用了。
有什么建议吗？谢谢！
I'm writing an FTP client using Twisted that downloads a lot of files and I'm trying to do it pretty intelligently. However, I've been having the problem that I'll download several files very quickly (sometimes ~20 per batch, sometimes ~250) and then the downloading will hang, only to eventually have connections time out and then the download and hang start all over again. I'm using a DeferredSemaphore to only download 3 files at a time, but I now suspect that this is probably not the right way to avoid throttling the server.
Here is the code in question:
def downloadFiles(self, result, directory):
# make download directory if it doesn't already exist
if not os.path.exists(directory['filename']):
os.makedirs(directory['filename'])
log.msg("Downloading files in %r..." % directory['filename'])
files = filterFiles(None, self.fileListProtocol)
# from http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker/2862440#2862440
# use a DeferredSemaphore to limit the number of files downloaded simultaneously from the directory to 3
sem = DeferredSemaphore(3)
jobs = [sem.run(self.downloadFile, f, directory) for f in files]
d = gatherResults(jobs)
return d
def downloadFile(self, f, directory):
filename = os.path.join(directory['filename'], f['filename']).encode('ascii')
log.msg('Downloading %r...' % filename)
d = self.ftpClient.retrieveFile(filename, FTPFile(filename))
return d
You'll noticed that I'm reusing an FTP connection (active, by the way) and using my own FTPFile instance to make sure the local file object gets closed when the file download connection is 'lost' (ie completed). Looking at FTPClient I wonder if I should be using queueCommand directly. To be honest, I got lost following the retrieveFile command to _openDataConnection and beyond, so maybe it's already being used.
Any suggestions? Thanks!
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我建议使用queueCommand，正如您所建议的，我怀疑您正在使用的信号量可能会导致您出现问题。我相信使用queueCommand会将您的FTPClient限制为单个活动连接（尽管我只是推测），因此如果您想快速完成操作，您可能需要考虑创建一些FTPClient实例并将下载作业传递给它们。如果您使用 queueStringCommand，您会得到a Deferred，您可以使用它来确定每个客户端的状态，甚至可以在回调中将另一个作业添加到该客户端的队列中。
I would suggest using queueCommand, as you suggested I'd suspect the semaphore you're using is probably causing you issues. I believe using queueCommand will limit your FTPClient to a single active connection (though I'm just speculating), so you may want to think about creating a few FTPClient instances and passing download jobs to them if you want to do things quickly. If you use queueStringCommand, you get a Deferred that you can use to determine where each client is up to, and even add another job to the queue for that client in the callback.