如何使用openurl &多处理同时获取URL不同的数据？

发布于 2024-11-06 07:26:38 字数 940 浏览 3 评论 0原文

这个循环帮助从网络服务获取一些文本数据：

while True:
    u = urllib2.urlopen('http://url/%d.csv' %inputd)
    f=open('/folder/%s.csv' $inputd,'a')
    csvread = csv.reader(u)
    csvwrite = csv.writer(f)
    csvwrite.writerows(csvread)
    print 'OK','%e'%inputd
    time.sleep(30)

现在我尝试使用多重处理同时获取不同的数据标题：

import multiprocessing as mp
import urllib2,csv,random,logging,csv,time

inputd=(abc,def,ghi)

def dataget():
    u = urllib2.urlopen('http://url/%d.csv' %inputd)
    f=open('/folder/%s.csv' $inputd,'a')
    csvread = csv.reader(u)
    csvwrite = csv.writer(f)
    csvwrite.writerows(csvread)
    print 'OK','%e' % inputd
    time.sleep(30)

process=[]
for s in inputd:
    p = mp.Process(target=dataget) 
    ps.append(p)
    p.start()

我希望这可以完成“http://url/abc.csv”的获取和保存； “http://url/def.csv”；同时“http://url/ghi.csv”，然后在time.sleep之后重新执行。但它不起作用，或者同时执行相同的过程，我犯了什么逻辑错误？

原文

This loop help get some text data from web service:

while True:
    u = urllib2.urlopen('http://url/%d.csv' %inputd)
    f=open('/folder/%s.csv' $inputd,'a')
    csvread = csv.reader(u)
    csvwrite = csv.writer(f)
    csvwrite.writerows(csvread)
    print 'OK','%e'%inputd
    time.sleep(30)

now I try to get different data title in the same time use multiprocessing:

import multiprocessing as mp
import urllib2,csv,random,logging,csv,time

inputd=(abc,def,ghi)

def dataget():
    u = urllib2.urlopen('http://url/%d.csv' %inputd)
    f=open('/folder/%s.csv' $inputd,'a')
    csvread = csv.reader(u)
    csvwrite = csv.writer(f)
    csvwrite.writerows(csvread)
    print 'OK','%e' % inputd
    time.sleep(30)

process=[]
for s in inputd:
    p = mp.Process(target=dataget) 
    ps.append(p)
    p.start()

I hope this can do the getting and saving of "http://url/abc.csv"; "http://url/def.csv"; "http://url/ghi.csv" at the same time, then re-do it again after time.sleep. But it doesn't work, or do same process in the same time, what logic error did I make?

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

优雅的叶子 2024-11-13 07:26:38

您的多进程实现只会执行一次 url 抓取。
如果您希望数据获取一次又一次地执行这些操作，那么您就错过了 While True。

也许像这样它会起作用：

import multiprocessing as mp
import urllib2,csv,random,logging,csv,time

inputd=(abc,def,ghi)

def dataget(param):
    while True:
        u = urllib2.urlopen('http://url/%d.csv' %param)
        f=open('/folder/%s.csv' %param,'a')
        csvread = csv.reader(u)
        csvwrite = csv.writer(f)
        csvwrite.writerows(csvread)
        print 'OK','%e'%param
        time.sleep(30)

process=[]
for s in inputd:
    p = mp.Process(target=dataget, args=(s,)) 
    process.append(p)
    p.start()

Your multiprocess implementation will do the url grabbing only once.
If you want that your dataget do the stuff again and again, you missed the While True.

Maybe like this it will work :

import multiprocessing as mp
import urllib2,csv,random,logging,csv,time

inputd=(abc,def,ghi)

def dataget(param):
    while True:
        u = urllib2.urlopen('http://url/%d.csv' %param)
        f=open('/folder/%s.csv' %param,'a')
        csvread = csv.reader(u)
        csvwrite = csv.writer(f)
        csvwrite.writerows(csvread)
        print 'OK','%e'%param
        time.sleep(30)

process=[]
for s in inputd:
    p = mp.Process(target=dataget, args=(s,)) 
    process.append(p)
    p.start()

回复收藏 0 原文

~没有更多了~