python geocoder调用在一定数量的执行后冻结
我正在研究Python应用程序,该应用程序使用地理编码器库来找到给定地址的纬度和经度信息,并将其存储在Redis密钥中。所有信息均已序列化并存储在单个redis密钥中,从那里,Nodejs应用程序将读取此数据并将其渲染到地图。
现在,当我运行代码时,对于某些地址,地理信息不可用,我将这些值存储在字符串na
作为数组中的第一个元素中。对于具有正确的地理信息的所有地址,数组中的第一个元素是true
。
我还有另一个python脚本,该脚本检查是否可用于这些缺失条目的地理信息,如果有的话,它将使用此信息更新redis值。因此,我有以下代码:
latlonData = pickle.loads(r.get("latlon_all"))
i = 0
for orgid, row in latlonData.items():
if latlonData[orgid][0] == "na":
accuracy, latitude, longitude = self.getCoordinates(row[3])
if latitude is not None:
latlonData[orgid] = [accuracy, latitude, longitude, latlonData[orgid][3], latlonData[orgid][4]]
print(latlonData[orgid])
i += 1
r.append('lat_lon_append', latlonData[orgid])
r.set('latlon_all', pickle.dumps(latlonData))
getCoordinates()
函数如下:
import pickle
import geocoder
def getCoordinates(self, query):
accuracy, latitude, longitude = 'na', None, None
apiKey = 'MYAPIKEY'
geoCodes = geocoder.google(query, key=apiKey)
print(geoCodes.json)
if geoCodes.json is not None:
if geoCodes.json['status'] == 'OK':
latitude = geoCodes.json['lat']
longitude = geoCodes.json['lng']
quality = geoCodes.json.get('quality', None)
if quality:
if geoCodes.json['quality'] == 'STREET':
accuracy = 'true'
else:
accuracy = 'false'
else:
accuracy = 'false'
else:
pass
return accuracy, latitude, longitude
我有许多地址(12000+),几乎对于4600,地理信息丢失。我循环浏览每个地址,并检查na
值是否存在,如果是这样,请致电Google Geocode API获取信息。一段时间后,代码被冻结了。我在那里打印了i
值,在此之后,它是2497,代码完全冷冻并且没有显示任何错误。我必须手动杀死该过程以停止执行。
我试图以检查i
的值的条件运行脚本,如果它超过2496,则只会查询地理信息。但这也经过一段迭代后也被冻结了。我是Python的新手,不确定这里发生了什么。当我检查限制时,Google API没有最大限制,但是每秒50个请求的限制。我认为我的代码在极限之下,但在API调用之间添加了2秒钟的睡眠。脚本仍然具有相同的行为,并且被冻结了。如果有人可以检查此代码并建议我这样做的更好的方法,而不会
在我杀死代码执行时添加堆栈跟踪的情况下,这将是有帮助的:
Traceback (most recent call last):
File "/scripts/temp/geo_na_process_google.py", line 94, in <module>
geoObj.process()
File "/scripts/temp/geo_na_process_google.py", line 16, in process
self.checkRecords()
File "/scripts/temp/geo_na_process_google.py", line 25, in checkRecords
accuracy, latitude, longitude = self.getCoordinates(row[3])
File "/scripts/temp/geo_na_process_google.py", line 43, in getCoordinates
geoCodes = geocoder.google(query, key=apiKey)
File "Library/Python/3.8/lib/python/site-packages/geocoder/api.py", line 232, in google
return get(location, provider='google', **kwargs)
File "Library/Python/3.8/lib/python/site-packages/geocoder/api.py", line 198, in get
return options[provider][method](location, **kwargs)
File "Library/Python/3.8/lib/python/site-packages/geocoder/base.py", line 410, in __init__
self._initialize()
File "Library/Python/3.8/lib/python/site-packages/geocoder/base.py", line 457, in _initialize
json_response = self._connect()
File "Library/Python/3.8/lib/python/site-packages/geocoder/base.py", line 479, in _connect
self.response = response = self.rate_limited_get(
File "Library/Python/3.8/lib/python/site-packages/geocoder/google.py", line 269, in rate_limited_get
return self.rate_limited_get_for_dev(*args, **kwargs)
File "Library/Python/3.8/lib/python/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "Library/Python/3.8/lib/python/site-packages/ratelim/__init__.py", line 33, in wrapped_f
time.sleep(self.__time_interval - time_delta + 1)
KeyboardInterrupt
I am working on a Python application, which uses the geocoder library to find the latitude and longitude information for a given address and store it in a Redis key. All the information is serialized and stored in a single Redis key and from there a NodeJS application reads this data and renders it to a Map.
Now, when I run the code, for some of the addresses, the geoinformation is not available and I store these values with the string na
as the first element in the array. For all the addresses, which has correct geo info, the first element in the array is true
.
I have another Python script, which checks whether the geo-information is available for these missing entries, and if it is there, it will update the Redis value with this info. So I have the following code:
latlonData = pickle.loads(r.get("latlon_all"))
i = 0
for orgid, row in latlonData.items():
if latlonData[orgid][0] == "na":
accuracy, latitude, longitude = self.getCoordinates(row[3])
if latitude is not None:
latlonData[orgid] = [accuracy, latitude, longitude, latlonData[orgid][3], latlonData[orgid][4]]
print(latlonData[orgid])
i += 1
r.append('lat_lon_append', latlonData[orgid])
r.set('latlon_all', pickle.dumps(latlonData))
and the getCoordinates()
function is as follows:
import pickle
import geocoder
def getCoordinates(self, query):
accuracy, latitude, longitude = 'na', None, None
apiKey = 'MYAPIKEY'
geoCodes = geocoder.google(query, key=apiKey)
print(geoCodes.json)
if geoCodes.json is not None:
if geoCodes.json['status'] == 'OK':
latitude = geoCodes.json['lat']
longitude = geoCodes.json['lng']
quality = geoCodes.json.get('quality', None)
if quality:
if geoCodes.json['quality'] == 'STREET':
accuracy = 'true'
else:
accuracy = 'false'
else:
accuracy = 'false'
else:
pass
return accuracy, latitude, longitude
I have a number of addresses (12000+), and almost for 4600, the geoinformation is missing. I loop through every address and check whether the na
value is there and if it is so, calling Google GeoCode API to get the info. Now after some time, the code is getting frozen. I have printed the i
value there and it is 2497 after this the code is completely frozen and is not showing any error. I have to kill the process manually to stop the execution.
I tried to run the script with a condition that checks the value for i
and if it is more than 2496, then only it will query the geoinformation. But this is also getting frozen after some iteration. I am new to Python and not sure what is happening here. When I checked the limits, there are no maximum limits for the Google API, but there is a limit of 50 requests per second. I think my code is well under the limits, but have added a sleep for 2 seconds between the API calls. Still the script has the same behavior and it is frozen. It would be helpful if someone can check this code and advice me on a better way of doing this without getting stuck after a certain number of iternations
Adding the stack trace when I kill the code execution:
Traceback (most recent call last):
File "/scripts/temp/geo_na_process_google.py", line 94, in <module>
geoObj.process()
File "/scripts/temp/geo_na_process_google.py", line 16, in process
self.checkRecords()
File "/scripts/temp/geo_na_process_google.py", line 25, in checkRecords
accuracy, latitude, longitude = self.getCoordinates(row[3])
File "/scripts/temp/geo_na_process_google.py", line 43, in getCoordinates
geoCodes = geocoder.google(query, key=apiKey)
File "Library/Python/3.8/lib/python/site-packages/geocoder/api.py", line 232, in google
return get(location, provider='google', **kwargs)
File "Library/Python/3.8/lib/python/site-packages/geocoder/api.py", line 198, in get
return options[provider][method](location, **kwargs)
File "Library/Python/3.8/lib/python/site-packages/geocoder/base.py", line 410, in __init__
self._initialize()
File "Library/Python/3.8/lib/python/site-packages/geocoder/base.py", line 457, in _initialize
json_response = self._connect()
File "Library/Python/3.8/lib/python/site-packages/geocoder/base.py", line 479, in _connect
self.response = response = self.rate_limited_get(
File "Library/Python/3.8/lib/python/site-packages/geocoder/google.py", line 269, in rate_limited_get
return self.rate_limited_get_for_dev(*args, **kwargs)
File "Library/Python/3.8/lib/python/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "Library/Python/3.8/lib/python/site-packages/ratelim/__init__.py", line 33, in wrapped_f
time.sleep(self.__time_interval - time_delta + 1)
KeyboardInterrupt
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论