当前位置： 文江博客话题详情

REST API请求在气流中失败（＆＃x27;连接中止。

发布于 2025-01-30 13:13:13 字数 9069 浏览 3 评论 0原文

我使用请求库通过REST API请求数据。一切都在本地工作，但是当我部署到GCP Composer实例时，我会遇到错误。

def get_ids():
    s = requests.Session()
    retries = Retry(total=5, backoff_factor=1, status_forcelist=[429])
    s.mount('http://', HTTPAdapter(max_retries=retries))

    ids = []
    counter = 0
    url = "some_url"  + "/?limit=50"

    r = s.get(url, headers=headers)
    data = r.json()

    for i in data['items']:
        if 'custom' in i['user']:
            ids(i['user']['custom']['id'])

    while 'next' in  data['_links']:
        time.sleep(0.5)
        try:
            r = s.get("some url"+  data['_links']['next']['href'], headers=headers)
            data = r.json()
            counter+=1
            for i in data['items']:
                if 'custom' in i['user']:
                    try:
                        ids(i['user']['custom']['accountId'])
                    except KeyError:
                        pass
            
    return ids

在我的本地计算机上运行，​​但这是我在气流/作曲家中运行的错误：

[2022-05-19 01:43:10,041] {subprocess.py:78} INFO - Traceback (most recent call last):
[2022-05-19 01:43:10,053] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
[2022-05-19 01:43:10,066] {subprocess.py:78} INFO -     httplib_response = self._make_request(
[2022-05-19 01:43:10,074] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
[2022-05-19 01:43:10,082] {subprocess.py:78} INFO -     six.raise_from(e, None)
[2022-05-19 01:43:10,092] {subprocess.py:78} INFO -   File "<string>", line 3, in raise_from
[2022-05-19 01:43:10,111] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
[2022-05-19 01:43:10,118] {subprocess.py:78} INFO -     httplib_response = conn.getresponse()
[2022-05-19 01:43:10,136] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 1348, in getresponse
[2022-05-19 01:43:10,146] {subprocess.py:78} INFO -     response.begin()
[2022-05-19 01:43:10,160] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 316, in begin
[2022-05-19 01:43:10,180] {subprocess.py:78} INFO -     version, status, reason = self._read_status()
[2022-05-19 01:43:10,192] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 277, in _read_status
[2022-05-19 01:43:10,214] {subprocess.py:78} INFO -     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
[2022-05-19 01:43:10,224] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/socket.py", line 669, in readinto
[2022-05-19 01:43:10,243] {subprocess.py:78} INFO -     return self._sock.recv_into(b)
[2022-05-19 01:43:10,252] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/ssl.py", line 1241, in recv_into
[2022-05-19 01:43:10,274] {subprocess.py:78} INFO -     return self.read(nbytes, buffer)
[2022-05-19 01:43:10,281] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/ssl.py", line 1099, in read
[2022-05-19 01:43:10,301] {subprocess.py:78} INFO -     return self._sslobj.read(len, buffer)
[2022-05-19 01:43:10,322] {subprocess.py:78} INFO - ConnectionResetError: [Errno 104] Connection reset by peer
[2022-05-19 01:43:10,330] {subprocess.py:78} INFO - 
[2022-05-19 01:43:10,350] {subprocess.py:78} INFO - During handling of the above exception, another exception occurred:
[2022-05-19 01:43:10,359] {subprocess.py:78} INFO - 
[2022-05-19 01:43:10,379] {subprocess.py:78} INFO - Traceback (most recent call last):
[2022-05-19 01:43:10,386] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
[2022-05-19 01:43:10,406] {subprocess.py:78} INFO -     resp = conn.urlopen(
[2022-05-19 01:43:10,416] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
[2022-05-19 01:43:10,436] {subprocess.py:78} INFO -     retries = retries.increment(
[2022-05-19 01:43:10,443] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
[2022-05-19 01:43:10,463] {subprocess.py:78} INFO -     raise six.reraise(type(error), error, _stacktrace)
[2022-05-19 01:43:10,469] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
[2022-05-19 01:43:10,490] {subprocess.py:78} INFO -     raise value.with_traceback(tb)
[2022-05-19 01:43:10,518] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
[2022-05-19 01:43:10,539] {subprocess.py:78} INFO -     httplib_response = self._make_request(
[2022-05-19 01:43:10,575] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
[2022-05-19 01:43:10,578] {subprocess.py:78} INFO -     six.raise_from(e, None)
[2022-05-19 01:43:10,594] {subprocess.py:78} INFO -   File "<string>", line 3, in raise_from
[2022-05-19 01:43:10,621] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
[2022-05-19 01:43:10,639] {subprocess.py:78} INFO -     httplib_response = conn.getresponse()
[2022-05-19 01:43:10,675] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 1348, in getresponse
[2022-05-19 01:43:10,683] {subprocess.py:78} INFO -     response.begin()
[2022-05-19 01:43:10,700] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 316, in begin
[2022-05-19 01:43:10,707] {subprocess.py:78} INFO -     version, status, reason = self._read_status()
[2022-05-19 01:43:10,727] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 277, in _read_status
[2022-05-19 01:43:10,735] {subprocess.py:78} INFO -     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
[2022-05-19 01:43:10,755] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/socket.py", line 669, in readinto
[2022-05-19 01:43:10,763] {subprocess.py:78} INFO -     return self._sock.recv_into(b)
[2022-05-19 01:43:10,782] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/ssl.py", line 1241, in recv_into
[2022-05-19 01:43:10,803] {subprocess.py:78} INFO -     return self.read(nbytes, buffer)
[2022-05-19 01:43:10,906] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/ssl.py", line 1099, in read
[2022-05-19 01:43:10,913] {subprocess.py:78} INFO -     return self._sslobj.read(len, buffer)
[2022-05-19 01:43:10,927] {subprocess.py:78} INFO - urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2022-05-19 01:43:10,930] {subprocess.py:78} INFO - 
[2022-05-19 01:43:10,932] {subprocess.py:78} INFO - During handling of the above exception, another exception occurred:
[2022-05-19 01:43:10,937] {subprocess.py:78} INFO - 
[2022-05-19 01:43:10,940] {subprocess.py:78} INFO - Traceback (most recent call last):
[2022-05-19 01:43:10,943] {subprocess.py:78} INFO -   File "my_script.py", line 114, in <module>
[2022-05-19 01:43:10,945] {subprocess.py:78} INFO -     main(api_key, environment)
[2022-05-19 01:43:10,946] {subprocess.py:78} INFO -   File "my_script.py", line 95, in main
[2022-05-19 01:43:10,950] {subprocess.py:78} INFO -     ids = get_ids()
[2022-05-19 01:43:10,953] {subprocess.py:78} INFO -   File "my_script.py", line 39, in get_ids_ids
[2022-05-19 01:43:10,954] {subprocess.py:78} INFO -     r = s.get("some url", headers=headers)
[2022-05-19 01:43:10,956] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in get
[2022-05-19 01:43:10,958] {subprocess.py:78} INFO -     return self.request('GET', url, **kwargs)
[2022-05-19 01:43:10,964] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
[2022-05-19 01:43:10,966] {subprocess.py:78} INFO -     resp = self.send(prep, **send_kwargs)
[2022-05-19 01:43:10,966] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
[2022-05-19 01:43:10,967] {subprocess.py:78} INFO -     r = adapter.send(request, **kwargs)
[2022-05-19 01:43:10,969] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
[2022-05-19 01:43:10,972] {subprocess.py:78} INFO -     raise ConnectionError(err, request=request)
[2022-05-19 01:43:10,979] {subprocess.py:78} INFO - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2022-05-19 01:43:12,592] {subprocess.py:82} INFO - Command exited with return code 1
[2022-05-19 01:43:13,014] {taskinstance.py:1465} ERROR - Task failed with exception

这里有指导吗？我尝试增加睡眠，增加退缩，但没有任何效果。谢谢

原文

I'm requesting data through a rest API using the requests library. Everything works locally, but when I deploy to my GCP composer instance I experience errors.

def get_ids():
    s = requests.Session()
    retries = Retry(total=5, backoff_factor=1, status_forcelist=[429])
    s.mount('http://', HTTPAdapter(max_retries=retries))

    ids = []
    counter = 0
    url = "some_url"  + "/?limit=50"

    r = s.get(url, headers=headers)
    data = r.json()

    for i in data['items']:
        if 'custom' in i['user']:
            ids(i['user']['custom']['id'])

    while 'next' in  data['_links']:
        time.sleep(0.5)
        try:
            r = s.get("some url"+  data['_links']['next']['href'], headers=headers)
            data = r.json()
            counter+=1
            for i in data['items']:
                if 'custom' in i['user']:
                    try:
                        ids(i['user']['custom']['accountId'])
                    except KeyError:
                        pass
            
    return ids

Runs on my local machine, but this is the error I get running in airflow/composer:

[2022-05-19 01:43:10,041] {subprocess.py:78} INFO - Traceback (most recent call last):
[2022-05-19 01:43:10,053] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
[2022-05-19 01:43:10,066] {subprocess.py:78} INFO -     httplib_response = self._make_request(
[2022-05-19 01:43:10,074] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
[2022-05-19 01:43:10,082] {subprocess.py:78} INFO -     six.raise_from(e, None)
[2022-05-19 01:43:10,092] {subprocess.py:78} INFO -   File "<string>", line 3, in raise_from
[2022-05-19 01:43:10,111] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
[2022-05-19 01:43:10,118] {subprocess.py:78} INFO -     httplib_response = conn.getresponse()
[2022-05-19 01:43:10,136] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 1348, in getresponse
[2022-05-19 01:43:10,146] {subprocess.py:78} INFO -     response.begin()
[2022-05-19 01:43:10,160] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 316, in begin
[2022-05-19 01:43:10,180] {subprocess.py:78} INFO -     version, status, reason = self._read_status()
[2022-05-19 01:43:10,192] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 277, in _read_status
[2022-05-19 01:43:10,214] {subprocess.py:78} INFO -     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
[2022-05-19 01:43:10,224] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/socket.py", line 669, in readinto
[2022-05-19 01:43:10,243] {subprocess.py:78} INFO -     return self._sock.recv_into(b)
[2022-05-19 01:43:10,252] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/ssl.py", line 1241, in recv_into
[2022-05-19 01:43:10,274] {subprocess.py:78} INFO -     return self.read(nbytes, buffer)
[2022-05-19 01:43:10,281] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/ssl.py", line 1099, in read
[2022-05-19 01:43:10,301] {subprocess.py:78} INFO -     return self._sslobj.read(len, buffer)
[2022-05-19 01:43:10,322] {subprocess.py:78} INFO - ConnectionResetError: [Errno 104] Connection reset by peer
[2022-05-19 01:43:10,330] {subprocess.py:78} INFO - 
[2022-05-19 01:43:10,350] {subprocess.py:78} INFO - During handling of the above exception, another exception occurred:
[2022-05-19 01:43:10,359] {subprocess.py:78} INFO - 
[2022-05-19 01:43:10,379] {subprocess.py:78} INFO - Traceback (most recent call last):
[2022-05-19 01:43:10,386] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
[2022-05-19 01:43:10,406] {subprocess.py:78} INFO -     resp = conn.urlopen(
[2022-05-19 01:43:10,416] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
[2022-05-19 01:43:10,436] {subprocess.py:78} INFO -     retries = retries.increment(
[2022-05-19 01:43:10,443] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
[2022-05-19 01:43:10,463] {subprocess.py:78} INFO -     raise six.reraise(type(error), error, _stacktrace)
[2022-05-19 01:43:10,469] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
[2022-05-19 01:43:10,490] {subprocess.py:78} INFO -     raise value.with_traceback(tb)
[2022-05-19 01:43:10,518] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
[2022-05-19 01:43:10,539] {subprocess.py:78} INFO -     httplib_response = self._make_request(
[2022-05-19 01:43:10,575] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
[2022-05-19 01:43:10,578] {subprocess.py:78} INFO -     six.raise_from(e, None)
[2022-05-19 01:43:10,594] {subprocess.py:78} INFO -   File "<string>", line 3, in raise_from
[2022-05-19 01:43:10,621] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
[2022-05-19 01:43:10,639] {subprocess.py:78} INFO -     httplib_response = conn.getresponse()
[2022-05-19 01:43:10,675] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 1348, in getresponse
[2022-05-19 01:43:10,683] {subprocess.py:78} INFO -     response.begin()
[2022-05-19 01:43:10,700] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 316, in begin
[2022-05-19 01:43:10,707] {subprocess.py:78} INFO -     version, status, reason = self._read_status()
[2022-05-19 01:43:10,727] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/http/client.py", line 277, in _read_status
[2022-05-19 01:43:10,735] {subprocess.py:78} INFO -     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
[2022-05-19 01:43:10,755] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/socket.py", line 669, in readinto
[2022-05-19 01:43:10,763] {subprocess.py:78} INFO -     return self._sock.recv_into(b)
[2022-05-19 01:43:10,782] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/ssl.py", line 1241, in recv_into
[2022-05-19 01:43:10,803] {subprocess.py:78} INFO -     return self.read(nbytes, buffer)
[2022-05-19 01:43:10,906] {subprocess.py:78} INFO -   File "/opt/python3.8/lib/python3.8/ssl.py", line 1099, in read
[2022-05-19 01:43:10,913] {subprocess.py:78} INFO -     return self._sslobj.read(len, buffer)
[2022-05-19 01:43:10,927] {subprocess.py:78} INFO - urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2022-05-19 01:43:10,930] {subprocess.py:78} INFO - 
[2022-05-19 01:43:10,932] {subprocess.py:78} INFO - During handling of the above exception, another exception occurred:
[2022-05-19 01:43:10,937] {subprocess.py:78} INFO - 
[2022-05-19 01:43:10,940] {subprocess.py:78} INFO - Traceback (most recent call last):
[2022-05-19 01:43:10,943] {subprocess.py:78} INFO -   File "my_script.py", line 114, in <module>
[2022-05-19 01:43:10,945] {subprocess.py:78} INFO -     main(api_key, environment)
[2022-05-19 01:43:10,946] {subprocess.py:78} INFO -   File "my_script.py", line 95, in main
[2022-05-19 01:43:10,950] {subprocess.py:78} INFO -     ids = get_ids()
[2022-05-19 01:43:10,953] {subprocess.py:78} INFO -   File "my_script.py", line 39, in get_ids_ids
[2022-05-19 01:43:10,954] {subprocess.py:78} INFO -     r = s.get("some url", headers=headers)
[2022-05-19 01:43:10,956] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in get
[2022-05-19 01:43:10,958] {subprocess.py:78} INFO -     return self.request('GET', url, **kwargs)
[2022-05-19 01:43:10,964] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
[2022-05-19 01:43:10,966] {subprocess.py:78} INFO -     resp = self.send(prep, **send_kwargs)
[2022-05-19 01:43:10,966] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
[2022-05-19 01:43:10,967] {subprocess.py:78} INFO -     r = adapter.send(request, **kwargs)
[2022-05-19 01:43:10,969] {subprocess.py:78} INFO -   File "/tmp/venv/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
[2022-05-19 01:43:10,972] {subprocess.py:78} INFO -     raise ConnectionError(err, request=request)
[2022-05-19 01:43:10,979] {subprocess.py:78} INFO - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2022-05-19 01:43:12,592] {subprocess.py:82} INFO - Command exited with return code 1
[2022-05-19 01:43:13,014] {taskinstance.py:1465} ERROR - Task failed with exception

any guidance here? I've tried increasing the sleep, increasing the backoff, but nothing's worked. Thanks

如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论， 你可以免费 注册 一个本站的账号。
列表为空，暂无数据

关于作者

风月客

暂无简介

文章
评论
26 人气
发私信

相关话题

更多

热门标签

操作系统 程序设计 IT运维 Linux系统管理 JavaScript 服务器应用 solaris C/C++ PHP Shell BSD Vue.js aix Oracle Python HTML 系统管理 HTML5 CSS 前端
更多

推荐作者

十二

文章 0 评论 0

飞烟轻若梦

文章 0 评论 0

OPleyuhuo

文章 0 评论 0

wxb0109

文章 0 评论 0

旧城空念

文章 0 评论 0

-小熊_

文章 0 评论 0

更多

友情链接

文江博客
    我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站，即表示您同意使用 Cookies 和您的相关数据。
    原文