在Python中最大偏移时检索所有API响应

发布于 2025-02-01 01:12:38 字数 903 浏览 3 评论 0原文

我正在尝试从此

def pull_api_data():
    offset_limit = 0
    teltel_data = []
    # Loop through the results and add if present
    while True:
        print("Skip", offset_limit, "rows before beginning to return results")
        querystring = {"offset": "{}".format(offset_limit), "filter": "starttime>="'{}'.format(date_filter), "limit" : "5000"}
        response = session.get(url=url, headers=the_headers, params=querystring)
        data = response.json()['data']

        # Do we have more data from teltel ?
        if len(data) == 0:
            break
        # If yes ,then add the data to the main list ,teltel_data
        teltel_data.extend(data)
        # Increase offset_limit to skip the already added data
        offset_limit = offset_limit + 5000

    # transform the raw data by converting it to a dataframe and do necessary cleaning
    

pull_api_data()

I am attempting to retrieve data from this API that has a max offset of 200000. The records I am attempting to pull are more than the max offset. Below is a sample of the code I am using but when I reach the offset limit of 200000 it breaks (the API doesn't return any helpful response in terms of how many pages/requests I need to do that's why I am going until there are no more results ). I need to find a way to loop through and pull all the data. Thanks

def pull_api_data():
    offset_limit = 0
    teltel_data = []
    # Loop through the results and add if present
    while True:
        print("Skip", offset_limit, "rows before beginning to return results")
        querystring = {"offset": "{}".format(offset_limit), "filter": "starttime>="'{}'.format(date_filter), "limit" : "5000"}
        response = session.get(url=url, headers=the_headers, params=querystring)
        data = response.json()['data']

        # Do we have more data from teltel ?
        if len(data) == 0:
            break
        # If yes ,then add the data to the main list ,teltel_data
        teltel_data.extend(data)
        # Increase offset_limit to skip the already added data
        offset_limit = offset_limit + 5000

    # transform the raw data by converting it to a dataframe and do necessary cleaning
    

pull_api_data()

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文