urlfetchapp.fetch()似乎不改变用户代理

发布于 2025-02-02 14:33:28 字数 655 浏览 4 评论 0原文

尝试使用Google Apps脚本从网站获取数据,以将其直接放入电子表格中。 python请求等效效果很好的情况似乎不起作用。

Python代码:

page = requests.get("someurl?as_data_structure", headers={'user-agent':'testagent'})

燃气代码:

var page = UrlFetchApp.fetch("someurl?as_data_structure", headers={'user-agent':'testagent'});

唯一必需的标题是用户代理,如果我不包括标题,我通常会从Python代码中获得的错误是我通常会从Python代码中获得的。我是JS的新手,但据我所知,这是正确的方法。

编辑: 现在将标题放在正确的位置,但问题仍然存在,与以前完全相同。

var options = {"headers": {"User-Agent": "testagent"}};
var page = UrlFetchApp.fetch("someurl?as_data_structure", options);

Trying to grab data from a website using Google Apps Script to put it directly into a spreadsheet. The fetch does not seem to be working, where the Python requests equivalent works just fine.

Python code:

page = requests.get("someurl?as_data_structure", headers={'user-agent':'testagent'})

GAS code:

var page = UrlFetchApp.fetch("someurl?as_data_structure", headers={'user-agent':'testagent'});

The only required header is the user-agent, and the error I am getting from the GAS code is what I would usually get from the Python code if I hadn't included the header. I am new to js but as far as I know this is the proper way to do it..?

EDIT:
Now got the headers in the right place but the issue persists, exactly the same error as before.

var options = {"headers": {"User-Agent": "testagent"}};
var page = UrlFetchApp.fetch("someurl?as_data_structure", options);

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

乱了心跳 2025-02-09 14:33:28

Star★(在左上方)问题在这里让Google开发人员优先考虑此问题。


Google并不总是记录其限制(烦人?)。这样的限制是更改用户代理。它是修复的,

"User-Agent": "Mozilla/5.0 (compatible; Google-Apps-Script)"

无法更改它。

样本测试:

function testUrlFetchAppHeaders() {
  var options = {
    headers: {
      'User-Agent':
        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
    },
  };
  var fakeRequest = UrlFetchApp.getRequest(
    'https://www.httpbin.org/headers',
    options
  );//providing fake assurance
  var realRequest = UrlFetchApp.fetch(
    'https://www.httpbin.org/headers',
    options
  );//like a wrecking ball
  Logger.log({ fake: fakeRequest, real: realRequest });
}

样本响应:

{
  "fake": {
    "headers": {
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
    },
    "method": "get",
    "payload": "",
    "followRedirects": true,
    "validateHttpsCertificates": true,
    "useIntranet": false,
    "contentType": null,
    "url": "https://www.httpbin.org/headers"
  },
  "real": {
    "headers": {
      "Accept-Encoding": "gzip,deflate,br",
      "Host": "www.httpbin.org",
      "User-Agent": "Mozilla/5.0 (compatible; Google-Apps-Script)"
    }
  }
}

getRequest(url)

返回如果调用操作将提出的请求。

此方法实际上没有发出请求。

它也不能准确返回要提出的请求。

Star ★(on top left) the issue here for Google developers to prioritize the issue.


Google doesn't always document it's restrictions(Annoying?). One such restriction is changing the user agent. It's fixed to

"User-Agent": "Mozilla/5.0 (compatible; Google-Apps-Script)"

You can't change it.

Sample Test:

function testUrlFetchAppHeaders() {
  var options = {
    headers: {
      'User-Agent':
        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
    },
  };
  var fakeRequest = UrlFetchApp.getRequest(
    'https://www.httpbin.org/headers',
    options
  );//providing fake assurance
  var realRequest = UrlFetchApp.fetch(
    'https://www.httpbin.org/headers',
    options
  );//like a wrecking ball
  Logger.log({ fake: fakeRequest, real: realRequest });
}

Sample Response:

{
  "fake": {
    "headers": {
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
    },
    "method": "get",
    "payload": "",
    "followRedirects": true,
    "validateHttpsCertificates": true,
    "useIntranet": false,
    "contentType": null,
    "url": "https://www.httpbin.org/headers"
  },
  "real": {
    "headers": {
      "Accept-Encoding": "gzip,deflate,br",
      "Host": "www.httpbin.org",
      "User-Agent": "Mozilla/5.0 (compatible; Google-Apps-Script)"
    }
  }
}

getRequest(url)

Returns the request that would be made if the operation was invoked.

This method does not actually issue the request.

Neither does it accurately return the request that would be made.

孤蝉 2025-02-09 14:33:28

标题属于选项:

var options = {"headers": {"User-Agent": "testagent"}};
var page = UrlFetchApp.fetch("someurl?as_data_structure", options);

the headers belong into the options:

var options = {"headers": {"User-Agent": "testagent"}};
var page = UrlFetchApp.fetch("someurl?as_data_structure", options);
~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文