Nginx ingress 中如何透传upstream header？

Question

 

最近发现ingress中的一个问题，通过ingress 反向代理后端的nginx server时 发现跨域配置丢失（cors headers)，没办法透传过去。

请求的链路是这样的：client --> ingress --> nginx server

遇到的问题是，在nginx server上面add header 添加跨域头部配置，然后通过client 访问的时候，获取不到该头部：

a、 client 访问 nginx server    可以获取到

b、client 访问ingress 获取不到

c、client 访问ingress ，但ingress upstream 配置直接指定到nginx server，绕过lua逻辑       可以获取到

 

我在nginx server 上加了跨域配置如下：

location ~ ^/note {
  proxy_redirect         off;
  proxy_set_header       Host $host;
  proxy_set_header  X-Real-IP  $remote_addr;
  proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Referer $http_referer;
  proxy_pass  http://testweb;
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  add_header Access-Control-Allow-Origin http://testcors.abc.com;
  add_header Access-Control-Allow-Headers 'DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';
  add_header Access-Control-Allow-Methods 'GET, POST, OPTIONS';
}

 

做了以下的测试：

(1) client --> ingress --> nginx   跨域头部丢失

root@kafka-03:~# curl http://test.abc.com/note/view?id=93cM4ef4M8732M0cf509c  -x"${Ingress-IP}:80" -i
HTTP/1.1 200 
Server: nginx
Date: Tue, 06 Aug 2019 07:57:00 GMT
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive

 

(2) client --> nginx --> nginx   有跨域头部

root@kafka-03:/data/services/nginx_vhost# curl root@kafka-03:~# curl http://test.abc.com/note/view?id=93cM4ef4M8732M0cf509c  -x"${NginxServer}:80" -i
HTTP/1.1 200 
Server: nginx
Date: Tue, 06 Aug 2019 07:57:00 GMT
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: http://testcors.abc.com
Access-Control-Allow-Headers: DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization
Access-Control-Allow-Methods: GET, POST, OPTIONS

 

(3) client --> ingress(修改nginx.conf 直接将请求转发给后端nginx server) --> nginx  

测试目的：测试绕过lua的处理逻辑，通过ingress 中自带nginx 能否透传cors header  

ingress 原生的实现是通过lua 实现动态负载均衡，配置如下：

proxy_pass http://upstream_balancer; 

尝试过编辑ingress 生成的nginx.conf，将upstream 配置改成原生nginx支持的配置，例如：

upstream testcors-web
{
    server 1.1.1.1:80 max_fails=2 fail_timeout=10s   weight=1;
}
server{
    location / {
    ... ...
    proxy_pass http://testcors-web;
  }
}

这样是可以拿到cors 的header

 

通过上面的测试，可以断定nginx 是可以透传头部的，是lua 的处理逻辑导致header 没有传递过来。

ingress 生成的nginx配置如下：

location / {
  set $namespace      "admin";
  set $ingress_name   "zhoubao-42099b4af02";
  set $service_name   "zhoubao-42099b4af02";
  set $service_port   "80";
  set $location_path  "/";
  proxy_pass_header location;

  header_filter_by_lua_block {

  }
  body_filter_by_lua_block {

  }

  log_by_lua_block {

      balancer.log()

      monitor.call()

  }

  port_in_redirect off;

  set $proxy_upstream_name    "admin-zhoubao-42099b4af02-80";
  set $proxy_host             $proxy_upstream_name;

  client_max_body_size                    1m;

  proxy_set_header Host                   $best_http_host;

  # Pass the extracted client certificate to the backend

  # Allow websocket connections
  proxy_set_header                        Upgrade           $http_upgrade;

  proxy_set_header                        Connection        $connection_upgrade;

  proxy_set_header X-Request-ID           $req_id;
  proxy_set_header X-Real-IP              $the_real_ip;

  proxy_set_header X-Forwarded-For        $the_real_ip;

  proxy_set_header X-Forwarded-Host       $best_http_host;
  proxy_set_header X-Forwarded-Port       $pass_port;
  proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

  proxy_set_header X-Original-URI         $request_uri;                                                                                                                                                     
  proxy_set_header X-Forwarded-For        $the_real_ip;

  proxy_set_header X-Forwarded-Host       $best_http_host;
  proxy_set_header X-Forwarded-Port       $pass_port;
  proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

  proxy_set_header X-Original-URI         $request_uri;

  proxy_set_header X-Scheme               $pass_access_scheme;

  # Pass the original X-Forwarded-For
  proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

  # mitigate HTTPoxy Vulnerability
  # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
  proxy_set_header Proxy                  "";

  proxy_pass_request_headers on;

  # Custom headers to proxied server

  proxy_connect_timeout                   10s;
  proxy_send_timeout                      60s;
  proxy_read_timeout                      60s;

  proxy_buffering                         off;
  proxy_buffer_size                       4k;
  proxy_buffers                           4 4k;
  proxy_request_buffering                 on;


  proxy_http_version                      1.1;

  proxy_cookie_domain                     off;
  proxy_cookie_path                       off;

  # In case of errors try the next upstream server before returning an error
  proxy_next_upstream                     http_502 http_504 error timeout invalid_header;
  proxy_next_upstream_tries               3;

  proxy_pass http://upstream_balancer;

  proxy_redirect                          off;
}

尝试过在lua中打印ngx.reps.headers 只能拿到content-type 和 connetion 的响应头

大家帮忙看看有没有思路？

 

 

 

 

 

 

Answer 1

已解决，在header_filter_by_lua 里面打印了下转发的后端服务器ip:port 发现，nginx server 配置有问题（容器化部署的，业务侧配置了upstream 直接绕过了nginx 访问后端服务，导致在后端nginx server 上配置的add header 不生效）

            header_filter_by_lua_block {
                local upstream_ip = ngx.var["upstream_addr"]
                ngx.log(ngx.ERR,"test ip:" , upstream_ip)
            }

 

2019/08/09 12:46:33 [error] 33985#33985: *9385076 [lua] header_filter_by_lua:3: test ip:10.1.1.1:8073 while reading response header from upstream, client: 10.1.1.10, server: test.abc.com, request: "GET http://test.abc.com/note/view?id=93cM4ef4M8732M0cf509c HTTP/1.1", upstream: "http://10.1.1.1:8073/note/view?id=93cM4ef4M8732M0cf509c", host: "test.abc.com"

修改了下service target_port 即可