2016年07月03日 from—http://www.itts-union.com/2923.html

Catalog

[隐藏]

概述

  linuxcurl是通过url语法在命令行下上传或下载文件的工具软件,它支持http,https,ftp,ftps,telnet等多种协议,常被用来抓取网页和监控Web服务器状态。

  官网下载地址 http://curl.haxx.se/download/curl-7.46.0.tar.gz

作用1 测试某一条url的相关情况

包括是否缓存,返回字段是否正确等。相当于模拟浏览器客户端做get操作

1
2
3
4
5
6
7
8
9
10
11
<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -I 'http://app613.imgcache.qzoneapp.com/app613/frontend_swf/version.js'
HTTP/1.1 200 OK
Server: NWS_TCloud_S1
Connection: keep-alive
Date: Sun, 03 Jul 2016 08:48:13 GMT
Cache-Control: max-age=6000
Expires: Sun, 03 Jul 2016 10:28:13 GMT
Last-Modified: Thu, 30 Jun 2016 09:12:30 GMT
Content-Type: application/x-javascript; charset=utf-8
Content-Length: 69
X-Cache-Lookup: Hit From MemCache<br></p>

作用2 验证某节点ip缓存情况

1
2
3
4
5
6
7
8
9
10
11
<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -I 'http://app613.imgcache.qzoneapp.com/app613/frontend_swf/version.js' -x 106.74.23.144:80
HTTP/1.1 200 OK
Server: NWS_TCloud_S1
Connection: keep-alive
Date: Sun, 03 Jul 2016 08:50:34 GMT
Cache-Control: max-age=6000
Expires: Sun, 03 Jul 2016 10:30:34 GMT
Last-Modified: Thu, 30 Jun 2016 09:12:30 GMT
Content-Type: application/x-javascript; charset=utf-8
Content-Length: 69
X-Cache-Lookup: Hit From Disktank<br></p>

PS:其中的-x绑定代理服务器ip的用法,类似于wget中的:

# wget -e http-proxy=119.97.137.178 http://sdgautopatch.cloudcdn.net/test008/6wireshark-win64-1.2.9.exe

作用3 验证referer头部白名单是否生效

场景

 使用-H 来添加自定义的头部,模拟特殊条件下源站/cdn节点对特性是否支持,结果返回是否符合预期。下面的这个例子,就是用来验证在控制台上添加了referer白名单后,模拟该referer头部返回是否符合预期。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -I 'http://res2.eqh5.com/' -x 106.74.23.144:80 -H"referer:http://*.yqh5.cn"
HTTP/1.1 200 OK
Server: NWS_TCloud_S1
Connection: keep-alive
Date: Sun, 03 Jul 2016 08:59:50 GMT
Cache-Control: max-age=31536000
Expires: Mon, 03 Jul 2017 08:59:50 GMT
Last-Modified: Tue, 12 Apr 2016 01:59:32 GMT
Content-Type: text/html
Content-Length: 612
X-Cache-Lookup: Hit From Disktank
Access-Control-Allow-Origin: *
Content-Disposition: inline; filename=""
Accept-Ranges: bytes<br></p>

 

PS 上面的

curl -I ‘http://res2.eqh5.com/’ -x 106.74.23.144:80 -H”referer:http://*.yqh5.cn”

也可以写为:

curl -I ‘http://res2.eqh5.com/’ -x 106.74.23.144:80  -e “http://pachong.org/”

作用4 验证节点/源站对分片的支持情况

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -Iv 'http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip' -x 106.74.23.144:80 -H'range:bytes=0-1000'
* About to connect() to proxy 106.74.23.144 port 80 (#0)
*   Trying 106.74.23.144... connected
* Connected to 106.74.23.144 (106.74.23.144) port 80 (#0)
> HEAD http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: update01.skyworth-cloud.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> range:bytes=0-1000
>
< HTTP/1.1 206 Partial Content
HTTP/1.1 206 Partial Content
< Server: nws_ocmid_hy
Server: nws_ocmid_hy
< Connection: close
Connection: close
< Date: Sun, 03 Jul 2016 09:02:24 GMT
Date: Sun, 03 Jul 2016 09:02:24 GMT
< Cache-Control: max-age=600
Cache-Control: max-age=600
< Expires: Sun, 03 Jul 2016 09:12:24 GMT
Expires: Sun, 03 Jul 2016 09:12:24 GMT
< Last-Modified: Mon, 23 May 2016 20:10:06 GMT
Last-Modified: Mon, 23 May 2016 20:10:06 GMT
< Content-Range: bytes 0-1000/437405243
Content-Range: bytes 0-1000/437405243
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Content-Length: 1001
Content-Length: 1001
< X-Daa-Tunnel: hop_count=2
X-Daa-Tunnel: hop_count=2
< X-Cache-Lookup: Hit From Disktank Upstream
X-Cache-Lookup: Hit From Disktank Upstream
< X-Cache-Lookup: Hit From Disktank
X-Cache-Lookup: Hit From Disktank
< X-Daa-Tunnel: hop_count=2
X-Daa-Tunnel: hop_count=2
< X-Cache-Lookup: Hit From Inner Cluster
X-Cache-Lookup: Hit From Inner Cluster
<
* Closing connection #0<br><br></p>

备注

上面的curl -Iv ‘http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip’ -x 106.74.23.144:80 -H’range:bytes=0-1000′

也可以写成

curl -Iv ‘http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip’ -x 106.74.23.144:80 -r 0-1000

作用5 验证文件在节点和源站上大小是否一致(通过md5判断)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -o pic1 'http://as.eqxiu.com/show/visitor/4a2d8aae508506ef0150a6fcebbf2f53' -x 106.74.23.144:80
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10126  100 10126    0     0  40629      0 --:--:-- --:--:-- --:--:-- 54149
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]# md5sum pic1
4b7671dbe1a634541726bd9473d14dcc  pic1
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]# curl -o pic2 'http://as.eqxiu.com/show/visitor/4a2d8aae508506ef0150a6fcebbf2f53' -x 115.159.20.55:80
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10126  100 10126    0     0  4629k      0 --:--:-- --:--:-- --:--:-- 9888k
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]# md5sum pic2
4b7671dbe1a634541726bd9473d14dcc  pic2<br></p>

备注

上面的-o 代表把文件下载到指定的路径下,-x代表使用指定的ip或者域名作为代理。

 

作用6 模拟https访问验证https证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@VM_117_214_centos ~]#  curl -sv 'https://mccdn.qcloud.com/static/img/57c45bb3997a277e5996dcf13314b365/user-40.png' -o /dev/null -k
* About to connect() to mccdn.qcloud.com port 443 (#0)
*   Trying 113.207.16.63... connected
* Connected to mccdn.qcloud.com (113.207.16.63) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*     subject: CN=*.qcloud.com,OU=R&D,O=Shenzhen Tencent Computer Systems Company Limited,L=shenzhen,ST=guangdong,C=CN
*     start date: May 25 00:00:00 2016 GMT
*     expire date: Jul 24 23:59:59 2019 GMT
*     common name: *.qcloud.com
*     issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /static/img/57c45bb3997a277e5996dcf13314b365/user-40.png HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: mccdn.qcloud.com
> Accept: */*
< HTTP/1.1 200 OK
< Server: nnws/1.7.3.8
< Date: Sat, 20 Aug 2016 12:43:55 GMT
< Content-Type: image/png
< Content-Length: 3558
< Connection: keep-alive
< Cache-Control: max-age=2592000
< Expires: Mon, 19 Sep 2016 12:43:54 GMT
< Last-Modified: Wed, 25 May 2016 08:06:29 GMT
< X-Cache-Lookup: Hit From Disktank
{ [data not shown]
* Connection #0 to host mccdn.qcloud.com left intact
* Closing connection #0

作用7 curl中绑定host测试方法

等同于在windows中绑定host进行测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[root@VM_230113centos ~]# curl -I -H 'host:7u2q8y.com3.z0.glb.qiniucdn.com' http://src.com.z0.glb.qiniudns.com/M-_WoLgVqduR7VFfrhnmWwL8YgA=/luI-
lideKDodDAYi55H6D2HV6HcO/000000.ts
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Sat, 02 Jan 2016 11:39:23 GMT
Content-Type: video/mp2t
Content-Length: 673416
Connection: keep-alive
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 2592000
Cache-Control: public, max-age=31536000
Content-Disposition: inline; filename="000000.ts"
Content-Transfer-Encoding: binary
Etag: "Fs–ekImTQ2ZMjO_MdTeTN0Yew_5"
Last-Modified: Sun, 22 Nov 2015 06:28:55 GMT
X-Log: mc.g;IO:1
X-Reqid: iToAAMIdW1cTmSUU
X-Qiniu-Zone: 0
[root@VM_230113centos ~]# curl -I -H 'host:7u2q8y.com3.z0.glb.qiniucdn.com' http://211.161.127.147/M-_WoLgVqduR7VFfrhnmWwL8YgA=/luI-lideKDodDAYi55H6D2HV6HcO/000000.ts
HTTP/1.1 200 OK
Server: NWS_Appimg_HY
Connection: keep-alive
Date: Sat, 02 Jan 2016 11:45:50 GMT
Cache-Control: max-age=31536000
Expires: Sun, 01 Jan 2017 11:45:50 GMT
Last-Modified: Sun, 22 Nov 2015 06:28:55 GMT
Content-Type: video/mp2t
Content-Length: 673416
X-Cache-Lookup: Hit From Disktank
Access-Control-Allow-Origin: *
Accept-Ranges: bytes
Etag: "Fs–ekImTQ2ZMjO_MdTeTN0Yew_5"
X-ReqId: ATUAABGu8oBVjyUU
Content-Disposition: inline; filename="000000.ts"

其中:

7u2q8y.com3.z0.glb.qiniucdn.com 为用户的域名

src.com.z0.glb.qiniudns.com 为源站或者回源host地址,也可以为ip地址形式

211.161.127.147 为域名接入cdn之后解析到的cdn节点ip地址

通过验证源站url和cdn节点ip地址后的curl反馈可以判断出到底是源站有问题,还是cdn节点有问题

作用8 使用curl抓取返回时间

判断网页访问各耗时时间

1
2
3
4
5
6
7
curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://www.itts-union.com
This command will output this:
[root@VM_117_214_centos ~]#  curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://www.itts-union.com/1326.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 74470    0 74470    0     0  52368      0 --:--:--  0:00:01 --:--:--  116k
Connect: 0.799 TTFB: 1.309 Total time: 1.422

其他的不常见用法,仅做了解

1、抓取页面内容到一个文件中

1
[root@www.itts-union.com]# curl -o home.html  http://www.itts-union.com

 

2、模拟表单信息,模拟登录,保存cookie信息

1
[root@www.itts-union.com]# curl -c ./cookie_c.txt -F log=aaaa -F pwd=****** http://www.itts-union.com/wp-login.php

 

3、使用cookie文件

1
[root@www.itts-union.com]# curl -b ./cookie_c.txt  http://www.itts-union.com/wp-admin

 

4、显示下载进度条

1
2
[root@VM_117_214_centos ~]# curl -o smokeping.tar.gz http://oss.oetiker.ch/smokeping/pub/smokeping-2.6.9.tar.gz -#
######################################################################## 100.0%

 

5、通过ftp下载文件

1
2
3
4
[zhangy@www.itts-union.com ~]$ curl -u 用户名:密码 -O http://www.itts-union.com/demo/curtain/bbstudy_files/style.css
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  Dload  Upload   Total   Spent    Left  Speed
  101  1934  101  1934    0     0   3184      0 –:–:– –:–:– –:–:–  7136

  

或者用下面的方式

[[email protected] ~]$ curl -O ftp://用户名:密码@ip:port/demo/curtain/bbstudy_files/style.css

[[email protected] ~]$ curl -O ftp://用户名:密码@ip:port/demo/curtain/bbstudy_files/style.css

 

6、通过ftp上传

[[email protected] ~]$ curl -T test.sql ftp://用户名:密码@ip:port/demo/curtain/bbstudy_files/

 

7、如下载网页发现乱码,可以使用iconv转码:

1
2
3
4
5
[root@VM_117_214_centos
 ~]# curl -o smokeping.tar.gz 
http://oss.oetiker.ch/smokeping/pub/smokeping-2.6.9.tar.gz -# | iconv 
-fgb2312
######################################################################## 100.0%

 

8、验证https访问

1
2
3
curl -sv 
'https://mccdn.qcloud.com/static/img/57c45bb3997a277e5996dcf13314b365/user-40.png'
 -o/dev/null --resolve "mccdn.qcloud.com:443:221.204.60.11"


@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
注:
“The fear of the Lord is the beginning of wisdom, and the knowledge of the Holy One is wisdom.”

 

原文由博主方胜山编辑撰写,版权归博主所有。更多相关资讯请关注微信“ebookyx”原文地址 http://www.itts-union.com/2923.html 转载请注明出处!
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@