使用multi curl進(jìn)行http并發(fā)訪問(wèn)

wtkc 2014-11-22

展開(kāi)全文

curl是一款利用URL語(yǔ)法進(jìn)行文件傳輸?shù)墓ぞ撸С侄喾N協(xié)議,，包括FTP, FTPS, HTTP, HTTPS, GOPHER, TELNET等，我們既可以在命令行上使用它,，也可以利用 libcurl進(jìn)行相關(guān)編程,。相信大部分同學(xué)都應(yīng)該使用過(guò)libcurl的easy 接口,，easy接口的使用非常的簡(jiǎn)單，curl_easy_init用來(lái)初始化一個(gè)easy curl對(duì)象,，curl_easy_setopt對(duì)easy curl對(duì)象進(jìn)行相關(guān)設(shè)置,，最后curl_easy_perform執(zhí)行curl請(qǐng)求,，返回相應(yīng)結(jié)果。easy接口是阻塞的,，也就是說(shuō)必須等到上一個(gè)curl請(qǐng)求執(zhí)行完后,，下一個(gè)curl請(qǐng)求才能繼續(xù)執(zhí)行，在一般的應(yīng)用場(chǎng)合,，這種阻塞的訪問(wèn)方式是沒(méi)有問(wèn)題的,，但是當(dāng)程序需要進(jìn)行多次curl并發(fā)請(qǐng)求的時(shí)候，easy接口就無(wú)能為力了,，這個(gè)時(shí)候curl提供的multi接口就派上用場(chǎng)了,，網(wǎng)上關(guān)于libcurl的multi接口的使用資料比較少（百度出來(lái)的大部分都是php multi curl的資料），curl官網(wǎng)上貌似也只有相關(guān)函數(shù)的說(shuō)明,，有實(shí)際demo才能讓我們更快速的上手使用,，所以下面結(jié)合實(shí)際例子來(lái)講講multi curl接口的使用方法。

相比而言,，multi接口的使用會(huì)比easy 接口稍微復(fù)雜點(diǎn),，畢竟multi接口是依賴easy接口的，首先粗略的講下其使用流程：curl_multi _init初始化一個(gè)multi curl對(duì)象,，為了同時(shí)進(jìn)行多個(gè)curl的并發(fā)訪問(wèn),，我們需要初始化多個(gè)easy curl對(duì)象，使用curl_easy_setopt進(jìn)行相關(guān)設(shè)置,，然后調(diào)用curl_multi _add_handle把easy curl對(duì)象添加到multi curl對(duì)象中，添加完畢后執(zhí)行curl_multi_perform方法進(jìn)行并發(fā)的訪問(wèn),，訪問(wèn)結(jié)束后curl_multi_remove_handle移除相關(guān)easy curl對(duì)象,，curl_easy_cleanup清除easy curl對(duì)象，最后curl_multi_cleanup清除multi curl對(duì)象,。

上面的介紹只是給大家一個(gè)大概的印象,，實(shí)際使用中還有很多細(xì)節(jié)需要注意，好了,，代碼才能說(shuō)明一切,，下面的例子使用multi curl方式進(jìn)行多次http并發(fā)訪問(wèn)，并輸出訪問(wèn)結(jié)果,。

[cpp] view plain copy

#include <string>
#include <iostream>
#include <curl/curl.h>
#include <sys/time.h>
#include <unistd.h>
using namespace std;
size_t curl_writer(void *buffer, size_t size, size_t count, void * stream)
{
std::string * pStream = static_cast<std::string *>(stream);
(*pStream).append((char *)buffer, size * count);
return size * count;
};
/**
* 生成一個(gè)easy curl對(duì)象,，進(jìn)行一些簡(jiǎn)單的設(shè)置操作
*/
CURL * curl_easy_handler(const std::string & sUrl,
const std::string & sProxy,
std::string & sRsp,
unsigned int uiTimeout)
{
CURL * curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_URL, sUrl.c_str());
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1);
if (uiTimeout > 0)
{
curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, uiTimeout);
}
if (!sProxy.empty())
{
curl_easy_setopt(curl, CURLOPT_PROXY, sProxy.c_str());
}
// write function //
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_writer);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &sRsp);
return curl;
}
/**
* 使用select函數(shù)監(jiān)聽(tīng)multi curl文件描述符的狀態(tài)
* 監(jiān)聽(tīng)成功返回0，監(jiān)聽(tīng)失敗返回-1
*/
int curl_multi_select(CURLM * curl_m)
{
int ret = 0;
struct timeval timeout_tv;
fd_set fd_read;
fd_set fd_write;
fd_set fd_except;
int max_fd = -1;
// 注意這里一定要清空f(shuō)dset,curl_multi_fdset不會(huì)執(zhí)行fdset的清空操作 //
FD_ZERO(&fd_read);
FD_ZERO(&fd_write);
FD_ZERO(&fd_except);
// 設(shè)置select超時(shí)時(shí)間 //
timeout_tv.tv_sec = 1;
timeout_tv.tv_usec = 0;
// 獲取multi curl需要監(jiān)聽(tīng)的文件描述符集合 fd_set //
curl_multi_fdset(curl_m, &fd_read, &fd_write, &fd_except, &max_fd);
/**
* When max_fd returns with -1,
* you need to wait a while and then proceed and call curl_multi_perform anyway.
* How long to wait? I would suggest 100 milliseconds at least,
* but you may want to test it out in your own particular conditions to find a suitable value.
*/
if (-1 == max_fd)
{
return -1;
}
/**
* 執(zhí)行監(jiān)聽(tīng),，當(dāng)文件描述符狀態(tài)發(fā)生改變的時(shí)候返回
* 返回0,，程序調(diào)用curl_multi_perform通知curl執(zhí)行相應(yīng)操作
* 返回-1，表示select錯(cuò)誤
* 注意：即使select超時(shí)也需要返回0,，具體可以去官網(wǎng)看文檔說(shuō)明
*/
int ret_code = ::select(max_fd + 1, &fd_read, &fd_write, &fd_except, &timeout_tv);
switch(ret_code)
{
case -1:
/* select error */
ret = -1;
break;
case 0:
/* select timeout */
default:
/* one or more of curl's file descriptors say there's data to read or write*/
ret = 0;
break;
}
return ret;
}
#define MULTI_CURL_NUM 3
// 這里設(shè)置你需要訪問(wèn)的url //
std::string URL = "http://";
// 這里設(shè)置代理ip和端口 //
std::string PROXY = "ip:port";
// 這里設(shè)置超時(shí)時(shí)間 //
unsigned int TIMEOUT = 2000; /* ms */
/**
* multi curl使用demo
*/
int curl_multi_demo(int num)
{
// 初始化一個(gè)multi curl 對(duì)象 //
CURLM * curl_m = curl_multi_init();
std::string RspArray[num];
CURL * CurlArray[num];
// 設(shè)置easy curl對(duì)象并添加到multi curl對(duì)象中 //
for (int idx = 0; idx < num; ++idx)
{
CurlArray[idx] = NULL;
CurlArray[idx] = curl_easy_handler(URL, PROXY, RspArray[idx], TIMEOUT);
if (CurlArray[idx] == NULL)
{
return -1;
}
curl_multi_add_handle(curl_m, CurlArray[idx]);
}
/*
* 調(diào)用curl_multi_perform函數(shù)執(zhí)行curl請(qǐng)求
* url_multi_perform返回CURLM_CALL_MULTI_PERFORM時(shí),，表示需要繼續(xù)調(diào)用該函數(shù)直到返回值不是CURLM_CALL_MULTI_PERFORM為止
* running_handles變量返回正在處理的easy curl數(shù)量,，running_handles為0表示當(dāng)前沒(méi)有正在執(zhí)行的curl請(qǐng)求
*/
int running_handles;
while (CURLM_CALL_MULTI_PERFORM == curl_multi_perform(curl_m, &running_handles))
{
cout << running_handles << endl;
}
/**
* 為了避免循環(huán)調(diào)用curl_multi_perform產(chǎn)生的cpu持續(xù)占用的問(wèn)題，采用select來(lái)監(jiān)聽(tīng)文件描述符
*/
while (running_handles)
{
if (-1 == curl_multi_select(curl_m))
{
cerr << "select error" << endl;
break;
} else {
// select監(jiān)聽(tīng)到事件,，調(diào)用curl_multi_perform通知curl執(zhí)行相應(yīng)的操作 //
while (CURLM_CALL_MULTI_PERFORM == curl_multi_perform(curl_m, &running_handles))
{
cout << "select: " << running_handles << endl;
}
}
cout << "select: " << running_handles << endl;
}
// 輸出執(zhí)行結(jié)果 //
int msgs_left;
CURLMsg * msg;
while((msg = curl_multi_info_read(curl_m, &msgs_left)))
{
if (CURLMSG_DONE == msg->msg)
{
int idx;
for (idx = 0; idx < num; ++idx)
{
if (msg->easy_handle == CurlArray[idx]) break;
}
if (idx == num)
{
cerr << "curl not found" << endl;
} else
{
cout << "curl [" << idx << "] completed with status: "
<< msg->data.result << endl;
cout << "rsp: " << RspArray[idx] << endl;
}
}
}
// 這里要注意cleanup的順序 //
for (int idx = 0; idx < num; ++idx)
{
curl_multi_remove_handle(curl_m, CurlArray[idx]);
}
for (int idx = 0; idx < num; ++idx)
{
curl_easy_cleanup(CurlArray[idx]);
}
curl_multi_cleanup(curl_m);
return 0;
}
/**
* easy curl使用demo
*/
int curl_easy_demo(int num)
{
std::string RspArray[num];
for (int idx = 0; idx < num; ++idx)
{
CURL * curl = curl_easy_handler(URL, PROXY, RspArray[idx], TIMEOUT);
CURLcode code = curl_easy_perform(curl);
cout << "curl [" << idx << "] completed with status: "
<< code << endl;
cout << "rsp: " << RspArray[idx] << endl;
// clear handle //
curl_easy_cleanup(curl);
}
return 0;
}
#define USE_MULTI_CURL
struct timeval begin_tv, end_tv;
int main(int argc, char * argv[])
{
if (argc < 2)
{
return -1;
}
int num = atoi(argv[1]);
// 獲取開(kāi)始時(shí)間 //
gettimeofday(&begin_tv, NULL);
#ifdef USE_MULTI_CURL
// 使用multi接口進(jìn)行訪問(wèn) //
curl_multi_demo(num);
#else
// 使用easy接口進(jìn)行訪問(wèn) //
curl_easy_demo(num);
#endif
// 獲取結(jié)束時(shí)間 //
struct timeval end_tv;
gettimeofday(&end_tv, NULL);
// 計(jì)算執(zhí)行延時(shí)并輸出,，用于比較 //
int eclapsed = (end_tv.tv_sec - begin_tv.tv_sec) * 1000 +
(end_tv.tv_usec - begin_tv.tv_usec) / 1000;
cout << "eclapsed time:" << eclapsed << "ms" << endl;
return 0;
}

上面的代碼在關(guān)鍵位置都做了詳細(xì)的注釋，相信應(yīng)該不難看懂,。

本站是提供個(gè)人知識(shí)管理的網(wǎng)絡(luò)存儲(chǔ)空間,，所有內(nèi)容均由用戶發(fā)布，不代表本站觀點(diǎn),。請(qǐng)注意甄別內(nèi)容中的聯(lián)系方式,、誘導(dǎo)購(gòu)買等信息，謹(jǐn)防詐騙,。如發(fā)現(xiàn)有害或侵權(quán)內(nèi)容,，請(qǐng)點(diǎn)擊一鍵舉報(bào)。

轉(zhuǎn)藏 分享

QQ空間 QQ好友新浪微博微信

獻(xiàn)花（0） +1

來(lái)自： wtkc > 《libcurl》

舉報(bào)/認(rèn)領(lǐng)