Java广度优先爬虫示例(抓取复旦新闻信息) - Hi_Amos
- - 博客园_首页这个爬虫是近半个月前学习爬虫技术的一个小例子,比较简单,怕时间久了会忘,这里简单总结一下.主要用到的外部Jar包有HttpClient4.3.4,HtmlParser2.1,使用的开发工具(IDE)为intelij 13.1,Jar包管理工具为Maven,不习惯用intelij的同学,也可以使用eclipse新建一个项目..
Java广度优先爬虫示例(抓取复旦新闻信息) - Hi_Amos
标签：
java
爬虫
复旦
| 发表时间：2014-07-23 01:34 | 作者：Hi_Amos
出处：
一.使用的技术
这个爬虫是近半个月前学习爬虫技术的一个小例子,比较简单,怕时间久了会忘,这里简单总结一下.主要用到的外部Jar包有HttpClient4.3.4,HtmlParser2.1,使用的开发工具(IDE)为intelij 13.1,Jar包管理工具为Maven,不习惯用intelij的同学,也可以使用eclipse新建一个项目.
二.爬虫基本知识
1.什么是网络爬虫?(爬虫的基本原理)
网络爬虫,拆开来讲, 网络即指互联网,互联网就像一个蜘蛛网一样, 爬虫就像是蜘蛛一样可以到处爬来爬去,把爬来的数据再进行加工处理.
百科上的解释: 网络爬虫（又被称为网页蜘蛛，网络机器人，在FOAF社区中间，更经常的称为网页追逐者），是一种按照一定的规则，自动的抓取万维网信息的程序或者脚本。另外一些不常使用的名字还有蚂蚁，自动索引，模拟程序或者蠕虫。
基本原理:传统爬虫从一个或若干初始网页的URL开始，获得初始网页上的URL，在抓取网页的过程中，不断从当前页面上抽取新的URL放入队列,直到满足系统的一定停止条件，流程图所示。聚焦爬虫的工作流程较为复杂，需要根据一定的网页分析算法过滤与主题无关的链接，保留有用的链接并将其放入等待抓取的URL队列。然后，它将根据一定的搜索策略从队列中选择下一步要抓取的网页URL，并重复上述过程，直到达到系统的某一条件时停止
2.常用的爬虫策略有哪些?
网页的抓取策略可以分为深度优先、广度优先和最佳优先三种。深度优先在很多情况下会导致爬虫的陷入(trapped)问题，目前常见的是广度优先和最佳优先方法。
2.1广度优先(Width-First)
广度优先遍历是连通图的一种遍历策略。因为它的思想是从一个顶点V0开始，辐射状地优先遍历其周围较广的区域,故得名.
其基本思想:
1)、从图中某个顶点V0出发，并访问此顶点；
2)、从V0出发，访问V0的各个未曾访问的邻接点W1，W2，…,Wk;然后,依次从W1,W2,…,Wk出发访问各自未被访问的邻接点；
3)、重复步骤2，直到全部顶点都被访问为止。
如下图所示:
2.2深度优先(Depth-First)
假设初始状态是图中所有顶点都未被访问，则深度优先搜索方法的步骤是：
1）选取图中某一顶点Vi为出发点，访问并标记该顶点；
2）以Vi为当前顶点，依次搜索Vi的每个邻接点Vj，若Vj未被访问过，则访问和标记邻接点Vj，若Vj已被访问过，则搜索Vi的下一个邻接点；
3）以Vj为当前顶点，重复步骤2，直到图中和Vi有路径相通的顶点都被访问为止；
4）若图中尚有顶点未被访问过（非连通的情况下），则可任取图中的一个未被访问的顶点作为出发点，重复上述过程，直至图中所有顶点都被访问。
下面以一个有向图和一个无向图为例:
广度和深度和区别:
广度优先遍历是以层为顺序，将某一层上的所有节点都搜索到了之后才向下一层搜索；而深度优先遍历是将某一条枝桠上的所有节点都搜索到了之后，才转向搜索另一条枝桠上的所有节点。
2.3 最佳优先搜索
最佳优先搜索策略按照一定的网页分析算法，预测候选URL与目标网页的相似度，或与主题的相关性，并选取评价最好的一个或几个URL进行抓取。它只访问经过网页分析算法预测为“有用”的网页。这种搜索适合暗网数据的爬取,只要符合要求的内容.
3.本文爬虫示例图
本文介绍的例子是抓取新闻类的信息,因为一般新闻类的信息,重要的和时间近的都会放在首页,处在网络层中比较深的信息的重要性一般将逐级降低,所以广度优先算法更适合,下图是本文将要抓取的网页结构图:
三.广度优先爬虫示例
1.需求:抓取复旦新闻信息(只抓取100个网页信息)
这里只抓取100条信息,并用url必须以new.fudan.edu.cn开头.
2.代码实现
使用maven引入外部jar包:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.4</version>
</dependency>
<dependency>
<groupId>org.htmlparser</groupId>
<artifactId>htmlparser</artifactId>
<version>2.1</version>
</dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.4</version>
</dependency>
<dependency>
<groupId>org.htmlparser</groupId>
<artifactId>htmlparser</artifactId>
<version>2.1</version>
</dependency>
程序主入口:
package com.amos.crawl;
import java.util.Set;
/**
* Created by amosli on 14-7-10.
*/
public class MyCrawler {
/**
* 使用种子初始化URL队列
*
* @param seeds
*/
private void initCrawlerWithSeeds(String[] seeds) {
for (int i = 0; i < seeds.length; i++) {
LinkQueue.addUnvisitedUrl(seeds[i]);
}
}
public void crawling(String[] seeds) {
//定义过滤器,提取以http://news.fudan.edu.cn/的链接
LinkFilter filter = new LinkFilter() {
@Override
public boolean accept(String url) {
if (url.startsWith("http://news.fudan.edu.cn")) {
return true;
}
return false;
}
};
//初始化URL队列
initCrawlerWithSeeds(seeds);
int count=0;
//循环条件:待抓取的链接不为空抓取的网页最多100条
while (!LinkQueue.isUnvisitedUrlsEmpty() && LinkQueue.getVisitedUrlNum() <= 100) {
System.out.println("count:"+(++count));
//附头URL出队列
String visitURL = (String) LinkQueue.unVisitedUrlDeQueue();
DownLoadFile downloader = new DownLoadFile();
//下载网页
downloader.downloadFile(visitURL);
//该URL放入怩访问的URL中
LinkQueue.addVisitedUrl(visitURL);
//提取出下载网页中的URL
Set<String> links = HtmlParserTool.extractLinks(visitURL, filter);
//新的未访问的URL入列
for (String link : links) {
System.out.println("link:"+link);
LinkQueue.addUnvisitedUrl(link);
}
}
}
public static void main(String args[]) {
//程序入口
MyCrawler myCrawler = new MyCrawler();
myCrawler.crawling(new String[]{"http://news.fudan.edu.cn/news/"});
}
}
import java.util.Set;
/**
* Created by amosli on 14-7-10.
*/
public class MyCrawler {
/**
* 使用种子初始化URL队列
*
* @param seeds
*/
private void initCrawlerWithSeeds(String[] seeds) {
for (int i = 0; i < seeds.length; i++) {
LinkQueue.addUnvisitedUrl(seeds[i]);
}
}
public void crawling(String[] seeds) {
//定义过滤器,提取以http://news.fudan.edu.cn/的链接
LinkFilter filter = new LinkFilter() {
@Override
public boolean accept(String url) {
if (url.startsWith("http://news.fudan.edu.cn")) {
return true;
}
return false;
}
};
//初始化URL队列
initCrawlerWithSeeds(seeds);
int count=0;
//循环条件:待抓取的链接不为空抓取的网页最多100条
while (!LinkQueue.isUnvisitedUrlsEmpty() && LinkQueue.getVisitedUrlNum() <= 100) {
System.out.println("count:"+(++count));
//附头URL出队列
String visitURL = (String) LinkQueue.unVisitedUrlDeQueue();
DownLoadFile downloader = new DownLoadFile();
//下载网页
downloader.downloadFile(visitURL);
//该URL放入怩访问的URL中
LinkQueue.addVisitedUrl(visitURL);
//提取出下载网页中的URL
Set<String> links = HtmlParserTool.extractLinks(visitURL, filter);
//新的未访问的URL入列
for (String link : links) {
System.out.println("link:"+link);
LinkQueue.addUnvisitedUrl(link);
}
}
}
public static void main(String args[]) {
//程序入口
MyCrawler myCrawler = new MyCrawler();
myCrawler.crawling(new String[]{"http://news.fudan.edu.cn/news/"});
}
}
工具类:Tools.java
package com.amos.tool;
import java.io.*;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.UnknownHostException;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Locale;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLException;
import org.apache.http.*;
import org.apache.http.client.CircularRedirectException;
import org.apache.http.client.CookieStore;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.client.RedirectStrategy;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpHead;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.client.utils.URIUtils;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.conn.HttpClientConnectionManager;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.SSLContextBuilder;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.cookie.Cookie;
import org.apache.http.impl.client.*;
import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
import org.apache.http.impl.cookie.BasicClientCookie;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.Args;
import org.apache.http.util.Asserts;
import org.apache.http.util.TextUtils;
import org.omg.CORBA.Request;
/**
* Created by amosli on 14-6-25.
*/
public class Tools {
/**
* 写文件到本地
*
* @param httpEntity
* @param filename
*/
public static void saveToLocal(HttpEntity httpEntity, String filename) {
try {
File dir = new File(Configuration.FILEDIR);
if (!dir.isDirectory()) {
dir.mkdir();
}
File file = new File(dir.getAbsolutePath() + "/" + filename);
FileOutputStream fileOutputStream = new FileOutputStream(file);
InputStream inputStream = httpEntity.getContent();
byte[] bytes = new byte[1024];
int length = 0;
while ((length = inputStream.read(bytes)) > 0) {
fileOutputStream.write(bytes, 0, length);
}
inputStream.close();
fileOutputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* 写文件到本地
*
* @param bytes
* @param filename
*/
public static void saveToLocalByBytes(byte[] bytes, String filename) {
try {
File dir = new File(Configuration.FILEDIR);
if (!dir.isDirectory()) {
dir.mkdir();
}
File file = new File(dir.getAbsolutePath() + "/" + filename);
FileOutputStream fileOutputStream = new FileOutputStream(file);
fileOutputStream.write(bytes);
//fileOutputStream.write(bytes, 0, bytes.length);
fileOutputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* 输出
* @param string
*/
public static void println(String string){
System.out.println("string:"+string);
}
/**
* 输出
* @param string
*/
public static void printlnerr(String string){
System.err.println("string:"+string);
}
/**
* 使用ssl通道并设置请求重试处理
* @return
*/
public static CloseableHttpClient createSSLClientDefault() {
try {
SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {
//信任所有
public boolean isTrusted(X509Certificate[] chain,String authType) throws CertificateException {
return true;
}
}).build();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext);
//设置请求重试处理,重试机制,这里如果请求失败会重试5次
HttpRequestRetryHandler retryHandler = new HttpRequestRetryHandler() {
@Override
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
if (executionCount >= 5) {
// Do not retry if over max retry count
return false;
}
if (exception instanceof InterruptedIOException) {
// Timeout
return false;
}
if (exception instanceof UnknownHostException) {
// Unknown host
return false;
}
if (exception instanceof ConnectTimeoutException) {
// Connection refused
return false;
}
if (exception instanceof SSLException) {
// SSL handshake exception
return false;
}
HttpClientContext clientContext = HttpClientContext.adapt(context);
HttpRequest request = clientContext.getRequest();
boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
if (idempotent) {
// Retry if the request is considered idempotent
return true;
}
return false;
}
};
//请求参数设置,设置请求超时时间为20秒,连接超时为10秒,不允许循环重定向
RequestConfig requestConfig = RequestConfig.custom()
.setConnectionRequestTimeout(20000).setConnectTimeout(20000)
.setCircularRedirectsAllowed(false)
.build();
Cookie cookie ;
return HttpClients.custom().setSSLSocketFactory(sslsf)
.setUserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36")
.setMaxConnPerRoute(25).setMaxConnPerRoute(256)
.setRetryHandler(retryHandler)
.setRedirectStrategy(new SelfRedirectStrategy())
.setDefaultRequestConfig(requestConfig)
.build();
} catch (KeyManagementException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (KeyStoreException e) {
e.printStackTrace();
}
return HttpClients.createDefault();
}
/**
* 带cookiestore
* @param cookieStore
* @return
*/
public static CloseableHttpClient createSSLClientDefaultWithCookie(CookieStore cookieStore) {
try {
SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {
//信任所有
public boolean isTrusted(X509Certificate[] chain,String authType) throws CertificateException {
return true;
}
}).build();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext);
//设置请求重试处理,重试机制,这里如果请求失败会重试5次
HttpRequestRetryHandler retryHandler = new HttpRequestRetryHandler() {
@Override
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
if (executionCount >= 5) {
// Do not retry if over max retry count
return false;
}
if (exception instanceof InterruptedIOException) {
// Timeout
return false;
}
if (exception instanceof UnknownHostException) {
// Unknown host
return false;
}
if (exception instanceof ConnectTimeoutException) {
// Connection refused
return false;
}
if (exception instanceof SSLException) {
// SSL handshake exception
return false;
}
HttpClientContext clientContext = HttpClientContext.adapt(context);
HttpRequest request = clientContext.getRequest();
boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
if (idempotent) {
// Retry if the request is considered idempotent
return true;
}
return false;
}
};
//请求参数设置,设置请求超时时间为20秒,连接超时为10秒,不允许循环重定向
RequestConfig requestConfig = RequestConfig.custom()
.setConnectionRequestTimeout(20000).setConnectTimeout(20000)
.setCircularRedirectsAllowed(false)
.build();
return HttpClients.custom().setSSLSocketFactory(sslsf)
.setUserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36")
.setMaxConnPerRoute(25).setMaxConnPerRoute(256)
.setRetryHandler(retryHandler)
.setRedirectStrategy(new SelfRedirectStrategy())
.setDefaultRequestConfig(requestConfig)
.setDefaultCookieStore(cookieStore)
.build();
} catch (KeyManagementException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (KeyStoreException e) {
e.printStackTrace();
}
return HttpClients.createDefault();
}
}
import java.io.*;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.UnknownHostException;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Locale;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLException;
import org.apache.http.*;
import org.apache.http.client.CircularRedirectException;
import org.apache.http.client.CookieStore;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.client.RedirectStrategy;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpHead;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.client.utils.URIUtils;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.conn.HttpClientConnectionManager;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.SSLContextBuilder;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.cookie.Cookie;
import org.apache.http.impl.client.*;
import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
import org.apache.http.impl.cookie.BasicClientCookie;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.Args;
import org.apache.http.util.Asserts;
import org.apache.http.util.TextUtils;
import org.omg.CORBA.Request;
/**
* Created by amosli on 14-6-25.
*/
public class Tools {
/**
* 写文件到本地
*
* @param httpEntity
* @param filename
*/
public static void saveToLocal(HttpEntity httpEntity, String filename) {
try {
File dir = new File(Configuration.FILEDIR);
if (!dir.isDirectory()) {
dir.mkdir();
}
File file = new File(dir.getAbsolutePath() + "/" + filename);
FileOutputStream fileOutputStream = new FileOutputStream(file);
InputStream inputStream = httpEntity.getContent();
byte[] bytes = new byte[1024];
int length = 0;
while ((length = inputStream.read(bytes)) > 0) {
fileOutputStream.write(bytes, 0, length);
}
inputStream.close();
fileOutputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* 写文件到本地
*
* @param bytes
* @param filename
*/
public static void saveToLocalByBytes(byte[] bytes, String filename) {
try {
File dir = new File(Configuration.FILEDIR);
if (!dir.isDirectory()) {
dir.mkdir();
}
File file = new File(dir.getAbsolutePath() + "/" + filename);
FileOutputStream fileOutputStream = new FileOutputStream(file);
fileOutputStream.write(bytes);
//fileOutputStream.write(bytes, 0, bytes.length);
fileOutputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* 输出
* @param string
*/
public static void println(String string){
System.out.println("string:"+string);
}
/**
* 输出
* @param string
*/
public static void printlnerr(String string){
System.err.println("string:"+string);
}
/**
* 使用ssl通道并设置请求重试处理
* @return
*/
public static CloseableHttpClient createSSLClientDefault() {
try {
SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {
//信任所有
public boolean isTrusted(X509Certificate[] chain,String authType) throws CertificateException {
return true;
}
}).build();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext);
//设置请求重试处理,重试机制,这里如果请求失败会重试5次
HttpRequestRetryHandler retryHandler = new HttpRequestRetryHandler() {
@Override
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
if (executionCount >= 5) {
// Do not retry if over max retry count
return false;
}
if (exception instanceof InterruptedIOException) {
// Timeout
return false;
}
if (exception instanceof UnknownHostException) {
// Unknown host
return false;
}
if (exception instanceof ConnectTimeoutException) {
// Connection refused
return false;
}
if (exception instanceof SSLException) {
// SSL handshake exception
return false;
}
HttpClientContext clientContext = HttpClientContext.adapt(context);
HttpRequest request = clientContext.getRequest();
boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
if (idempotent) {
// Retry if the request is considered idempotent
return true;
}
return false;
}
};
//请求参数设置,设置请求超时时间为20秒,连接超时为10秒,不允许循环重定向
RequestConfig requestConfig = RequestConfig.custom()
.setConnectionRequestTimeout(20000).setConnectTimeout(20000)
.setCircularRedirectsAllowed(false)
.build();
Cookie cookie ;
return HttpClients.custom().setSSLSocketFactory(sslsf)
.setUserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36")
.setMaxConnPerRoute(25).setMaxConnPerRoute(256)
.setRetryHandler(retryHandler)
.setRedirectStrategy(new SelfRedirectStrategy())
.setDefaultRequestConfig(requestConfig)
.build();
} catch (KeyManagementException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (KeyStoreException e) {
e.printStackTrace();
}
return HttpClients.createDefault();
}
/**
* 带cookiestore
* @param cookieStore
* @return
*/
public static CloseableHttpClient createSSLClientDefaultWithCookie(CookieStore cookieStore) {
try {
SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {
//信任所有
public boolean isTrusted(X509Certificate[] chain,String authType) throws CertificateException {
return true;
}
}).build();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext);
//设置请求重试处理,重试机制,这里如果请求失败会重试5次
HttpRequestRetryHandler retryHandler = new HttpRequestRetryHandler() {
@Override
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
if (executionCount >= 5) {
// Do not retry if over max retry count
return false;
}
if (exception instanceof InterruptedIOException) {
// Timeout
return false;
}
if (exception instanceof UnknownHostException) {
// Unknown host
return false;
}
if (exception instanceof ConnectTimeoutException) {
// Connection refused
return false;
}
if (exception instanceof SSLException) {
// SSL handshake exception
return false;
}
HttpClientContext clientContext = HttpClientContext.adapt(context);
HttpRequest request = clientContext.getRequest();
boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
if (idempotent) {
// Retry if the request is considered idempotent
return true;
}
return false;
}
};
//请求参数设置,设置请求超时时间为20秒,连接超时为10秒,不允许循环重定向
RequestConfig requestConfig = RequestConfig.custom()
.setConnectionRequestTimeout(20000).setConnectTimeout(20000)
.setCircularRedirectsAllowed(false)
.build();
return HttpClients.custom().setSSLSocketFactory(sslsf)
.setUserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36")
.setMaxConnPerRoute(25).setMaxConnPerRoute(256)
.setRetryHandler(retryHandler)
.setRedirectStrategy(new SelfRedirectStrategy())
.setDefaultRequestConfig(requestConfig)
.setDefaultCookieStore(cookieStore)
.build();
} catch (KeyManagementException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (KeyStoreException e) {
e.printStackTrace();
}
return HttpClients.createDefault();
}
}
将网页写入到本地的下载类: DownLoadFile.java
package com.amos.crawl;
import com.amos.tool.Configuration;
import com.amos.tool.Tools;
import org.apache.http.*;
import org.apache.http.client.HttpClient;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.impl.client.AutoRetryHttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.protocol.HttpContext;
import javax.net.ssl.SSLException;
import java.io.*;
import java.net.UnknownHostException;
/**
* Created by amosli on 14-7-9.
*/
public class DownLoadFile {
public String getFileNameByUrl(String url, String contentType) {
//移除http http://
url = url.contains("http://") ? url.substring(7) : url.substring(8);
//text/html类型
if (url.contains(".html")) {
url = url.replaceAll("[\\?/:*|<>\"]", "_");
} else if (contentType.indexOf("html") != -1) {
url = url.replaceAll("[\\?/:*|<>\"]", "_") + ".html";
} else {
url = url.replaceAll("[\\?/:*|<>\"]", "_") + "." + contentType.substring(contentType.lastIndexOf("/") + 1);
}
return url;
}
/**
* 将网页写入到本地
* @param data
* @param filePath
*/
private void saveToLocal(byte[] data, String filePath) {
try {
DataOutputStream out = new DataOutputStream(new FileOutputStream(new File(filePath)));
for(int i=0;i<data.length;i++){
out.write(data[i]);
}
out.flush();
out.close();
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* 写文件到本地
*
* @param httpEntity
* @param filename
*/
public static void saveToLocal(HttpEntity httpEntity, String filename) {
try {
File dir = new File(Configuration.FILEDIR);
if (!dir.isDirectory()) {
dir.mkdir();
}
File file = new File(dir.getAbsolutePath() + "/" + filename);
FileOutputStream fileOutputStream = new FileOutputStream(file);
InputStream inputStream = httpEntity.getContent();
if (!file.exists()) {
file.createNewFile();
}
byte[] bytes = new byte[1024];
int length = 0;
while ((length = inputStream.read(bytes)) > 0) {
fileOutputStream.write(bytes, 0, length);
}
inputStream.close();
fileOutputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
public String downloadFile(String url) {
//文件路径
String filePath=null;
//1.生成HttpClient对象并设置参数
HttpClient httpClient = Tools.createSSLClientDefault();
//2.HttpGet对象并设置参数
HttpGet httpGet = new HttpGet(url);
//设置get请求超时5s
//方法1
//httpGet.getParams().setParameter("connectTimeout",5000);
//方法2
RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(5000).build();
httpGet.setConfig(requestConfig);
try {
HttpResponse httpResponse = httpClient.execute(httpGet);
int statusCode = httpResponse.getStatusLine().getStatusCode();
if(statusCode!= HttpStatus.SC_OK){
System.err.println("Method failed:"+httpResponse.getStatusLine());
filePath=null;
}
filePath=getFileNameByUrl(url,httpResponse.getEntity().getContentType().getValue());
saveToLocal(httpResponse.getEntity(),filePath);
} catch (Exception e) {
e.printStackTrace();
}
return filePath;
}
public static void main(String args[]) throws IOException {
String url = "http://websearch.fudan.edu.cn/search_dep.html";
HttpClient httpClient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
HttpResponse httpResponse = httpClient.execute(httpGet);
Header contentType = httpResponse.getEntity().getContentType();
System.out.println("name:" + contentType.getName() + "value:" + contentType.getValue());
System.out.println(new DownLoadFile().getFileNameByUrl(url, contentType.getValue()));
}
}
import com.amos.tool.Configuration;
import com.amos.tool.Tools;
import org.apache.http.*;
import org.apache.http.client.HttpClient;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.impl.client.AutoRetryHttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.protocol.HttpContext;
import javax.net.ssl.SSLException;
import java.io.*;
import java.net.UnknownHostException;
/**
* Created by amosli on 14-7-9.
*/
public class DownLoadFile {
public String getFileNameByUrl(String url, String contentType) {
//移除http http://
url = url.contains("http://") ? url.substring(7) : url.substring(8);
//text/html类型
if (url.contains(".html")) {
url = url.replaceAll("[\\?/:*|<>\"]", "_");
} else if (contentType.indexOf("html") != -1) {
url = url.replaceAll("[\\?/:*|<>\"]", "_") + ".html";
} else {
url = url.replaceAll("[\\?/:*|<>\"]", "_") + "." + contentType.substring(contentType.lastIndexOf("/") + 1);
}
return url;
}
/**
* 将网页写入到本地
* @param data
* @param filePath
*/
private void saveToLocal(byte[] data, String filePath) {
try {
DataOutputStream out = new DataOutputStream(new FileOutputStream(new File(filePath)));
for(int i=0;i<data.length;i++){
out.write(data[i]);
}
out.flush();
out.close();
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* 写文件到本地
*
* @param httpEntity
* @param filename
*/
public static void saveToLocal(HttpEntity httpEntity, String filename) {
try {
File dir = new File(Configuration.FILEDIR);
if (!dir.isDirectory()) {
dir.mkdir();
}
File file = new File(dir.getAbsolutePath() + "/" + filename);
FileOutputStream fileOutputStream = new FileOutputStream(file);
InputStream inputStream = httpEntity.getContent();
if (!file.exists()) {
file.createNewFile();
}
byte[] bytes = new byte[1024];
int length = 0;
while ((length = inputStream.read(bytes)) > 0) {
fileOutputStream.write(bytes, 0, length);
}
inputStream.close();
fileOutputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
public String downloadFile(String url) {
//文件路径
String filePath=null;
//1.生成HttpClient对象并设置参数
HttpClient httpClient = Tools.createSSLClientDefault();
//2.HttpGet对象并设置参数
HttpGet httpGet = new HttpGet(url);
//设置get请求超时5s
//方法1
//httpGet.getParams().setParameter("connectTimeout",5000);
//方法2
RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(5000).build();
httpGet.setConfig(requestConfig);
try {
HttpResponse httpResponse = httpClient.execute(httpGet);
int statusCode = httpResponse.getStatusLine().getStatusCode();
if(statusCode!= HttpStatus.SC_OK){
System.err.println("Method failed:"+httpResponse.getStatusLine());
filePath=null;
}
filePath=getFileNameByUrl(url,httpResponse.getEntity().getContentType().getValue());
saveToLocal(httpResponse.getEntity(),filePath);
} catch (Exception e) {
e.printStackTrace();
}
return filePath;
}
public static void main(String args[]) throws IOException {
String url = "http://websearch.fudan.edu.cn/search_dep.html";
HttpClient httpClient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
HttpResponse httpResponse = httpClient.execute(httpGet);
Header contentType = httpResponse.getEntity().getContentType();
System.out.println("name:" + contentType.getName() + "value:" + contentType.getValue());
System.out.println(new DownLoadFile().getFileNameByUrl(url, contentType.getValue()));
}
}
创建一个过滤接口:LinkFilter.java
package com.amos.crawl;
/**
* Created by amosli on 14-7-10.
*/
public interface LinkFilter {
public boolean accept(String url);
}
/**
* Created by amosli on 14-7-10.
*/
public interface LinkFilter {
public boolean accept(String url);
}
使用HtmlParser的过滤url的方法:HtmlParserTool.java
package com.amos.crawl;
import org.htmlparser.Node;
import org.htmlparser.NodeFilter;
import org.htmlparser.Parser;
import org.htmlparser.filters.NodeClassFilter;
import org.htmlparser.filters.OrFilter;
import org.htmlparser.tags.LinkTag;
import org.htmlparser.util.NodeList;
import java.util.HashSet;
import java.util.Set;
/**
* Created by amosli on 14-7-10.
*/
public class HtmlParserTool {
public static Set<String> extractLinks(String url, LinkFilter filter) {
Set<String> links = new HashSet<String>();
try {
Parser parser = new Parser(url);
parser.setEncoding("GBK");
//过滤<frame>标签的filter,用来提取frame标签里的src属性
NodeFilter framFilter = new NodeFilter() {
@Override
public boolean accept(Node node) {
if (node.getText().contains("frame src=")) {
return true;
} else {
return false;
}
}
};
//OrFilter来设置过滤<a>标签和<frame>标签
OrFilter linkFilter = new OrFilter(new NodeClassFilter(LinkTag.class), framFilter);
//得到所有经过过滤的标签
NodeList list = parser.extractAllNodesThatMatch(linkFilter);
for (int i = 0; i < list.size(); i++) {
Node tag = list.elementAt(i);
if (tag instanceof LinkTag) {
tag = (LinkTag) tag;
String linkURL = ((LinkTag) tag).getLink();
//如果符合条件那么将url添加进去
if (filter.accept(linkURL)) {
links.add(linkURL);
}
} else {//frame 标签
//frmae里src属性的链接,如<frame src="test.html" />
String frame = tag.getText();
int start = frame.indexOf("src=");
frame = frame.substring(start);
int end = frame.indexOf(" ");
if (end == -1) {
end = frame.indexOf(">");
}
String frameUrl = frame.substring(5, end - 1);
if (filter.accept(frameUrl)) {
links.add(frameUrl);
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
return links;
}
}
import org.htmlparser.Node;
import org.htmlparser.NodeFilter;
import org.htmlparser.Parser;
import org.htmlparser.filters.NodeClassFilter;
import org.htmlparser.filters.OrFilter;
import org.htmlparser.tags.LinkTag;
import org.htmlparser.util.NodeList;
import java.util.HashSet;
import java.util.Set;
/**
* Created by amosli on 14-7-10.
*/
public class HtmlParserTool {
public static Set<String> extractLinks(String url, LinkFilter filter) {
Set<String> links = new HashSet<String>();
try {
Parser parser = new Parser(url);
parser.setEncoding("GBK");
//过滤<frame>标签的filter,用来提取frame标签里的src属性
NodeFilter framFilter = new NodeFilter() {
@Override
public boolean accept(Node node) {
if (node.getText().contains("frame src=")) {
return true;
} else {
return false;
}
}
};
//OrFilter来设置过滤<a>标签和<frame>标签
OrFilter linkFilter = new OrFilter(new NodeClassFilter(LinkTag.class), framFilter);
//得到所有经过过滤的标签
NodeList list = parser.extractAllNodesThatMatch(linkFilter);
for (int i = 0; i < list.size(); i++) {
Node tag = list.elementAt(i);
if (tag instanceof LinkTag) {
tag = (LinkTag) tag;
String linkURL = ((LinkTag) tag).getLink();
//如果符合条件那么将url添加进去
if (filter.accept(linkURL)) {
links.add(linkURL);
}
} else {//frame 标签
//frmae里src属性的链接,如<frame src="test.html" />
String frame = tag.getText();
int start = frame.indexOf("src=");
frame = frame.substring(start);
int end = frame.indexOf(" ");
if (end == -1) {
end = frame.indexOf(">");
}
String frameUrl = frame.substring(5, end - 1);
if (filter.accept(frameUrl)) {
links.add(frameUrl);
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
return links;
}
}
管理网页url的实现队列: Queue.java
package com.amos.crawl;
import java.util.LinkedList;
/**
* Created by amosli on 14-7-9.
*/
public class Queue {
//使用链表实现队列
private LinkedList queueList = new LinkedList();
//入队列
public void enQueue(Object object) {
queueList.addLast(object);
}
//出队列
public Object deQueue() {
return queueList.removeFirst();
}
//判断队列是否为空
public boolean isQueueEmpty() {
return queueList.isEmpty();
}
//判断队列是否包含ject元素..
public boolean contains(Object object) {
return queueList.contains(object);
}
//判断队列是否为空
public boolean empty() {
return queueList.isEmpty();
}
}
import java.util.LinkedList;
/**
* Created by amosli on 14-7-9.
*/
public class Queue {
//使用链表实现队列
private LinkedList queueList = new LinkedList();
//入队列
public void enQueue(Object object) {
queueList.addLast(object);
}
//出队列
public Object deQueue() {
return queueList.removeFirst();
}
//判断队列是否为空
public boolean isQueueEmpty() {
return queueList.isEmpty();
}
//判断队列是否包含ject元素..
public boolean contains(Object object) {
return queueList.contains(object);
}
//判断队列是否为空
public boolean empty() {
return queueList.isEmpty();
}
}
网页链接进出队列的管理:LinkQueue.java
package com.amos.crawl;
import java.util.HashSet;
import java.util.Set;
/**
* Created by amosli on 14-7-9.
*/
public class LinkQueue {
//已经访问的队列
private static Set visitedUrl = new HashSet();
//未访问的队列
private static Queue unVisitedUrl = new Queue();
//获得URL队列
public static Queue getUnVisitedUrl() {
return unVisitedUrl;
}
public static Set getVisitedUrl() {
return visitedUrl;
}
//添加到访问过的URL队列中
public static void addVisitedUrl(String url) {
visitedUrl.add(url);
}
//删除已经访问过的URL
public static void removeVisitedUrl(String url){
visitedUrl.remove(url);
}
//未访问的URL出队列
public static Object unVisitedUrlDeQueue(){
return unVisitedUrl.deQueue();
}
//保证每个URL只被访问一次,url不能为空,同时已经访问的URL队列中不能包含该url,而且因为已经出队列了所未访问的URL队列中也不能包含该url
public static void addUnvisitedUrl(String url){
if(url!=null&&!url.trim().equals("")&&!visitedUrl.contains(url)&&!unVisitedUrl.contains(url))
unVisitedUrl.enQueue(url);
}
//获得已经访问过的URL的数量
public static int getVisitedUrlNum(){
return visitedUrl.size();
}
//判断未访问的URL队列中是否为空
public static boolean isUnvisitedUrlsEmpty(){
return unVisitedUrl.empty();
}
}
import java.util.HashSet;
import java.util.Set;
/**
* Created by amosli on 14-7-9.
*/
public class LinkQueue {
//已经访问的队列
private static Set visitedUrl = new HashSet();
//未访问的队列
private static Queue unVisitedUrl = new Queue();
//获得URL队列
public static Queue getUnVisitedUrl() {
return unVisitedUrl;
}
public static Set getVisitedUrl() {
return visitedUrl;
}
//添加到访问过的URL队列中
public static void addVisitedUrl(String url) {
visitedUrl.add(url);
}
//删除已经访问过的URL
public static void removeVisitedUrl(String url){
visitedUrl.remove(url);
}
//未访问的URL出队列
public static Object unVisitedUrlDeQueue(){
return unVisitedUrl.deQueue();
}
//保证每个URL只被访问一次,url不能为空,同时已经访问的URL队列中不能包含该url,而且因为已经出队列了所未访问的URL队列中也不能包含该url
public static void addUnvisitedUrl(String url){
if(url!=null&&!url.trim().equals("")&&!visitedUrl.contains(url)&&!unVisitedUrl.contains(url))
unVisitedUrl.enQueue(url);
}
//获得已经访问过的URL的数量
public static int getVisitedUrlNum(){
return visitedUrl.size();
}
//判断未访问的URL队列中是否为空
public static boolean isUnvisitedUrlsEmpty(){
return unVisitedUrl.empty();
}
}
抓取思路是:首先给出要抓取的url==>查询符合条件的url,并将其加入到队列中==>按顺序取出队列中的url,并访问之,同时取出符合条件的url==>下载队列中的url网页,即按层探索,最多限制100条数据.
3.3 截图
本文链接： Java广度优先爬虫示例(抓取复旦新闻信息)，转载请注明。
相关 [java 爬虫 复旦] 推荐：
JAVA爬虫Nutch、WebCollector的正则约束
- - CSDN博客互联网推荐文章爬虫爬取时，需要约束爬取的范围. 基本所有的爬虫都是通过正则表达式来完成这个约束. 代表"http://www.xinhuanet.com/"后加任意个任意字符（可以是0个）. 通过这个正则可以约束爬虫的爬取范围,但是这个正则并不是表示爬取新华网所有的网页. 新华网并不是只有www.xinhuanet.com这一个域名，还有很多子域名，类似:news.xinhuanet.com.
webmagic 0.3.0 发布，高性能Java爬虫框架
- - 开源中国社区最新新闻webmagic的是一个无须配置、便于二次开发的爬虫框架，它提供简单灵活的API，只需少量代码即可实现一个爬虫. 之前两个版本的主题分别是"灵活"，"方便"，而此次更新的主题是"性能". 重写了XPath解析器，替换了HtmlCleaner. 这个XPath解析器(我另外建立了一个项目 Xsoup)基于Jsoup完成，比HtmlCleaner性能好一倍以上，同时有更友好的错误提示.
微博爬虫“免登录”技巧详解及Java实现
- - SegmentFault 最新的文章本文源地址： http://www.fullstackyang.com/...，转发请注明该地址或segmentfault地址，谢谢. 一、微博一定要登录才能抓取. 目前，对于微博的爬虫，大部分是基于模拟微博账号登录的方式实现的，这种方式如果真的运营起来，实际上是一件非常头疼痛苦的事，你可能每天都过得提心吊胆，生怕新浪爸爸把你的那些账号给封了，而且现在随着实名制的落地，获得账号的渠道估计也会变得越来越少.
Java豆瓣电影爬虫——抓取电影详情和电影短评数据
- - ImportNew采集豆瓣电影数据包括电影详情页数据和电影的短评数据. 需要保存这些详情字段如导演、编剧、演员等还有图中右下方的标签. 需要保存的字段有短评所属的电影名称，每条评论的详细信息如评论人名称、评论内容等. 有了如上的需求，需要设计表，其实很简单，只需要一张电影详情表movie和一张电影短评表comments，另外还需要一张存储网页提取的超链接的记录表record.
手把手教你搭建一个基于Java的分布式爬虫系统
- - DockOne.io【编者的话】在不用爬虫框架的情况，经过多方学习，尝试实现了一个分布式爬虫系统，并且可以将数据保存到不同地方，类似MySQL、HBase等. 基于面向接口的编码思想来开发，因此这个系统具有一定的扩展性，有兴趣的朋友直接看一下代码，就能理解其设计思想，虽然代码目前来说很多地方还是比较紧耦合，但只要花些时间和精力，很多都是可抽取出来并且可配置化的.
网络爬虫
- - 四火的唠叨文章系本人原创，转载请保持完整性并注明出自 《四火的唠叨》. 最近在写一个程序，去爬热门事件和热门关键词网站上的数据. 网络爬虫也叫做网络蜘蛛，是一种互联网机器人，把需要的网页撷取下来，组织成适当格式存储. 它是搜索引擎的重要组成部分，虽然从技术实现上来说，它的难度往往要小于对于得到的网页信息的处理.
Google 图片爬虫
- - 吴良超的学习笔记这里的 Google 图片爬虫指的是爬取在 Google 上通过关键词搜索得到的图片，由于最近需要一些特定领域的图片，而且现有的数据库满足不了要求，因此就想通过 Google 搜索筛选出这些特定领域的图片，然后下载下来后再进行人工筛选. 这里采用了两种方法，区别在于是否需要解析网页端的 JS 代码.
JSOUP实现简单爬虫
- - ITeye博客这个说是简单爬虫 其实连个爬虫也算不上吧 功能太精简了.... 流程很简单: 输入几个初始的网页 然后通过JSOUP获取网页中的a标签的href的值. 接着把新得到的地址放入任务队列中. 实现中的worker是一个单线程的派发器 用于产生Parser. Parser用于完成网页的保存 网页的解析 以及入队列操作.
最全Python爬虫总结
- - CSDN博客综合推荐文章最近总是要爬取一些东西，索性就把Python爬虫的相关内容都总结起来了，自己多动手还是好. （2）保存爬取的图片/视频和文件和网页. （7）某个网站的站内所有目录爬虫. （9）爬虫框架Scrapy . 二，保存爬取的图片/视频和文件和网页. #图片/视频和文件和网页的地址抓取下来后，利用模块urllib里的urlretrieve()方法下载下来：.