mahout部署实践

标签: mahout 实践 | 发表时间:2014-11-08 23:19 | 作者:u010414066
出处:http://blog.csdn.net

一 下载mahout并解压

unzip  unzip mahout-distribution-0.9-src.zip 

二 设置环境变量

1一些说明

JAVA_HOME mahout运行需指定jdk的目录
MAHOUT_JAVA_HOME指定此变量可覆盖JAVA_HOME值
HADOOP_HOME  如果配置,则在hadoop分布式平台上运行,否则单机运行
HADOOP_CONF_DIR指定hadoop的配置文件目录
MAHOUT_LOCAL  如果此变量值丌为空,则单机运行mahout。
MAHOUT_CONF_DIR mahout配置文件的路径,默认值是$MAHOUT_HOME/src/conf
MAHOUT_HEAPSIZE mahout运行时可用的最大heap大小

2具体操作

hadoop@namenode:~/mahout-distribution-0.9$ sudo vim /etc/profile
环境变量的修改,在该文件最后面添加
export JAVA_HOME=/usr/programs/jdk1.7.0_65
export HADOOP_HOME=/home/hadoop/hadoop-1.2.1
export HADOOP_CONF_DIR=/home/hadoop/hadoop-1.2.1/conf
export MAHOUT_HOME=/home/hadoop/mahout-distribution-0.9
export MAHOUT_CONF_DIR=/home/hadoop/mahout-distribution-0.9/conf
 PATH=$MAHOUT_CONF_DIR:$MAHOUT_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH

然后source /etc/profile

3一个问题

如果你遇到了如下问题

Could not find mahout-examples-*.job in /home/hadoop/mahout-distribution-0.9 or /home/hadoop/mahout-distribution-0.9/examples/target, please run 'mvn install' to create the .job file 

原因是下载的版本不对,你是不是下的有源代码的版本呀,下没有源代码的版本试试

三 验证是否安装成功

hadoop@namenode:~$ mahout
MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
Warning: $HADOOP_HOME is deprecated.


Running on hadoop, using /home/hadoop/hadoop-1.2.1/bin/hadoop and HADOOP_CONF_DIR=/home/hadoop/hadoop-1.2.1/conf
MAHOUT-JOB: /home/hadoop/mahout-distribution-0.9/mahout-examples-0.9-job.jar
Warning: $HADOOP_HOME is deprecated.


An example program must be given as the first argument.
Valid program names are:
  arff.vector: : Generate Vectors from an ARFF file or directory
  baumwelch: : Baum-Welch algorithm for unsupervised HMM training
  canopy: : Canopy clustering
  cat: : Print a file or resource as the logistic regression models would see it
  cleansvd: : Cleanup and verification of SVD output
  clusterdump: : Dump cluster output to text
  clusterpp: : Groups Clustering Output In Clusters
  cmdump: : Dump confusion matrix in HTML or text formats
  concatmatrices: : Concatenates 2 matrices of same cardinality into a single matrix
  cvb: : LDA via Collapsed Variation Bayes (0th deriv. approx)
  cvb0_local: : LDA via Collapsed Variation Bayes, in memory locally.
  evaluateFactorization: : compute RMSE and MAE of a rating matrix factorization against probes
  fkmeans: : Fuzzy K-means clustering
  hmmpredict: : Generate random sequence of observations by given HMM
  itemsimilarity: : Compute the item-item-similarities for item-based collaborative filtering
  kmeans: : K-means clustering
  lucene.vector: : Generate Vectors from a Lucene index
  lucene2seq: : Generate Text SequenceFiles from a Lucene index
  matrixdump: : Dump matrix in CSV format
  matrixmult: : Take the product of two matrices
  parallelALS: : ALS-WR factorization of a rating matrix
  qualcluster: : Runs clustering experiments and summarizes results in a CSV
  recommendfactorized: : Compute recommendations using the factorization of a rating matrix
  recommenditembased: : Compute recommendations using item-based collaborative filtering
  regexconverter: : Convert text files on a per line basis based on regular expressions
  resplit: : Splits a set of SequenceFiles into a number of equal splits
  rowid: : Map SequenceFile<Text,VectorWritable> to {SequenceFile<IntWritable,VectorWritable>, SequenceFile<IntWritable,Text>}
  rowsimilarity: : Compute the pairwise similarities of the rows of a matrix
  runAdaptiveLogistic: : Score new production data using a probably trained and validated AdaptivelogisticRegression model
  runlogistic: : Run a logistic regression model against CSV data
  seq2encoded: : Encoded Sparse Vector generation from Text sequence files
  seq2sparse: : Sparse Vector generation from Text sequence files
  seqdirectory: : Generate sequence files (of Text) from a directory
  seqdumper: : Generic Sequence File dumper
  seqmailarchives: : Creates SequenceFile from a directory containing gzipped mail archives
  seqwiki: : Wikipedia xml dump to sequence file
  spectralkmeans: : Spectral k-means clustering
  split: : Split Input data into test and train sets
  splitDataset: : split a rating dataset into training and probe parts
  ssvd: : Stochastic SVD
  streamingkmeans: : Streaming k-means clustering
  svd: : Lanczos Singular Value Decomposition
  testnb: : Test the Vector-based Bayes classifier
  trainAdaptiveLogistic: : Train an AdaptivelogisticRegression model
  trainlogistic: : Train a logistic regression using stochastic gradient descent
  trainnb: : Train the Vector-based Bayes classifier
  transpose: : Take the transpose of a matrix
  validateAdaptiveLogistic: : Validate an AdaptivelogisticRegression model against hold-out data set
  vecdist: : Compute the distances between a set of Vectors (or Cluster or Canopy, they must fit in memory) and a list of Vectors
  vectordump: : Dump vectors from a sequence file to text
  viterbi: : Viterbi decoding of hidden states from given output states sequence

以上则表示安转成功

四 测试kmeans算法

1.下载测试数据


hadoop@namenode:~$ wget http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_control.data
--2014-11-08 06:40:16--  http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_control.data
Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.1.87
Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.1.87|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 288374 (282K) [text/plain]
Saving to: `synthetic_control.data'


100%[=======================================================================>] 288,374     79.5K/s   in 3.5s    


2014-11-08 06:40:20 (79.5 KB/s) - `synthetic_control.data' saved [288374/288374]

2.把测试数据放到hdfs中

hadoop@namenode:~$ hadoop fs -mkdir ./testdata
Warning: $HADOOP_HOME is deprecated.

hadoop@namenode:~$ hadoop fs -ls
Warning: $HADOOP_HOME is deprecated.

Found 4 items
drwxr-xr-x   - hadoop        supergroup          0 2014-11-06 07:48 /user/hadoop/input
drwxr-xr-x   - hadoop        supergroup          0 2014-11-06 07:49 /user/hadoop/output
drwxr-xr-x   - Administrator supergroup          0 2014-11-06 08:01 /user/hadoop/output1
drwxr-xr-x   - hadoop        supergroup          0 2014-11-08 06:41 /user/hadoop/testdata
hadoop@namenode:~$ hadoop fs -put synthetic_control.data  ./testdata
Warning: $HADOOP_HOME is deprecated.

hadoop@namenode:~$ hadoop fs -ls
Warning: $HADOOP_HOME is deprecated.

Found 4 items
drwxr-xr-x   - hadoop        supergroup          0 2014-11-06 07:48 /user/hadoop/input
drwxr-xr-x   - hadoop        supergroup          0 2014-11-06 07:49 /user/hadoop/output
drwxr-xr-x   - Administrator supergroup          0 2014-11-06 08:01 /user/hadoop/output1
drwxr-xr-x   - hadoop        supergroup          0 2014-11-08 06:42 /user/hadoop/testdata
hadoop@namenode:~$ hadoop fs -ls ./testdata
Warning: $HADOOP_HOME is deprecated.

Found 1 items
-rw-r--r--   1 hadoop supergroup     288374 2014-11-08 06:42 /user/hadoop/testdata/synthetic_control.data

3.开始测试

hadoop@namenode:~$ mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
Warning: $HADOOP_HOME is deprecated.


Running on hadoop, using /home/hadoop/hadoop-1.2.1/bin/hadoop and HADOOP_CONF_DIR=/home/hadoop/hadoop-1.2.1/conf
MAHOUT-JOB: /home/hadoop/mahout-distribution-0.9/mahout-examples-0.9-job.jar
Warning: $HADOOP_HOME is deprecated.


14/11/08 06:47:25 WARN driver.MahoutDriver: No org.apache.mahout.clustering.syntheticcontrol.kmeans.Job.props found on classpath, will use command-line arguments only
14/11/08 06:47:25 INFO kmeans.Job: Running with default arguments
14/11/08 06:47:27 INFO kmeans.Job: Preparing Input
14/11/08 06:47:27 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/11/08 06:47:30 INFO input.FileInputFormat: Total input paths to process : 1
14/11/08 06:47:30 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/11/08 06:47:30 WARN snappy.LoadSnappy: Snappy native library not loaded
14/11/08 06:47:31 INFO mapred.JobClient: Running job: job_201411080632_0002
14/11/08 06:47:32 INFO mapred.JobClient:  map 0% reduce 0%
14/11/08 06:48:18 INFO mapred.JobClient:  map 100% reduce 0%
14/11/08 06:48:21 INFO mapred.JobClient: Job complete: job_201411080632_0002
14/11/08 06:48:21 INFO mapred.JobClient: Counters: 19
14/11/08 06:48:21 INFO mapred.JobClient:   Job Counters 
14/11/08 06:48:21 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=19688
14/11/08 06:48:21 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/11/08 06:48:21 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/11/08 06:48:21 INFO mapred.JobClient:     Rack-local map tasks=1
14/11/08 06:48:21 INFO mapred.JobClient:     Launched map tasks=1
14/11/08 06:48:21 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
14/11/08 06:48:21 INFO mapred.JobClient:   File Output Format Counters 
14/11/08 06:48:21 INFO mapred.JobClient:     Bytes Written=335470
14/11/08 06:48:21 INFO mapred.JobClient:   FileSystemCounters
14/11/08 06:48:21 INFO mapred.JobClient:     HDFS_BYTES_READ=288503
14/11/08 06:48:21 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=58838
14/11/08 06:48:21 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=335470
14/11/08 06:48:21 INFO mapred.JobClient:   File Input Format Counters 
14/11/08 06:48:21 INFO mapred.JobClient:     Bytes Read=288374
14/11/08 06:48:21 INFO mapred.JobClient:   Map-Reduce Framework
14/11/08 06:48:21 INFO mapred.JobClient:     Map input records=600
14/11/08 06:48:21 INFO mapred.JobClient:     Physical memory (bytes) snapshot=38473728
14/11/08 06:48:21 INFO mapred.JobClient:     Spilled Records=0
14/11/08 06:48:21 INFO mapred.JobClient:     CPU time spent (ms)=910
14/11/08 06:48:21 INFO mapred.JobClient:     Total committed heap usage (bytes)=16252928
14/11/08 06:48:21 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=347992064
14/11/08 06:48:21 INFO mapred.JobClient:     Map output records=600
14/11/08 06:48:21 INFO mapred.JobClient:     SPLIT_RAW_BYTES=129
14/11/08 06:48:21 INFO kmeans.Job: Running random seed to get initial clusters
14/11/08 06:48:21 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
14/11/08 06:48:21 INFO compress.CodecPool: Got brand-new compressor
14/11/08 06:48:22 INFO kmeans.RandomSeedGenerator: Wrote 6 Klusters to output/random-seeds/part-randomSeed
14/11/08 06:48:22 INFO kmeans.Job: Running KMeans with k = 6
14/11/08 06:48:22 INFO kmeans.KMeansDriver: Input: output/data Clusters In: output/random-seeds/part-randomSeed Out: output
14/11/08 06:48:22 INFO kmeans.KMeansDriver: convergence: 0.5 max Iterations: 10
14/11/08 06:48:22 INFO compress.CodecPool: Got brand-new decompressor
14/11/08 06:48:23 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/11/08 06:48:24 INFO input.FileInputFormat: Total input paths to process : 1
14/11/08 06:48:25 INFO mapred.JobClient: Running job: job_201411080632_0003
14/11/08 06:48:26 INFO mapred.JobClient:  map 0% reduce 0%
14/11/08 06:48:56 INFO mapred.JobClient:  map 100% reduce 0%
14/11/08 06:49:09 INFO mapred.JobClient:  map 100% reduce 100%
14/11/08 06:49:12 INFO mapred.JobClient: Job complete: job_201411080632_0003
14/11/08 06:49:12 INFO mapred.JobClient: Counters: 29
14/11/08 06:49:12 INFO mapred.JobClient:   Job Counters 
14/11/08 06:49:12 INFO mapred.JobClient:     Launched reduce tasks=1
14/11/08 06:49:12 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=21258
14/11/08 06:49:12 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/11/08 06:49:12 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/11/08 06:49:12 INFO mapred.JobClient:     Launched map tasks=1
14/11/08 06:49:12 INFO mapred.JobClient:     Data-local map tasks=1
14/11/08 06:49:12 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=12821
14/11/08 06:49:12 INFO mapred.JobClient:   File Output Format Counters 
14/11/08 06:49:12 INFO mapred.JobClient:     Bytes Written=7581
14/11/08 06:49:12 INFO mapred.JobClient:   FileSystemCounters
14/11/08 06:49:12 INFO mapred.JobClient:     FILE_BYTES_READ=10650
14/11/08 06:49:12 INFO mapred.JobClient:     HDFS_BYTES_READ=358672
14/11/08 06:49:12 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=141341
14/11/08 06:49:12 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=7581
14/11/08 06:49:12 INFO mapred.JobClient:   File Input Format Counters 
14/11/08 06:49:12 INFO mapred.JobClient:     Bytes Read=335470
14/11/08 06:49:12 INFO mapred.JobClient:   Map-Reduce Framework
14/11/08 06:49:12 INFO mapred.JobClient:     Map output materialized bytes=10650
14/11/08 06:49:12 INFO mapred.JobClient:     Map input records=600
14/11/08 06:49:12 INFO mapred.JobClient:     Reduce shuffle bytes=10650
14/11/08 06:49:12 INFO mapred.JobClient:     Spilled Records=12
14/11/08 06:49:12 INFO mapred.JobClient:     Map output bytes=10620
14/11/08 06:49:12 INFO mapred.JobClient:     Total committed heap usage (bytes)=132190208
14/11/08 06:49:12 INFO mapred.JobClient:     CPU time spent (ms)=7490
14/11/08 06:49:12 INFO mapred.JobClient:     Combine input records=0
14/11/08 06:49:12 INFO mapred.JobClient:     SPLIT_RAW_BYTES=122
14/11/08 06:49:12 INFO mapred.JobClient:     Reduce input records=6
14/11/08 06:49:12 INFO mapred.JobClient:     Reduce input groups=6
14/11/08 06:49:12 INFO mapred.JobClient:     Combine output records=0
14/11/08 06:49:12 INFO mapred.JobClient:     Physical memory (bytes) snapshot=183877632
14/11/08 06:49:12 INFO mapred.JobClient:     Reduce output records=6
14/11/08 06:49:12 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=696659968
14/11/08 06:49:12 INFO mapred.JobClient:     Map output records=6
14/11/08 06:49:12 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/11/08 06:49:14 INFO input.FileInputFormat: Total input paths to process : 1
14/11/08 06:49:15 INFO mapred.JobClient: Running job: job_201411080632_0004
14/11/08 06:49:16 INFO mapred.JobClient:  map 0% reduce 0%
14/11/08 06:50:02 INFO mapred.JobClient:  map 100% reduce 0%
14/11/08 06:50:15 INFO mapred.JobClient:  map 100% reduce 100%
14/11/08 06:50:19 INFO mapred.JobClient: Job complete: job_201411080632_0004
14/11/08 06:50:19 INFO mapred.JobClient: Counters: 29
14/11/08 06:50:19 INFO mapred.JobClient:   Job Counters 
14/11/08 06:50:19 INFO mapred.JobClient:     Launched reduce tasks=1
14/11/08 06:50:19 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=25946
14/11/08 06:50:19 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/11/08 06:50:19 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/11/08 06:50:19 INFO mapred.JobClient:     Rack-local map tasks=1
14/11/08 06:50:19 INFO mapred.JobClient:     Launched map tasks=1
14/11/08 06:50:19 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=12738
14/11/08 06:50:19 INFO mapred.JobClient:   File Output Format Counters 
14/11/08 06:50:19 INFO mapred.JobClient:     Bytes Written=7581
14/11/08 06:50:19 INFO mapred.JobClient:   FileSystemCounters
14/11/08 06:50:19 INFO mapred.JobClient:     FILE_BYTES_READ=13890
14/11/08 06:50:19 INFO mapred.JobClient:     HDFS_BYTES_READ=351142
14/11/08 06:50:19 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=147821
14/11/08 06:50:19 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=7581
14/11/08 06:50:19 INFO mapred.JobClient:   File Input Format Counters 
14/11/08 06:50:19 INFO mapred.JobClient:     Bytes Read=335470
14/11/08 06:50:19 INFO mapred.JobClient:   Map-Reduce Framework
14/11/08 06:50:19 INFO mapred.JobClient:     Map output materialized bytes=13890
14/11/08 06:50:19 INFO mapred.JobClient:     Map input records=600
14/11/08 06:50:19 INFO mapred.JobClient:     Reduce shuffle bytes=13890
14/11/08 06:50:19 INFO mapred.JobClient:     Spilled Records=12
14/11/08 06:50:19 INFO mapred.JobClient:     Map output bytes=13860
14/11/08 06:50:19 INFO mapred.JobClient:     Total committed heap usage (bytes)=132190208
14/11/08 06:50:19 INFO mapred.JobClient:     CPU time spent (ms)=6550
14/11/08 06:50:19 INFO mapred.JobClient:     Combine input records=0
14/11/08 06:50:19 INFO mapred.JobClient:     SPLIT_RAW_BYTES=122
14/11/08 06:50:19 INFO mapred.JobClient:     Reduce input records=6
14/11/08 06:50:19 INFO mapred.JobClient:     Reduce input groups=6
14/11/08 06:50:19 INFO mapred.JobClient:     Combine output records=0
14/11/08 06:50:19 INFO mapred.JobClient:     Physical memory (bytes) snapshot=183332864
14/11/08 06:50:19 INFO mapred.JobClient:     Reduce output records=6
14/11/08 06:50:19 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=695611392
14/11/08 06:50:19 INFO mapred.JobClient:     Map output records=6
14/11/08 06:50:19 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/11/08 06:50:20 INFO input.FileInputFormat: Total input paths to process : 1
14/11/08 06:50:21 INFO mapred.JobClient: Running job: job_201411080632_0005
14/11/08 06:50:22 INFO mapred.JobClient:  map 0% reduce 0%
14/11/08 06:50:48 INFO mapred.JobClient:  map 100% reduce 0%
14/11/08 06:50:59 INFO mapred.JobClient:  map 100% reduce 33%
14/11/08 06:51:01 INFO mapred.JobClient:  map 100% reduce 100%
14/11/08 06:51:05 INFO mapred.JobClient: Job complete: job_201411080632_0005
14/11/08 06:51:05 INFO mapred.JobClient: Counters: 29
14/11/08 06:51:05 INFO mapred.JobClient:   Job Counters 
14/11/08 06:51:05 INFO mapred.JobClient:     Launched reduce tasks=1
14/11/08 06:51:05 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=17047
14/11/08 06:51:05 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/11/08 06:51:05 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/11/08 06:51:05 INFO mapred.JobClient:     Rack-local map tasks=1
14/11/08 06:51:05 INFO mapred.JobClient:     Launched map tasks=1
14/11/08 06:51:05 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=12804
14/11/08 06:51:05 INFO mapred.JobClient:   File Output Format Counters 
14/11/08 06:51:05 INFO mapred.JobClient:     Bytes Written=7581
14/11/08 06:51:05 INFO mapred.JobClient:   FileSystemCounters
14/11/08 06:51:05 INFO mapred.JobClient:     FILE_BYTES_READ=13890
14/11/08 06:51:05 INFO mapred.JobClient:     HDFS_BYTES_READ=351142
14/11/08 06:51:05 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=147821
14/11/08 06:51:05 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=7581
14/11/08 06:51:05 INFO mapred.JobClient:   File Input Format Counters 
14/11/08 06:51:05 INFO mapred.JobClient:     Bytes Read=335470
14/11/08 06:51:05 INFO mapred.JobClient:   Map-Reduce Framework
14/11/08 06:51:05 INFO mapred.JobClient:     Map output materialized bytes=13890
14/11/08 06:51:05 INFO mapred.JobClient:     Map input records=600
14/11/08 06:51:05 INFO mapred.JobClient:     Reduce shuffle bytes=13890
14/11/08 06:51:05 INFO mapred.JobClient:     Spilled Records=12
14/11/08 06:51:05 INFO mapred.JobClient:     Map output bytes=13860
14/11/08 06:51:05 INFO mapred.JobClient:     Total committed heap usage (bytes)=132190208
14/11/08 06:51:05 INFO mapred.JobClient:     CPU time spent (ms)=3280
14/11/08 06:51:05 INFO mapred.JobClient:     Combine input records=0
14/11/08 06:51:05 INFO mapred.JobClient:     SPLIT_RAW_BYTES=122
14/11/08 06:51:05 INFO mapred.JobClient:     Reduce input records=6
14/11/08 06:51:05 INFO mapred.JobClient:     Reduce input groups=6
14/11/08 06:51:05 INFO mapred.JobClient:     Combine output records=0
14/11/08 06:51:05 INFO mapred.JobClient:     Physical memory (bytes) snapshot=183197696
14/11/08 06:51:05 INFO mapred.JobClient:     Reduce output records=6
14/11/08 06:51:05 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=695611392
14/11/08 06:51:05 INFO mapred.JobClient:     Map output records=6

4观察输出

hadoop@namenode:~$ hadoop fs -ls ./output
Warning: $HADOOP_HOME is deprecated.


Found 15 items
-rw-r--r--   1 hadoop supergroup        194 2014-11-08 06:56 /user/hadoop/output/_policy
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:57 /user/hadoop/output/clusteredPoints
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:48 /user/hadoop/output/clusters-0
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:49 /user/hadoop/output/clusters-1
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:56 /user/hadoop/output/clusters-10-final
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:50 /user/hadoop/output/clusters-2
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:51 /user/hadoop/output/clusters-3
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:51 /user/hadoop/output/clusters-4
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:52 /user/hadoop/output/clusters-5
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:53 /user/hadoop/output/clusters-6
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:54 /user/hadoop/output/clusters-7
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:54 /user/hadoop/output/clusters-8
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:55 /user/hadoop/output/clusters-9
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:48 /user/hadoop/output/data
drwxr-xr-x   - hadoop supergroup          0 2014-11-08 06:48 /user/hadoop/output/random-seeds


作者:u010414066 发表于2014-11-8 23:19:28 原文链接
阅读:42 评论:0 查看评论

相关 [mahout 实践] 推荐:

mahout部署实践

- - CSDN博客云计算推荐文章
一 下载mahout并解压. JAVA_HOME mahout运行需指定jdk的目录. MAHOUT_JAVA_HOME指定此变量可覆盖JAVA_HOME值. HADOOP_HOME  如果配置,则在hadoop分布式平台上运行,否则单机运行. HADOOP_CONF_DIR指定hadoop的配置文件目录.

【甘道夫】Mahout推荐算法编程实践

- - CSDN博客云计算推荐文章
Taste是曾经风靡一时的推荐算法框架,后来被并入Mahout中,Mahout的部分推荐算法基于Taste实现. 下文介绍基于Taste实现最常用的UserCF和ItemCF. 本文不涉及UserCF和ItemCF算法的介绍,这方面网上资料很多,本文仅介绍如何基于Mahout编程实现. UserCF和ItemCF算法的输入数据是用户偏好,用户偏好数据可以有两种形式:.

Mahout介绍

- - 互联网 - ITeye博客
Mahout 是机器学习和数据挖掘的一个分布式框架,区别于其他的开源数据挖掘软件,它是基于hadoop之上的; 所以hadoop的优势就是Mahout的优势. http://mahout.apache.org/ 上说的Scalable就是指hadoop的可扩展性. Mahout用map-reduce实现了部分数据挖掘算法,解决了并行挖掘的问题.

mahout 实用教程之一

- - CSDN博客云计算推荐文章
mahout 实用教程 (一). 本文力求把mahout从使用的角度为读者建立一个框架,为后续的使用打下基础. 本文为原创文章转载请注明原网址 http://blog.csdn.net/comaple,谢谢. 下面首先给出源代码svn地址以及用于测试的公共数据集,大家可以下载并测试. mahout svn仓库地址: http://svn.apache.org/repos/asf/mahout/trunk.

Mahout: SVDRecommender SVD推荐算法

- -

Mahout实现的机器学习算法

- - ITeye博客
使用命令:mahout -h.   在Mahout实现的机器学习算法见下表:. EM聚类(期望最大化聚类). 并行FP Growth算法. 并行化了Watchmaker框架. 非Map-Reduce算法. 扩展了java的Collections类. Mahout最大的优点就是基于hadoop实现,把很多以前运行于单机上的算法,转化为了MapReduce模式,这样大大提升了算法可处理的数据量和处理性能.

[转]Mahout推荐算法基础

- - 小鸥的博客
Mahout推荐算法分为以下几大类. 2.相近的用户定义与数量. 2.用户数较少时计算速度快. 1.基于item的相似度. 1.item较少时就算速度更快. 2.当item的外部概念易于理解和获得是非常有用. 1基于SlopeOne算法(打分差异规则). 当item数目十分少了也很有效. 需要限制diffs的存储数目否则内存增长太快.

Apache Mahout 0.8 发布,机器学习库

- - 开源中国社区最新新闻
Apache Mahout 0.8 发布了,Apache Mahout 是 Apache Software Foundation (ASF) 开发的一个全新的开源项目,其主要目标是创建一些可伸缩的机器学习算法,供开发人员在 Apache 在许可下免费使用. 该项目已经发展到了它的最二个年头,目前只有一个公共发行版.

基于Mahout的电影推荐系统

- - CSDN博客推荐文章
Apache Mahout 是 Apache Software Foundation(ASF) 旗下的一个开源项目,提供一些可扩展的机器学习领域经典算法的实现,旨在帮助开发人员更加方便快捷地创建智能应用程序. 经典算法包括聚类、分类、协同过滤、进化编程等等,并且,在 Mahout 中还加入了对Apache Hadoop的支持,使这些算法可以更高效的运行在云计算环境中.

使用Mahout实现协同过滤 spark

- - zzm
Mahout使用了Taste来提高协同过滤算法的实现,它是一个基于Java实现的可扩展的,高效的推荐引擎. Taste既实现了最基本的基 于用户的和基于内容的推荐算法,同时也提供了扩展接口,使用户可以方便的定义和实现自己的推荐算法. 同时,Taste不仅仅只适用于Java应用程序,它 可以作为内部服务器的一个组件以HTTP和Web Service的形式向外界提供推荐的逻辑.