通过ngxtop实时监控web server的访问情况

      关于对nginx web server的实时访问的实时监控问题，我很久之前就想实现的，现在虽有nginx自带的status扩展，但那是全局的，无法细分到vhost，并且提供的metric也很少，加之目前是通过cacti每5分钟取nginx status，实时性也不是很好，记得前一阵为maptail兴奋一点，以为能够解决我的问题，可是他只是漂亮的在地图上显示实时用户的ip地理位置信息，没有其他功能了，不过他的思想很重要---"tail -f  log",如今的ngxtop其实也是这种思想，把功能做的丰富了点，我在今天测试的时候遇到一些问题，现在把过程整理下来，备忘也供大家参考。

ngxtop现在虽然还有些问题，如本文最后，但开源才15天，值得期待！
Real-time metrics for nginx server：https://github.com/lebinh/ngxtop 

1、安装python
tar xzvf Python-2.7.3
cd Python-2.7.3
./configure --prefix=/usr/local/python2.7
make 
make install

安装后python2.7所在路径
/usr/local/python2.7/bin/python2.7

为保持python2.4不变，建立python2.7的软连接指向python2.7所在路径
mv /usr/bin/python /usr/bin/python.bak
ln -s /usr/local/python2.7/bin/python2.7 /usr/bin/python2.7
ln -s /usr/bin/python2.7 /usr/bin/python


2、安装setuptools
wget https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg  --no-check-certificate
chmod +x setuptools-0.6c11-py2.7.egg
sh setuptools-0.6c11-py2.7.egg


3.1、通过pip安装ngxtop，因为源在国外，可能下载、安装过程会很慢
安装pip
https://pypi.python.org/packages/source/p/pip/pip-1.5.4.tar.gz#md5=834b2904f92d46aaa333267fb1c922bb
tar xzvf pip-1.5.4.tar.gz       
cd pip-1.5.4  
python setup.py install


安装ngxtop
/usr/local/bin/pip install ngxtop


3.2、直接通过ngxtop源码安装
wget https://github.com/lebinh/ngxtop/archive/1c200d51fbae7824a30159714669146d6b214210.zip
unzip ngxtop-1c200d51fbae7824a30159714669146d6b214210.zip
cd ngxtop-1c200d51fbae7824a30159714669146d6b214210
python setup.py install

如果提示没有pyparsing的话，下载安装即可
wget https://pypi.python.org/packages/source/p/pyparsing/pyparsing-2.0.1.zip
unzip pyparsing-2.0.1.zip
cd pyparsing-2.0.1
python setup.py instal


4、使用及效果截图
如果不出效果debug一下：
ngxtop -t 1 -l /var/log/nginx/access.log -d

默认输出：
ngxtop -t 1 -l /var/log/nginx/access.log


显示请求最多的ip和request
ngxtop -t 1 -l /var/log/nginx/access.log top remote_addr request


远程实时分析（这个很实用呦，不用每台server都安装ngxtop）
sshpass -f pass.txt ssh 192.168.0.100 tail -f /var/log/nginx/nginx.log|ngxtop

其他用法

ngxtop --help

5、测试过程遇到的问题
（1）执行ngxtop会报如下错误：ImportError: No module named _sqlite3
# /usr/local/bin/ngxtop --help
Traceback (most recent call last):
  File "/usr/local/bin/ngxtop", line 7, in <module>
    from ngxtop.ngxtop import main
  File "/usr/local/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 60, in <module>
    import sqlite3
  File "/usr/local/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3


解决方案：编辑Python-2.7.3/Modules/_sqlite/connection.c加入如下44-57行，然后重新编译python2.7即可
[root@admin200 Python-2.7.3]# vim Modules/_sqlite/connection.c 
  38 #if SQLITE_VERSION_NUMBER >= 3003008
  39 #ifndef SQLITE_OMIT_LOAD_EXTENSION
  40 #define HAVE_LOAD_EXTENSION
  41 #endif
  42 #endif
  43 
  44 /*** add by lai for **/
  45 #ifdef SQLITE_INT64_TYPE
  46 typedef SQLITE_INT64_TYPE sqlite_int64;
  47 typedef unsigned SQLITE_INT64_TYPE sqlite_uint64;
  48 #elif defined(_MSC_VER) || defined(__BORLANDC__)
  49 typedef __int64 sqlite_int64;
  50 typedef unsigned __int64 sqlite_uint64;
  51 #else
  52 typedef long long int sqlite_int64;
  53 typedef unsigned long long int sqlite_uint64;
  54 #endif
  55 typedef sqlite_int64 sqlite3_int64;
  56 typedef sqlite_uint64 sqlite3_uint64;
  57 /*** end ***/
  58 
  59 
  60 static int pysqlite_connection_set_isolation_level(pysqlite_Connection* self, PyObject* isolation_level);
  61 static void _pysqlite_drop_unused_cursor_references(pysqlite_Connection* self);
  62 




（2）、无法启动ngxtop，报错：OSError: [Errno 2] No such file or directory
[root@admin200 nginx]# /usr/local/bin/ngxtop
Traceback (most recent call last):
  File "/usr/local/bin/ngxtop", line 11, in <module>
    sys.exit(main())
  File "/usr/local/python2.7/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 446, in main
    process(args)
  File "/usr/local/python2.7/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 408, in process
    config = get_nginx_conf_path()
  File "/usr/local/python2.7/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 120, in get_nginx_conf_path
    proc = subprocess.Popen(['nginx', '-V'], stderr=subprocess.PIPE)
  File "/usr/local/python2.7/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/local/python2.7/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
[root@admin200 nginx]# 


解决方法：因nginx为自定义安装目录，ngxtop需要查找nginx的pid，故需要将nginx路径加入环境变量，或者做个符号链接
cp /usr/local/nginx/sbin/nginx  /usr/bin/
或者
ln -s /usr/local/nginx/sbin/nginx /usr/bin/nginx


（3）远程实时分析其他server的log
sshpass -f pass.txt ssh 192.168.0.100 tail -f /var/log/nginx/bod-access.log|ngxtop

运行几秒就报错了，该问题已想开发者反馈
Traceback (most recent call last):
  File "/usr/local/bin/ngxtop", line 11, in <module>
    sys.exit(main())
  File "build/bdist.linux-x86_64/egg/ngxtop/ngxtop.py", line 495, in main
  File "build/bdist.linux-x86_64/egg/ngxtop/ngxtop.py", line 463, in process
  File "build/bdist.linux-x86_64/egg/ngxtop/ngxtop.py", line 357, in process_log
  File "build/bdist.linux-x86_64/egg/ngxtop/ngxtop.py", line 305, in process

sqlite3.ProgrammingError: library routine called out of sequence