Erlang集群自动化添加节点指南

Erlang的集群是由各个节点组成的，一个节点有一个名字来标识，而不管这个节点在网络的物理位置，所以在部署Erlang集群的时候就很方便。只要在集群里新启动一个节点，给个相对固定的引导的节点，让新节点和这个引导节点取得联系，由引导节点把新节点介绍入集群就OK了。
在实践中，新采购的机器上通常配置好IP,以及ssh访问权限。 我们需要在新机器上手工安装Erlang系统，部署新应用，然后启动应用节点，加入集群服务，这个步骤很繁琐。我们希望能够自动化去做这个事情。common_test的ct_系列模块来救助了。
common_test是A framework for automated testing of arbitrary target nodes，它随带的ct_ssh可以透过ssh在远程机器上执行各种各样的shell命令，通过scp传输数据;而ct_slave非常方便的可以连接到远程机器启动一个erlang节点。
pool（www.erlang.org/doc/man/pool.htm）模块也可以远程启动节点，但是它要依赖于操纵系统的ssh工具，需要在机器之间做ssh互信，也就是说ssh targetip这样的不能出现任何的交互,比如说键入密码,很不方便。

我们首先来演示下如何在远程机器上执行ssh命令：

我们开个ssh以用户yourname连接到本机127.0.0.1执行模拟上传文件，下载文件，执行文件拷贝操纵等。

＃首先我们准备个配置文件
$ cd ~
$ cat sshdemo.config
{sshdemo,
     [
     {ssh, "127.0.0.1"},
     {port, 22},
     {user, "yourname"},
     {password, "yourpassword"}
     ]
}.

$ mkdir /tmp/sshdemo
$ run_test -shell -config sshdemo.config
Erlang R14B01 (erts-5.8.5) 1 [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Common Test v1.5.1 starting (cwd is /Users/yufeng)

Eshell V5.8.5  (abort with ^G)
(ct@yufengdemacbook)1>
Installing: [{ct_config_plain,["/Users/yufeng/sshdemo.config"]}]

Updated /Users/yufeng/last_interactive.html
Any CT activities will be logged here

(ct@yufengdemacbook)1> {ok, CH}=ct_ssh:connect(sshdemo, sftp).
{ok,}
(ct@yufengdemacbook)2> ct_ssh:list_dir(CH, "/tmp/sshdemo").
{ok,["..","."]}
(ct@yufengdemacbook)3> ct_ssh:write_file(CH, "/tmp/sshdemo/test.dat", "hello").
ok
(ct@yufengdemacbook)4> ct_ssh:read_file(CH, "/tmp/sshdemo/test.dat").
{ok,<>}
(ct@yufengdemacbook)5> {ok, CH1}=ct_ssh:connect(sshdemo, ssh).
{ok,}
(ct@yufengdemacbook)6> ct_ssh:exec(CH1, "cp /tmp/sshdemo/test.dat /tmp/sshdemo/test1.dat").
{ok,[]}
(ct@yufengdemacbook)7>
...

$ ls /tmp/sshdemo
test.dat  test1.dat

通过上面的试验，我们可以很方便的透过ssh在目标机器安装我们的erlang系统以及部署我们的应用。接下来我们来演示下如何在目标机器开启一个节点：

$ run_test -shell -name [email protected]

Erlang R14B01 (erts-5.8.5) 1 [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Common Test v1.5.1 starting (cwd is /Users/yufeng)

Eshell V5.8.5  (abort with ^G)
([email protected])1>

Updated /Users/yufeng/last_interactive.html
Any CT activities will be logged here

([email protected])1> ct_slave:start('127.0.0.1', y, [{username, "yourname"},{password, "yourpassword"}]).
{error,boot_timeout,'[email protected]'}

我们可以看到我们可耻的失败了，我们没在默认的3秒内看到我们的节点启动成功。问题出在哪里呢？我们来系统的跟踪下：
首先从源码ct_slave.erl:L403中我们可以清楚的看到：

% spawn node remotely
spawn_remote_node(Host, Node, Options) ->
    Username = Options#options.username,
    Password = Options#options.password,
    ErlFlags = Options#options.erl_flags,
    SSHOptions = case {Username, Password} of
        {[], []}->
            [];
        {_, []}->
            [{user, Username}];
        {_, _}->
            [{user, Username}, {password, Password}]
    end ++ [{silently_accept_hosts, true}],
    check_for_ssh_running(),
    {ok, SSHConnRef} = ssh:connect(atom_to_list(Host), 22, SSHOptions),
    {ok, SSHChannelId} = ssh_connection:session_channel(SSHConnRef, infinity),
    ssh_connection:exec(SSHConnRef, SSHChannelId, get_cmd(Node, ErlFlags), infinity).
...
% wait N seconds until node is pingable
wait_for_node_alive(_Node, 0) ->
    pang;
wait_for_node_alive(Node, N) ->
    timer:sleep(1000),
    case net_adm:ping(Node) of
        pong->
            pong;
        pang->
            wait_for_node_alive(Node, N-1)
    end.
%%等待的逻辑
...
MasterHost = gethostname(),
    if
        MasterHost == Host ->
            spawn_local_node(Node, Options);
        true->
            spawn_remote_node(Host, Node, Options)
    end,
    BootTimeout = Options#options.boot_timeout,
    InitTimeout = Options#options.init_timeout,
    StartupTimeout = Options#options.startup_timeout,
    Result = case wait_for_node_alive(ENode, BootTimeout) of
        pong->
            call_functions(ENode, Functions2),
            receive
                {node_started, ENode}->
                    receive
                        {node_ready, ENode}->
                            {ok, ENode}
                    after StartupTimeout*1000->
                        {error, startup_timeout, ENode}
                    end
            after InitTimeout*1000 ->
                {error, init_timeout, ENode}
            end;
        pang->
            {error, boot_timeout, ENode}
    end,
...

我们来用强大的dbg跟踪下ct_slave的执行情况：

([email protected])2> dbg:tracer().
{ok,}
([email protected])3> dbg:p(all,c).
{ok,[{matched,'[email protected]',32}]}
([email protected])4> dbg:tpl(ssh_connection,exec, [{'_', [], [{return_trace}]}]).
{ok,[{matched,'[email protected]',1},{saved,1}]}
([email protected])5> dbg:tpl(ct_slave,wait_for_node_alive, [{'_', [], [{return_trace}]}]).
{ok,[{matched,'[email protected]',1},{saved,2}]}
([email protected])6> ct_slave:start('127.0.0.1', y, [{username, "yourname"},{password, "yourpassword"} ]).
() call ssh_connection:exec(,0,"erl -detached -noinput -setcookie VSVDYDCFVTPSDPBFHQMY -name y ",infinity)
() returned from ssh_connection:exec/4 -> success
() call ct_slave:wait_for_node_alive('[email protected]',3)
() call ct_slave:wait_for_node_alive('[email protected]',2)
() call ct_slave:wait_for_node_alive('[email protected]',1)
{error,boot_timeout,'[email protected]'}
([email protected])7> code:which(ct_slave).
"/usr/local/lib/erlang/lib/common_test-1.5.4/ebin/ct_slave.beam"

从上面的跟踪命令我们知道我们的ssh命令是执行成功的,但是在等待节点的时候失败了,通过查看进程和epmd也可以验证这一点：

$ ps -ef|grep beam
...
 500 36161 36120   0   0:00.24 ttys000    0:00.81 /usr/local/lib/erlang/erts-5.8.5/bin/beam.smp -- -root /usr/local/lib/erlang -progname erl -- -home /Users/yufeng -- -name [email protected] -s ct_run script_start -shell
 501 26067     1   0   0:00.03 ??         0:00.21 /opt/local/lib/erlang/erts-5.8.5/bin/beam -- -root /opt/local/lib/erlang -progname erl -- -home /Users/yufeng -noshell -noinput -noshell -noinput -setcookie VSVDYDCFVTPSDPBFHQMY -name y
...

$ epmd -names
epmd: up and running on port 4369 with data:
name x at port 49869
name y at port 49838

从上面的分析，我们可以清楚的看到：
我们想要的名字是 -name [email protected], 等待的也是这个名字，但是ct_slave启动的是-name y.这明显是个bug!
我们看下代码确定如何修复：

% get cmd for starting Erlang
get_cmd(Node, Flags) ->
    Cookie = erlang:get_cookie(),
    "erl -detached -noinput -setcookie "++ atom_to_list(Cookie) ++
    long_or_short() ++ atom_to_list(Node) ++ " " ++ Flags.

% make a Erlang node name from name and hostname
enodename(Host, Node) ->
    list_to_atom(atom_to_list(Node)++"@"++atom_to_list(Host)).

我们可以看到get_cmd只是把node名字加到命令行去，但是node名字里面没有host部分。
知道原因修正就容易了，改下这行就好：

ssh_connection:exec(SSHConnRef, SSHChannelId, get_cmd(enodename(Host, Node), ErlFlags), infinity).

我们来重新编译，安装下：

$ pwd
/Users/yufeng/otp
$ export ERL_TOP=`pwd`
$ cd lib/common_test/
$ make
$ sudo cp ebin/ct_slave.beam /usr/local/lib/erlang/lib/common_test-1.5.4/ebin/ct_slave.beam

代码升级好了，我们再试验下我们的猜想：

([email protected])8> l(ct_slave).
{module,ct_slave}
([email protected])9> ct_slave:start('127.0.0.1', y, [{username, "yourname"},{password, "yourpassword"}]).
(<0.39.0>) call ssh_connection:exec(<0.105.0>,0,"erl -detached -noinput -setcookie VSVDYDCFVTPSDPBFHQMY -name [email protected] ",infinity)
(<0.39.0>) returned from ssh_connection:exec/4 -> success
(<0.39.0>) call ct_slave:wait_for_node_alive('[email protected]',3)
(<0.39.0>) call ct_slave:wait_for_node_alive('[email protected]',2)
(<0.39.0>) returned from ct_slave:wait_for_node_alive/2 -> pong
(<0.39.0>) returned from ct_slave:wait_for_node_alive/2 -> pong
{ok,'[email protected]'}
([email protected])10> nodes().
['[email protected]']

这次试验成功了。我们透过远程启动节点，把[email protected]成功加入集群。

通过上面的2个步骤，我们就可以在新添加的裸机上方便的部署我们的Erlang系统，控制节点的运作和停止。

祝大家集群开心！