Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用

  • A+
摘要

需要实现目标,节点1down自动将web服务转到节点2上,不论node1和node2谁down掉,我们的web服务永远在线以此达到高可用

环境说明:

node1:ZhongH100.wxjr.com.cn 172.16.6.100
node2:ZhongH101.wxjr.com.cn 172.16.6.101
NFS存储:ZhongH102.wxjr.com.cn 172.16.6.102
VIP:172.16.7.200 node1和node2的主机名、SSH互通 
操作系统:都是CentOS 6.6 x86_64 (自己配置好yum源、epel源、关闭SELinux、关闭防火墙) 
注意:是[root@ZhongH100 ~]还是[root@ZhongH101 ~]或者是[root@ZhongH102 ~]
如果是[root@ZhongH ~]那么就是node1 node2两台机器都需要运行的

拓扑结构:

Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用


一、在node1和node2上互做对方主机名解析、同步时间

[root@ZhongH100 /tmp]# tail -1 /etc/hosts
172.16.6.101 ZhongH101.wxjr.com.cn
[root@ZhongH100 /tmp]# crontab -l
*/5 * * * * /usr/sbin/ntpdate pool.ntp.org >/dev/null 2>&1
[root@ZhongH100 /tmp]# 
[root@ZhongH101 ~]# tail -1 /etc/hosts
172.16.6.100 ZhongH100.wxjr.com.cn
[root@ZhongH101 /tmp]# crontab -l
*/5 * * * * /usr/sbin/ntpdate pool.ntp.org >/dev/null 2>&1
[root@ZhongH101 /tmp]# 

二、node1和node2两节点之间配置SSH互信

[root@ZhongH100 /tmp]# ssh-keygen  -t rsa -f ~/.ssh/id_rsa  -P ''
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
29:d6:6a:68:92:62:7b:84:c1:de:b1:65:20:bf:f4:19 root@ZhongH100.wxjr.com.cn
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
| . .             |
|. o .            |
| o + E . .       |
|. = B = S        |
| o * = o         |
|..+ o o          |
|...+ .           |
| ..              |
+-----------------+
[root@ZhongH100 /tmp]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@ZhongH101.wxjr.com.cn
The authenticity of host 'zhongh101.wxjr.com.cn (172.16.6.101)' can't be established.
RSA key fingerprint is e9:95:aa:7f:39:5b:52:a7:9b:5e:fe:98:19:82:14:e3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'zhongh101.wxjr.com.cn,172.16.6.101' (RSA) to the list of known hosts.
root@zhongh101.wxjr.com.cn's password:
Now try logging into the machine, with "ssh 'root@ZhongH101.wxjr.com.cn'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

[root@ZhongH100 /tmp]# 
[root@ZhongH101 ~]# ssh-keygen  -t rsa -f ~/.ssh/id_rsa  -P ''
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
7e:b2:24:1d:29:ea:41:6b:02:fc:41:e6:81:5a:4c:e3 root@ZhongH101.wxjr.com.cn
The key's randomart image is:
+--[ RSA 2048]----+
|  o              |
| +..             |
| .E+             |
|o.+ .    .       |
|o. o. . S        |
| ....o + .       |
|  ..= . = .      |
|   + . o +       |
|    .   .        |
+-----------------+
[root@ZhongH101 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@ZhongH100.wxjr.com.cn
The authenticity of host 'zhongh100.wxjr.com.cn (172.16.6.100)' can't be established.
RSA key fingerprint is 90:26:f4:28:31:04:03:6c:9f:ec:e4:09:04:32:92:ee.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'zhongh100.wxjr.com.cn,172.16.6.100' (RSA) to the list of known hosts.
root@zhongh100.wxjr.com.cn's password:
Now try logging into the machine, with "ssh 'root@ZhongH100.wxjr.com.cn'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

[root@ZhongH101 ~]# 

三、安装Heartbeat和Nginx

节点1、节点2上安装Heartbeat

[root@ZhongH /tmp]# yum -y install heartbeat*

节点1、节点2上编译安装Nginx

[root@ZhongH /tmp]# wget http://nginx.org/download/nginx-1.9.1.tar.gz
[root@ZhongH /tmp]# tar xf nginx-1.9.1.tar.gz
[root@ZhongH /tmp]# cd nginx-1.9.1
[root@ZhongH /tmp/nginx-1.9.1]# Username="www" && for i in `seq 1000 1500`;do [ -z "$(awk -F: '{print$3,$4}' /etc/passwd | grep "$i")" -a -z "$(awk -F: '{print$3}' /etc/group | grep "$i")" ] && UGID=$i && break;done && groupadd -g $UGID $Username && useradd -M -u $UGID -g $UGID -s /sbin/nologin $Username
[root@ZhongH /tmp/nginx-1.9.1]# yum install pcre-devel pcre -y
[root@ZhongH /tmp/nginx-1.9.1]# mkdir -p {/tmp/nginx,/var/run/nginx,/var/lock}
[root@ZhongH /tmp/nginx-1.9.1]# ./configure --prefix=/usr/local/nginx/ --user=www --group=www \
--error-log-path=/tmp/nginx/error.log --http-log-path=/tmp/nginx/access.log \
--pid-path=/var/run/nginx/nginx.pid --lock-path=/var/lock/nginx.lock \
--with-pcre --with-http_ssl_module --with-http_flv_module \
--with-http_spdy_module --with-http_gzip_static_module \
--with-http_stub_status_module --http-client-body-temp-path=/usr/local/nginx/client/ \
--http-proxy-temp-path=/usr/local/nginx/proxy/ --http-fastcgi-temp-path=/usr/local/nginx/fcgi/ \
--http-uwsgi-temp-path=/usr/local/nginx/uwsgi --http-scgi-temp-path=/usr/local/nginx/scgi
[root@ZhongH /tmp/nginx-1.9.1]# make -j $(awk '{if($1=="processor"){i++}}END{print i}' /proc/cpuinfo) && make install && echo $?
[root@ZhongH /tmp/nginx-1.9.1]# cd ../
[root@ZhongH /tmp]# echo "export PATH=/usr/local/nginx/sbin:\$PATH" > /etc/profile.d/nginx1.9.1.sh
[root@ZhongH /tmp]# . /etc/profile.d/nginx1.9.1.sh
[root@ZhongH /tmp]# which nginx
/usr/local/nginx/sbin/nginx
[root@ZhongH /tmp]# nginx -V
nginx version: nginx/1.9.1
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx/ --user=www --group=www --error-log-path=/tmp/nginx/error.log --http-log-path=/tmp/nginx/access.log --pid-path=/var/run/nginx/nginx.pid --lock-path=/var/lock/nginx.lock --with-pcre --with-http_ssl_module --with-http_flv_module --with-http_spdy_module --with-http_gzip_static_module --with-http_stub_status_module --http-client-body-temp-path=/usr/local/nginx/client/ --http-proxy-temp-path=/usr/local/nginx/proxy/ --http-fastcgi-temp-path=/usr/local/nginx/fcgi/ --http-uwsgi-temp-path=/usr/local/nginx/uwsgi --http-scgi-temp-path=/usr/local/nginx/scgi
[root@ZhongH /tmp]# wget http://www.dwhd.org/script/Nginx-init-CentOS -O /etc/rc.d/init.d/nginx
[root@ZhongH /tmp]# chmod +x /etc/rc.d/init.d/nginx
[root@ZhongH /tmp]# chkconfig --add nginx
[root@ZhongH /tmp]# chkconfig nginx on
[root@ZhongH /tmp]# chkconfig nginx off #取消nginx的开机启动是因为我们需要让heartbeat来控制nginx
[root@ZhongH /tmp]# chkconfig --list nginx
nginx           0:关闭  1:关闭  2:启用  3:启用  4:启用  5:启用  6:关闭
[root@ZhongH /tmp]# sed -i '$ i \\tinclude vhost/*.conf;' /usr/local/nginx/conf/nginx.conf
[root@ZhongH /tmp]# mkdir -p /usr/local/nginx/conf/vhost

四、配置node1和node2上web服务

1、节点1 配置虚拟主机

[root@ZhongH100 /tmp]# mkdir -p /home/wwwroot/ZhongH100.wxjr.com.cn
[root@ZhongH100 /tmp]# cat /usr/local/nginx/conf/vhost/ZhongH100.wxjr.com.cn.conf 我的节点2虚拟主机配置
server {
        listen 80;
        server_name ZhongH100.wxjr.com.cn;
        root /home/wwwroot/ZhongH100.wxjr.com.cn;
        index index.html index.php index.htm;
        access_log /tmp/nginx/ZhongH100.wxjr.com.cn_nginx.log combined;
}
[root@ZhongH100 /tmp]# echo "Hello node1" >> /home/wwwroot/ZhongH100.wxjr.com.cn/index.html
[root@ZhongH100 /tmp]# /etc/init.d/nginx configtest
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@ZhongH100 /tmp]# /etc/init.d/nginx restart
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
停止 nginx:                                               [失败]
正在启动 nginx:                                           [确定]
[root@ZhongH100 /tmp]# ss -tnl | grep 80
LISTEN     0      511                       *:80                       *:*     
[root@ZhongH100 ~]# sed -i '/<h1>Welcome to nginx!<\/h1>/a <h1> This is node1!</h1>' /usr/local/nginx/html/index.html

2、节点2 配置虚拟主机

[root@ZhongH101 /tmp]# mkdir -p /home/wwwroot/ZhongH101.wxjr.com.cn
[root@ZhongH101 /tmp]# cat /usr/local/nginx/conf/vhost/ZhongH101.wxjr.com.cn.conf 我的节点2虚拟主机配置
server {
        listen 80;
        server_name ZhongH101.wxjr.com.cn;
        root /home/wwwroot/ZhongH101.wxjr.com.cn;
        index index.html index.php index.htm;
        access_log /tmp/nginx/ZhongH101.wxjr.com.cn_nginx.log combined;
}
[root@ZhongH101 /tmp]# echo "Hello node2" >> /home/wwwroot/ZhongH101.wxjr.com.cn/index.html
[root@ZhongH101 /tmp]# /etc/init.d/nginx configtest
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@ZhongH101 /tmp]# /etc/init.d/nginx restart
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
停止 nginx:                                               [确定]
正在启动 nginx:                                           [确定]
[root@ZhongH101 /tmp]# ss -tnl | grep 80
LISTEN     0      511                       *:80                       *:* 
[root@ZhongH101 ~]# sed -i '/<h1>Welcome to nginx!<\/h1>/a <h1> This is node2!</h1>' /usr/local/nginx/html/index.html

检测下两个节点的web服务

Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用
Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用
Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用


五、配置node1和node2上Heartbeat
1、拷贝Heartbeat所需要的配置文件到Heartbeat的配置文件目录

[root@ZhongH100 /tmp]# cd /etc/ha.d/
[root@ZhongH100 /etc/ha.d]# cd /usr/share/doc/heartbeat-*
[root@ZhongH100 /usr/share/doc/heartbeat-3.0.4]# cd -
/etc/ha.d
[root@ZhongH100 /etc/ha.d]# cp /usr/share/doc/heartbeat-*/{authkeys,ha.cf,haresources} ./
#authkeys #是节点之间的认证key文件,我们不能让什么服务器都加入集群中来,加入集群中的节点都是需要认证的
#ha.cf #heartbeat的主配置文件
#haresources #集群资源管理配置文件(在heartbeat所有版本中都是支持haresources来配置集群中的资源的)

2、配置authkeys文件

[root@ZhongH100 /etc/ha.d]# echo -e "auth 1\n1 md5 $(cat /dev/urandom | head | openssl md5 | awk '{print $2}')" >> /etc/ha.d/authkeys
[root@ZhongH100 /etc/ha.d]# tail /etc/ha.d/authkeys
#
#       crc adds no security, except from packet corruption.
#               Use only on physically secure networks.
#
#auth 1
#1 crc
#2 sha1 HI!
#3 md5 Hello!
auth 1
1 md5 a21763677359cdeac3fd8cb1d5e79a1a
[root@ZhongH100 /etc/ha.d]# chmod 600 /etc/ha.d/authkeys
[root@ZhongH100 /etc/ha.d]# 

3、修改ha.cf配置文件,下面是我的配置范本,修改心跳信息的传播方式(这里是组播)

[root@ZhongH100 /etc/ha.d]# grep -Ev '^#|^$' ha.cf
logfacility     local0
mcast eth0 225.172.16.1 694 1 0 #mcast 组播通讯,参数如下: 通讯所用的接口 绑定的组播IP(224.0.0.0-239.255.255.255)通讯端口 ttl 是否允许环回。
auto_failback on
node ZhongH100.wxjr.com.cn #配置集群中的节点数
node ZhongH101.wxjr.com.cn #配置集群中的节点数

4、配置haresources文件,下面是我的配置范本

[root@ZhongH100 /etc/ha.d]# echo "ZhongH100.wxjr.com.cn IPaddr::172.16.7.200/16/eth0 nginx" >> haresources
[root@ZhongH100 /etc/ha.d]# grep -Ev '^#|^$' /etc/ha.d/haresources
ZhongH100.wxjr.com.cn IPaddr::172.16.7.200/16/eth0 nginx

5、将node1节点上的heartbeat配置文件传到node2节点上

[root@ZhongH100 /etc/ha.d]# scp authkeys ha.cf haresources ZhongH101.wxjr.com.cn:/etc/ha.d/
authkeys                                                                                                            100%  691     0.7KB/s   00:00
ha.cf                                                                                                               100%   10KB  10.3KB/s   00:00
haresources                                                                                                         100% 5962     5.8KB/s   00:00
[root@ZhongH100 /etc/ha.d]# ssh ZhongH101.wxjr.com.cn "ls -l /etc/ha.d" #通过ssh命令查看node2节点上的文件
总用量 44
-rw------- 1 root root   691 5月  28 22:25 authkeys
-rw-r--r-- 1 root root 10591 5月  28 22:25 ha.cf
-rwxr-xr-x 1 root root   745 12月  3 2013 harc
-rw-r--r-- 1 root root  5962 5月  28 22:25 haresources
drwxr-xr-x 2 root root  4096 5月  28 17:58 rc.d
-rw-r--r-- 1 root root   692 12月  3 2013 README.config
drwxr-xr-x 2 root root  4096 5月  28 17:58 resource.d
-rw-r--r-- 1 root root  2082 5月   6 18:52 shellfuncs
[root@ZhongH100 /etc/ha.d]# 

6、启动节点上的heartbeat,上面我们已经对node1和node2做了ssh互信,这里就直接在node1上用ssh远程执行命令来控制node2

[root@ZhongH100 /etc/ha.d]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO:  Resource is stopped
Done.

[root@ZhongH100 /etc/ha.d]# ssh ZhongH101.wxjr.com.cn "/etc/init.d/heartbeat start"
Starting High-Availability services: 2015/05/28_22:47:02 INFO:  Resource is stopped
Done.

[root@ZhongH100 /etc/ha.d]# 

六、测试Web集群
1、查看服务启动情况

[root@ZhongH100 ~]# netstat -ntulp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2015/master
tcp        0      0 0.0.0.0:63705               0.0.0.0:*                   LISTEN      1581/rpc.statd
tcp        0      0 127.0.0.1:6010              0.0.0.0:*                   LISTEN      2588/sshd
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1456/rpcbind
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      2157/nginx
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1781/sshd
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1610/cupsd
tcp        0      0 ::1:25                      :::*                        LISTEN      2015/master
tcp        0      0 ::1:6010                    :::*                        LISTEN      2588/sshd
tcp        0      0 :::19454                    :::*                        LISTEN      1581/rpc.statd
tcp        0      0 :::111                      :::*                        LISTEN      1456/rpcbind
tcp        0      0 :::22                       :::*                        LISTEN      1781/sshd
tcp        0      0 ::1:631                     :::*                        LISTEN      1610/cupsd
udp        0      0 127.0.0.1:909               0.0.0.0:*                               1581/rpc.statd
udp        0      0 0.0.0.0:783                 0.0.0.0:*                               1456/rpcbind
udp        0      0 0.0.0.0:50097               0.0.0.0:*                               1581/rpc.statd
udp        0      0 225.172.16.1:694            0.0.0.0:*                               1928/heartbeat: wri
udp        0      0 0.0.0.0:46815               0.0.0.0:*                               1928/heartbeat: wri
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               1328/dhclient
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1456/rpcbind
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1610/cupsd
udp        0      0 :::783                      :::*                                    1456/rpcbind
udp        0      0 :::41894                    :::*                                    1581/rpc.statd
udp        0      0 :::111                      :::*                                    1456/rpcbind
[root@ZhongH100 ~]# 

2、检查web页面
Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用
3、故障演示
1)、关闭node1节点上的heartbeat

[root@ZhongH100 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services: Done.

[root@ZhongH100 ~]# 

查看node1节点上的ip

[root@ZhongH100 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e6:29:99 brd ff:ff:ff:ff:ff:ff
    inet 172.16.6.100/16 brd 172.16.255.255 scope global eth0
    inet6 fe80::20c:29ff:fee6:2999/64 scope link
       valid_lft forever preferred_lft forever
3: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether da:7b:ee:d4:f3:0b brd ff:ff:ff:ff:ff:ff
[root@ZhongH100 ~]# 

查看node2节点上的ip

[root@ZhongH101 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:14:0f:ea brd ff:ff:ff:ff:ff:ff
    inet 172.16.6.101/16 brd 172.16.255.255 scope global eth0
    inet 172.16.7.200/16 brd 172.16.255.255 scope global secondary eth0
    inet6 fe80::20c:29ff:fe14:fea/64 scope link
       valid_lft forever preferred_lft forever
3: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether f2:d1:dc:4f:50:3a brd ff:ff:ff:ff:ff:ff
[root@ZhongH101 ~]# 

Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用
由此可见 已经转移到node2节点上了,大家可以在这是把node1的Heartbeat启动起来再看看,这里就不再演示了
Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用
Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用
2)、再次访问http://172.16.7.200
Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用
由此可见集群已经转移到node2上,从而实现了Web高可用。


七、架构问题扩展和追问以及解决方案

可以从上面的实验中可以看出,此集群架构存在一重要问题?
当主服器故障了或说宕机了,服务(这里是80端口)和VIP立即切换到从服务器上继续提供服务,这是没有问题的,若是这个集群提供的是一个可以上传的web服务,一用户刚刚上传一文件到主服务器上,这时主服务器宕了,用户一刷新发现刚刚上传的文件没有了,用户会怎么想呢?这会带来用户体验的缺失。

那就有人会问了,那有没有什么方法解决这个问题呢?答案是肯定有的,基本上有两种解决方案,
一种是各节点之间进行文件同步,另一种是各节点之间使用共享存储。下面我们就来说一说这两种方案。

第一种方案,我们说了使用节点间的文件同步,我们一般用Rsync+Inotify组合方案来解决节点之间的同步问题,
但有个问题,我们说有极端一点,还是刚才上传文件的问题,用户刚开始上传文件,这时主服务器宕机了,还没有同步,用户看不到文件,还有一点情况,用户上传的文件比较大,各节点之间正在同步,这时主服务器宕机了,用户看到的文件是不完整,也不能达到很好的用户体验。而且节点之间同步,还会占用大量的带宽。说了这么多,我们总结一下,这种方案的优缺点吧,
优点:可节点一台或多台服务器,节约成本,在小规模的集群中,文件同步效果还是不错的,在大规模的集群服务器不推荐使用,
缺点:节点之间同步文件,占用大量的网络带宽,降低整体集群性能,在比较烦忙的集群中不推荐使用。

第二种方案,就是我们所说的共享存储方案了,共享存储方案也有两种方案,
一种是文件级别的共享存储(如,NFS文件服务器),另一种是块级别的共享存储(如,iscsi共享存储)。
用这种方案就能很好的解决上述的问题,上面的集群中有两个节点,主节点与从节点共享一个文件服务器,当主节点提供服务时,文件服务器是挂载在主节点上,当主节点宕机了,从节点来提供服务,并挂载同一个文件服务器,这时我们就不用担心当主节点宕机了用户没有上传完的文件没有了,因为我们使用的同一共享存储,只要文件上传到时服务器中,我们就能看,不用担心文件不存在,或说得同步文件的问题了。在高可用集群中我比较推荐共享存储。下面我们就用NFS来演示一下各节点之间挂载共享存储。


八、共享存储配置
1、配置NFS服务器

[root@ZhongH102 ~]# mkdir -pv /home/wwwroot/data
mkdir: 已创建目录 "/home/wwwroot"
mkdir: 已创建目录 "/home/wwwroot/data"
[root@ZhongH102 ~]# echo "/home/wwwroot/data 172.16.0.0/8(ro,async)" >> /etc/exports
[root@ZhongH102 ~]# echo '<h1>Hello This is NFS Server</h1>' > /home/wwwroot/data/index.html
[root@ZhongH102 ~]# /etc/init.d/portmap start
bash: /etc/init.d/portmap: 没有那个文件或目录
[root@ZhongH102 ~]# /etc/init.d/portreserve start
正在启动 portreserve:                                     [确定]
[root@ZhongH102 ~]# /etc/init.d/nfs start
启动 NFS 服务:                                            [确定]
关掉 NFS 配额:                                            [确定]
启动 NFS mountd:                                          [确定]
启动 NFS 守护进程:                                        [确定]
正在启动 RPC idmapd:                                      [确定]
[root@ZhongH102 ~]# showmount -e 172.16.6.102
Export list for 172.16.6.102:
/home/wwwroot/data 172.16.0.0/8
[root@ZhongH102 ~]# 

2、节点测试挂载
1)、节点1

[root@ZhongH100 ~]# mount -t nfs 172.16.6.102:/home/wwwroot/data/ /usr/local/nginx/html
[root@ZhongH100 ~]# mount
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vgzhongH-data on /data type ext4 (rw,acl)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
172.16.6.102:/home/wwwroot/data/ on /usr/local/nginx/html type nfs (rw,vers=4,addr=172.16.6.102,clientaddr=172.16.6.100)
[root@ZhongH100 ~]# umount /usr/local/nginx/html/
[root@ZhongH100 ~]# mount
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vgzhongH-data on /data type ext4 (rw,acl)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
[root@ZhongH100 ~]# 

2)、节点2

[root@ZhongH101 ~]# mount -t nfs 172.16.6.102:/home/wwwroot/data/ /usr/local/nginx/html
[root@ZhongH101 ~]# mount
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vgzhongH-data on /data type ext4 (rw,acl)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
172.16.6.102:/home/wwwroot/data/ on /usr/local/nginx/html type nfs (rw,vers=4,addr=172.16.6.102,clientaddr=172.16.6.101)
[root@ZhongH101 ~]# umount /usr/local/nginx/html/
[root@ZhongH101 ~]# mount
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vgzhongH-data on /data type ext4 (rw,acl)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
[root@ZhongH101 ~]# 

3、修改node1和node2两节点上的haresources文件
1)、将haresources修改为下面的格式

[root@ZhongH100 ~]# grep -v '^#' /etc/ha.d/haresources
ZhongH100.wxjr.com.cn IPaddr::172.16.7.200/8/eth0 Filesystem::172.16.6.102:/home/wwwroot/data::/usr/local/nginx/html::nfs nginx
[root@ZhongH100 ~]# 

2)、将node1节点上的haresources文件同步到node2节点上

[root@ZhongH100 ~]# scp /etc/ha.d/haresources 172.16.6.101:/etc/ha.d/
haresources                                                                                                         100% 6028     5.9KB/s   00:00
[root@ZhongH100 ~]# 

4、重启heartbeat

[root@ZhongH100 ~]# ssh 172.16.6.101 "/etc/init.d/heartbeat restart"
Stopping High-Availability services: Done.

Waiting to allow resource takeover to complete:Done.

Starting High-Availability services: 2015/05/29_00:04:32 INFO:  Resource is stopped
Done.

[root@ZhongH100 ~]# /etc/init.d/heartbeat restart
Stopping High-Availability services: Done.

Waiting to allow resource takeover to complete:Done.

Starting High-Availability services: INFO:  Resource is stopped
Done.

[root@ZhongH100 ~]# 

5、检测端口和挂载

[root@ZhongH100 ~]# netstat -ntulp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:49752               0.0.0.0:*                   LISTEN      -
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2015/master
tcp        0      0 0.0.0.0:63705               0.0.0.0:*                   LISTEN      1581/rpc.statd
tcp        0      0 127.0.0.1:6010              0.0.0.0:*                   LISTEN      2588/sshd
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1456/rpcbind
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      8433/nginx
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1781/sshd
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1610/cupsd
tcp        0      0 ::1:25                      :::*                        LISTEN      2015/master
tcp        0      0 ::1:6010                    :::*                        LISTEN      2588/sshd
tcp        0      0 :::19454                    :::*                        LISTEN      1581/rpc.statd
tcp        0      0 :::26303                    :::*                        LISTEN      -
tcp        0      0 :::111                      :::*                        LISTEN      1456/rpcbind
tcp        0      0 :::22                       :::*                        LISTEN      1781/sshd
tcp        0      0 ::1:631                     :::*                        LISTEN      1610/cupsd
udp        0      0 0.0.0.0:62075               0.0.0.0:*                               7924/heartbeat: wri
udp        0      0 127.0.0.1:909               0.0.0.0:*                               1581/rpc.statd
udp        0      0 0.0.0.0:783                 0.0.0.0:*                               1456/rpcbind
udp        0      0 0.0.0.0:50097               0.0.0.0:*                               1581/rpc.statd
udp        0      0 225.172.16.1:694            0.0.0.0:*                               7924/heartbeat: wri
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               1328/dhclient
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1456/rpcbind
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1610/cupsd
udp        0      0 :::783                      :::*                                    1456/rpcbind
udp        0      0 :::41894                    :::*                                    1581/rpc.statd
udp        0      0 :::111                      :::*                                    1456/rpcbind
[root@ZhongH100 ~]# ssh 172.16.6.101 "netstat -ntulp"
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2010/master
tcp        0      0 0.0.0.0:63449               0.0.0.0:*                   LISTEN      1589/rpc.statd
tcp        0      0 127.0.0.1:6010              0.0.0.0:*                   LISTEN      2257/sshd
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1464/rpcbind
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1789/sshd
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1618/cupsd
tcp        0      0 ::1:25                      :::*                        LISTEN      2010/master
tcp        0      0 ::1:6010                    :::*                        LISTEN      2257/sshd
tcp        0      0 :::56044                    :::*                        LISTEN      1589/rpc.statd
tcp        0      0 :::111                      :::*                        LISTEN      1464/rpcbind
tcp        0      0 :::22                       :::*                        LISTEN      1789/sshd
tcp        0      0 ::1:631                     :::*                        LISTEN      1618/cupsd
udp        0      0 0.0.0.0:36225               0.0.0.0:*                               1589/rpc.statd
udp        0      0 127.0.0.1:917               0.0.0.0:*                               1589/rpc.statd
udp        0      0 0.0.0.0:791                 0.0.0.0:*                               1464/rpcbind
udp        0      0 225.172.16.1:694            0.0.0.0:*                               5316/heartbeat: wri
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               1336/dhclient
udp        0      0 0.0.0.0:28508               0.0.0.0:*                               5316/heartbeat: wri
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1464/rpcbind
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1618/cupsd
udp        0      0 :::791                      :::*                                    1464/rpcbind
udp        0      0 :::111                      :::*                                    1464/rpcbind
udp        0      0 :::35954                    :::*                                    1589/rpc.statd
[root@ZhongH100 ~]# mount
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vgzhongH-data on /data type ext4 (rw,acl)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
172.16.6.102:/home/wwwroot/data on /usr/local/nginx/html type nfs (rw,vers=4,addr=172.16.6.102,clientaddr=172.16.6.100)
[root@ZhongH100 ~]# ssh 172.16.6.101 "mount"
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vgzhongH-data on /data type ext4 (rw,acl)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
[root@ZhongH100 ~]# 

6、故障演示

[root@ZhongH100 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services: Done.

[root@ZhongH100 ~]# netstat -ntulp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2015/master
tcp        0      0 0.0.0.0:63705               0.0.0.0:*                   LISTEN      1581/rpc.statd
tcp        0      0 127.0.0.1:6010              0.0.0.0:*                   LISTEN      2588/sshd
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1456/rpcbind
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1781/sshd
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1610/cupsd
tcp        0      0 ::1:25                      :::*                        LISTEN      2015/master
tcp        0      0 ::1:6010                    :::*                        LISTEN      2588/sshd
tcp        0      0 :::19454                    :::*                        LISTEN      1581/rpc.statd
tcp        0      0 :::111                      :::*                        LISTEN      1456/rpcbind
tcp        0      0 :::22                       :::*                        LISTEN      1781/sshd
tcp        0      0 ::1:631                     :::*                        LISTEN      1610/cupsd
udp        0      0 127.0.0.1:909               0.0.0.0:*                               1581/rpc.statd
udp        0      0 0.0.0.0:783                 0.0.0.0:*                               1456/rpcbind
udp        0      0 0.0.0.0:50097               0.0.0.0:*                               1581/rpc.statd
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               1328/dhclient
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1456/rpcbind
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1610/cupsd
udp        0      0 :::783                      :::*                                    1456/rpcbind
udp        0      0 :::41894                    :::*                                    1581/rpc.statd
udp        0      0 :::111                      :::*                                    1456/rpcbind
[root@ZhongH100 ~]# ssh 172.16.6.101 "mount"
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vgzhongH-data on /data type ext4 (rw,acl)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
172.16.6.102:/home/wwwroot/data on /usr/local/nginx/html type nfs (rw,vers=4,addr=172.16.6.102,clientaddr=172.16.6.101)
[root@ZhongH100 ~]# 

7、查看web
Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用
8、查看系统日志

[root@ZhongH100 ~]# tail /var/log/messages
May 29 00:12:58 ZhongH100 heartbeat: [7905]: info: killing HBFIFO process 7923 with signal 15
May 29 00:12:58 ZhongH100 heartbeat: [7905]: info: killing HBWRITE process 7924 with signal 15
May 29 00:12:58 ZhongH100 heartbeat: [7905]: info: killing HBREAD process 7925 with signal 15
May 29 00:12:58 ZhongH100 heartbeat: [7905]: info: Core process 7923 exited. 3 remaining
May 29 00:12:58 ZhongH100 heartbeat: [7905]: info: Core process 7924 exited. 2 remaining
May 29 00:12:58 ZhongH100 heartbeat: [7905]: info: Core process 7925 exited. 1 remaining
May 29 00:12:58 ZhongH100 heartbeat: [7905]: info: zhongh100.wxjr.com.cn Heartbeat shutdown complete.
May 29 00:13:02 ZhongH100 dhclient[1328]: DHCPREQUEST on eth0 to 172.16.0.1 port 67 (xid=0x1014dbde)
May 29 00:13:02 ZhongH100 dhclient[1328]: DHCPACK from 172.16.0.1 (xid=0x1014dbde)
May 29 00:13:04 ZhongH100 dhclient[1328]: bound to 172.16.6.100 -- renewal in 722 seconds.
[root@ZhongH100 ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO:  Resource is stopped
Done.

[root@ZhongH100 ~]# tail /var/log/messages
May 29 00:15:09 ZhongH100 harc(default)[9023]: info: Running /etc/ha.d//rc.d/status status
May 29 00:15:10 ZhongH100 heartbeat: [9003]: info: Comm_now_up(): updating status to active
May 29 00:15:10 ZhongH100 heartbeat: [9003]: info: Local status now set to: 'active'
May 29 00:15:10 ZhongH100 heartbeat: [9003]: info: remote resource transition completed.
May 29 00:15:10 ZhongH100 heartbeat: [9003]: info: remote resource transition completed.
May 29 00:15:10 ZhongH100 heartbeat: [9003]: info: Local Resource acquisition completed. (none)
May 29 00:15:10 ZhongH100 heartbeat: [9003]: info: zhongh101.wxjr.com.cn wants to go standby [foreign]
May 29 00:15:11 ZhongH100 heartbeat: [9003]: info: standby: acquire [foreign] resources from zhongh101.wxjr.com.cn
May 29 00:15:11 ZhongH100 heartbeat: [9041]: info: acquire local HA resources (standby).
May 29 00:15:12 ZhongH100 ResourceManager(default)[9054]: info: Acquiring resource group: zhongh100.wxjr.com.cn IPaddr::172.16.7.200/8/eth0 Filesystem::172.16.6.102:/home/wwwroot/data::/usr/local/nginx/html::nfs nginx
[root@ZhongH100 ~]# 

 

[root@ZhongH100 ~]# ssh 172.16.6.101 "tail /var/log/messages"
May 29 00:15:11 ZhongH101 /usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_172.16.6.102:/home/wwwroot/data)[6953]: INFO:  Success
May 29 00:15:11 ZhongH101 ResourceManager(default)[6897]: info: Running /etc/ha.d/resource.d/IPaddr 172.16.7.200/8/eth0 stop
May 29 00:15:11 ZhongH101 IPaddr(IPaddr_172.16.7.200)[7079]: INFO: IP status = ok, IP_CIP=
May 29 00:15:11 ZhongH101 /usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.16.7.200)[7053]: INFO:  Success
May 29 00:15:11 ZhongH101 heartbeat: [6884]: info: foreign HA resource release completed (standby).
May 29 00:15:11 ZhongH101 heartbeat: [5311]: info: Local standby process completed [foreign].
May 29 00:15:12 ZhongH101 heartbeat: [5311]: WARN: 1 lost packet(s) for [zhongh100.wxjr.com.cn] [11:13]
May 29 00:15:12 ZhongH101 heartbeat: [5311]: info: remote resource transition completed.
May 29 00:15:12 ZhongH101 heartbeat: [5311]: info: No pkts missing from zhongh100.wxjr.com.cn!
May 29 00:15:12 ZhongH101 heartbeat: [5311]: info: Other node completed standby takeover of foreign resources.
[root@ZhongH100 ~]# 

经过上面的几步可以发现,实验成功,不论node1和node2谁挂掉了,我们的web服务都是正常的,实现了Web的高可用。


九、安装MariaDB实现WEB+SQL高可用

1、在NFS服务器上再加个NFS共享

(这里在NFS上做LVM分区就不多说了 不会的可以看我之前LVM的博文 点击去看LVM

[root@ZhongH102 ~]# df -hP
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vgzhongH-root   30G  3.3G   25G  12% /
tmpfs                      932M     0  932M   0% /dev/shm
/dev/sda1                  477M   34M  418M   8% /boot
/dev/mapper/vgzhongH-data  4.8G   10M  4.6G   1% /data
[root@ZhongH102 ~]# echo "/data 172.16.0.0/16(rw,all_squash,anonuid=1500,anongid=1500)" >> /etc/exports
[root@ZhongH102 ~]# /etc/init.d/portreserve restart
停止 portreserve:
正在启动 portreserve:                                     [确定]
[root@ZhongH102 ~]# /etc/init.d/nfs restart
关闭 NFS 守护进程:                                        [确定]
关闭 NFS mountd:                                          [确定]
关闭 NFS quotas:                                          [确定]
关闭 NFS 服务:                                            [确定]
Shutting down RPC idmapd:                                  [确定]
启动 NFS 服务:                                            [确定]
关掉 NFS 配额:                                            [确定]
启动 NFS mountd:                                          [确定]
启动 NFS 守护进程:                                        [确定]
正在启动 RPC idmapd:                                      [确定]
[root@ZhongH102 ~]# showmount -e 172.16.6.102
Export list for 172.16.6.102:
/data              172.16.0.0/16
/home/wwwroot/data 172.16.0.0/8
[root@ZhongH102 ~]# groupadd -g 1500 mysql && useradd -g mysql -u 1500 -s /sbin/nologin -M mysql #node1 node2 NFS3台服务器的mysql用户UID GID都要一样
[root@ZhongH102 ~]# tail -1 /etc/passwd
mysql:x:1500:1500::/home/mysql:/sbin/nologin
[root@ZhongH102 ~]# chown -R mysql.mysql /data/
[root@ZhongH102 ~]# 

2、node1安装MariaDB

[root@ZhongH100 ~]# yum groupinstall "Development tools" "Server Platform Development" -y
[root@ZhongH100 ~]# cd /tmp/ && wget http://www.cmake.org/files/v3.2/cmake-3.2.2.tar.gz
[root@ZhongH101 /tmp]# tar xf cmake-3.2.2.tar.gz
[root@ZhongH100 /tmp]# cd cmake-3.2.2
[root@ZhongH100 /tmp/cmake-3.2.2]# ./bootstrap
[root@ZhongH100 /tmp/cmake-3.2.2]# [ "$?" = "0" ] && make && make install && which cmake && cd ../
/usr/local/bin/cmake
[root@ZhongH100 /tmp]# wget "https://downloads.mariadb.org/interstitial/mariadb-10.0.19/source/mariadb-10.0.19.tar.gz/from/http%3A//mirrors.opencas.cn/mariadb" -O mariadb-10.0.19.tar.gz
[root@ZhongH100 /tmp]# tar xf mariadb-10.0.19.tar.gz
[root@ZhongH100 /tmp]# cd mariadb-10.0.19
[root@ZhongH100 /tmp/mariadb-10.0.19]# groupadd -g 1500 mysql && useradd -g mysql -u 1500 -s /sbin/nologin -M mysql #node1 node2 NFS3台服务器的mysql用户UID GID都要一样
[root@ZhongH100 /tmp/mariadb-10.0.19]# tail -1 /etc/passwd
mysql:x:1500:1500::/home/mysql:/sbin/nologin
[root@ZhongH100 /tmp/mariadb-10.0.19]# cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DMYSQL_DATADIR=/data/mysql \
-DWITH_SSL=system \
-DWITH_INNOBASE_STORAGE_ENGINE=1 \
-DWITH_ARCHIVE_STORAGE_ENGINE=1 \
-DWITH_BLACKHOLE_STORAGE_ENGINE=1 \
-DWITH_SPHINX_STORAGE_ENGINE=1 \
-DWITH_ARIA_STORAGE_ENGINE=1 \
-DWITH_XTRADB_STORAGE_ENGINE=1 \
-DWITH_PARTITION_STORAGE_ENGINE=1 \
-DWITH_FEDERATEDX_STORAGE_ENGINE=1 \
-DWITH_MYISAM_STORAGE_ENGINE=1 \
-DWITH_PERFSCHEMA_STORAGE_ENGINE=1 \
-DWITH_EXTRA_CHARSETS=all \
-DWITH_EMBEDDED_SERVER=1 \
-DWITH_READLINE=1 \
-DWITH_ZLIB=system \
-DWITH_LIBWRAP=0 \
-DEXTRA_CHARSETS=all \
-DENABLED_LOCAL_INFILE=1 \
-DMYSQL_UNIX_ADDR=/tmp/mysql.sock \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci #这里的编译参数根据自己的情况适当增减
[root@ZhongH100 /tmp/mariadb-10.0.19]# make -j $(awk '/processor/{i++}END{print i}' /proc/cpuinfo) && make install  && echo $?
[root@ZhongH100 /tmp/mariadb-10.0.19]# mount -t nfs 172.16.6.102:/data/ /data
[root@ZhongH100 /tmp/mariadb-10.0.19]# mount
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vgzhongH-data on /data type ext4 (rw,acl)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
172.16.6.102:/home/wwwroot/data on /usr/local/nginx/html type nfs (rw,vers=4,addr=172.16.6.102,clientaddr=172.16.6.100)
172.16.6.102:/data/ on /data type nfs (rw,vers=4,addr=172.16.6.102,clientaddr=172.16.6.100)
[root@ZhongH100 /tmp/mariadb-10.0.19]# cd /usr/local/mysql/
[root@ZhongH100 /usr/local/mysql]# cp -a support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@ZhongH100 /usr/local/mysql]# \cp support-files/my-large.cnf /etc/my.cnf
[root@ZhongH100 /usr/local/mysql]# sed -i '/query_cache_size/a datadir = /data/mysql' /etc/my.cnf
[root@ZhongH100 /usr/local/mysql]# mkdir -pv /data/mysql
mkdir: 已创建目录 "/data/mysql"
[root@ZhongH100 /usr/local/mysql]# /usr/local/mysql/scripts/mysql_install_db --user=mysql --datadir=/data/mysql/
[root@ZhongH100 /usr/local/mysql]# service mysqld start
Starting MySQL.                                            [确定]
[root@ZhongH100 /usr/local/mysql]# /usr/local/mysql/bin/mysql <<<"show databases;"
Database
information_schema
mysql
performance_schema
test
[root@ZhongH100 /usr/local/mysql]# scp /etc/rc.d/init.d/mysqld 172.16.6.101:/etc/rc.d/init.d/mysqld
mysqld                                                                                                              100%   12KB  11.9KB/s   00:00
[root@ZhongH100 /usr/local/mysql]# service mysqld stop
Shutting down MySQL..                                      [确定]
[root@ZhongH100 /usr/local/mysql]# cd /tmp/
[root@ZhongH100 /tmp]# scp cmake-3.2.2.tar.gz mariadb-10.0.19.tar.gz 172.16.6.101:/tmp
cmake-3.2.2.tar.gz                                                                                                  100% 6288KB   6.1MB/s   00:00
mariadb-10.0.19.tar.gz                                                                                              100%   54MB  53.6MB/s   00:01
[root@ZhongH100 /tmp]# 

3、node2安装MariaDB

[root@ZhongH101 ~]# yum groupinstall "Development tools" "Server Platform Development" -y
[root@ZhongH101 ~]# cd /tmp
[root@ZhongH101 /tmp]# tar xf cmake-3.2.2.tar.gz
[root@ZhongH101 /tmp]# cd cmake-3.2.2
[root@ZhongH101 /tmp/cmake-3.2.2]# ./bootstrap
[root@ZhongH101 /tmp/cmake-3.2.2]# [ "$?" = "0" ] && make && make install && which cmake && cd ../
[root@ZhongH101 /tmp]# tar xf mariadb-10.0.19.tar.gz
[root@ZhongH101 /tmp]# cd mariadb-10.0.19
[root@ZhongH101 /tmp/mariadb-10.0.19]# groupadd -g 1500 mysql && useradd -g mysql -u 1500 -s /sbin/nologin -M mysql #node1 node2 NFS3台服务器的mysql用户UID GID都要一样
[root@ZhongH101 /tmp/mariadb-10.0.19]# tail -1 /etc/passwd
mysql:x:1500:1500::/home/mysql:/sbin/nologin
[root@ZhongH101 /tmp/mariadb-10.0.19]# cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DMYSQL_DATADIR=/data/mysql \
-DWITH_SSL=system \
-DWITH_INNOBASE_STORAGE_ENGINE=1 \
-DWITH_ARCHIVE_STORAGE_ENGINE=1 \
-DWITH_BLACKHOLE_STORAGE_ENGINE=1 \
-DWITH_SPHINX_STORAGE_ENGINE=1 \
-DWITH_ARIA_STORAGE_ENGINE=1 \
-DWITH_XTRADB_STORAGE_ENGINE=1 \
-DWITH_PARTITION_STORAGE_ENGINE=1 \
-DWITH_FEDERATEDX_STORAGE_ENGINE=1 \
-DWITH_MYISAM_STORAGE_ENGINE=1 \
-DWITH_PERFSCHEMA_STORAGE_ENGINE=1 \
-DWITH_EXTRA_CHARSETS=all \
-DWITH_EMBEDDED_SERVER=1 \
-DWITH_READLINE=1 \
-DWITH_ZLIB=system \
-DWITH_LIBWRAP=0 \
-DEXTRA_CHARSETS=all \
-DENABLED_LOCAL_INFILE=1 \
-DMYSQL_UNIX_ADDR=/tmp/mysql.sock \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci #这里的编译参
[root@ZhongH101 /tmp/mariadb-10.0.19]# mount -t nfs 172.16.6.102:/data/ /data
[root@ZhongH101 /tmp/mariadb-10.0.19]# mount
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vgzhongH-data on /data type ext4 (rw,acl)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
172.16.6.102:/data/ on /data type nfs (rw,vers=4,addr=172.16.6.102,clientaddr=172.16.6.101)
[root@ZhongH101 /tmp/mariadb-10.0.19]# ls -l /data/
总用量 20
drwx------ 2 mysql mysql 16384 5月  24 03:28 lost+found
drwxr-xr-x 5 mysql mysql  4096 5月  29 12:54 mysql
[root@ZhongH101 /usr/local/mysql]# scp 172.16.6.100:/etc/my.cnf /etc/my.cnf
my.cnf                                                                                                              100% 4903     4.8KB/s   00:00
[root@ZhongH101 /usr/local/mysql]# service mysqld start
Starting MySQL.                                            [确定]
[root@ZhongH101 /usr/local/mysql]# ss -tnl | grep 3306
LISTEN     0      150                      :::3306                    :::*
[root@ZhongH101 /usr/local/mysql]# service mysqld stop
Shutting down MySQL..                                      [确定]
[root@ZhongH101 /usr/local/mysql]# 

3、修改haresources配置

[root@ZhongH /usr/local/mysql]# grep -v '^#' /etc/ha.d/haresources
ZhongH100.wxjr.com.cn IPaddr::172.16.7.200/16/eth0 Filesystem::172.16.6.102:/home/wwwroot/data::/usr/local/nginx/html::nfs Filesystem::172.16.6.102:/data::/data::nfs mysqld nginx
[root@ZhongH100 /tmp]# /etc/init.d/heartbeat restart
Stopping High-Availability services: Done.

Waiting to allow resource takeover to complete:Done.

Starting High-Availability services: INFO:  Resource is stopped
Done.

[root@ZhongH100 /tmp]# ssh root@172.16.6.101 "/etc/init.d/heartbeat restart"
Stopping High-Availability services: Done.

Waiting to allow resource takeover to complete:Done.

Starting High-Availability services: 2015/05/29_14:03:00 INFO:  Resource is stopped
Done.

4、查看结果

[root@ZhongH100 /tmp]# netstat -utnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2015/master
tcp        0      0 0.0.0.0:63705               0.0.0.0:*                   LISTEN      1581/rpc.statd
tcp        0      0 127.0.0.1:6010              0.0.0.0:*                   LISTEN      13048/sshd
tcp        0      0 0.0.0.0:27108               0.0.0.0:*                   LISTEN      -
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1456/rpcbind
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      53485/nginx
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      17419/sshd
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1610/cupsd
tcp        0      0 ::1:25                      :::*                        LISTEN      2015/master
tcp        0      0 ::1:6010                    :::*                        LISTEN      13048/sshd
tcp        0      0 :::19454                    :::*                        LISTEN      1581/rpc.statd
tcp        0      0 :::3306                     :::*                        LISTEN      53333/mysqld
tcp        0      0 :::111                      :::*                        LISTEN      1456/rpcbind
tcp        0      0 :::38934                    :::*                        LISTEN      -
tcp        0      0 :::22                       :::*                        LISTEN      17419/sshd
tcp        0      0 ::1:631                     :::*                        LISTEN      1610/cupsd
udp        0      0 127.0.0.1:909               0.0.0.0:*                               1581/rpc.statd
udp        0      0 0.0.0.0:783                 0.0.0.0:*                               1456/rpcbind
udp        0      0 0.0.0.0:50097               0.0.0.0:*                               1581/rpc.statd
udp        0      0 225.172.16.1:694            0.0.0.0:*                               52614/heartbeat: wr
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               1328/dhclient
udp        0      0 0.0.0.0:44624               0.0.0.0:*                               52614/heartbeat: wr
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1456/rpcbind
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1610/cupsd
udp        0      0 :::783                      :::*                                    1456/rpcbind
udp        0      0 :::41894                    :::*                                    1581/rpc.statd
udp        0      0 :::111                      :::*                                    1456/rpcbind
[root@ZhongH100 /tmp]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e6:29:99 brd ff:ff:ff:ff:ff:ff
    inet 172.16.6.100/16 brd 172.16.255.255 scope global eth0
    inet 172.16.7.200/16 brd 172.16.255.255 scope global secondary eth0
    inet6 fe80::20c:29ff:fee6:2999/64 scope link
       valid_lft forever preferred_lft forever
3: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether da:7b:ee:d4:f3:0b brd ff:ff:ff:ff:ff:ff
[root@ZhongH100 /tmp]# mount
/dev/mapper/vgzhongH-root on / type ext4 (rw,acl)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
172.16.6.102:/home/wwwroot/data on /usr/local/nginx/html type nfs (rw,vers=4,addr=172.16.6.102,clientaddr=172.16.6.100)
172.16.6.102:/data on /data type nfs (rw,vers=4,addr=172.16.6.102,clientaddr=172.16.6.100)
[root@ZhongH100 /tmp]# 

Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用
Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用

5、故障演示

[root@ZhongH100 /tmp]# /etc/init.d/heartbeat stop
Stopping High-Availability services: Done.

[root@ZhongH100 /tmp]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e6:29:99 brd ff:ff:ff:ff:ff:ff
    inet 172.16.6.100/16 brd 172.16.255.255 scope global eth0
    inet6 fe80::20c:29ff:fee6:2999/64 scope link
       valid_lft forever preferred_lft forever
3: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether da:7b:ee:d4:f3:0b brd ff:ff:ff:ff:ff:ff

Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用

[root@ZhongH100 /tmp]# ssh root@172.16.6.101 "netstat -utnlp"
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2010/master
tcp        0      0 0.0.0.0:63449               0.0.0.0:*                   LISTEN      1589/rpc.statd
tcp        0      0 127.0.0.1:6010              0.0.0.0:*                   LISTEN      10386/sshd
tcp        0      0 0.0.0.0:30991               0.0.0.0:*                   LISTEN      -
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1464/rpcbind
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      51991/nginx
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      14298/sshd
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1618/cupsd
tcp        0      0 :::56089                    :::*                        LISTEN      -
tcp        0      0 ::1:25                      :::*                        LISTEN      2010/master
tcp        0      0 ::1:6010                    :::*                        LISTEN      10386/sshd
tcp        0      0 :::3306                     :::*                        LISTEN      51839/mysqld
tcp        0      0 :::56044                    :::*                        LISTEN      1589/rpc.statd
tcp        0      0 :::111                      :::*                        LISTEN      1464/rpcbind
tcp        0      0 :::22                       :::*                        LISTEN      14298/sshd
tcp        0      0 ::1:631                     :::*                        LISTEN      1618/cupsd
udp        0      0 0.0.0.0:36225               0.0.0.0:*                               1589/rpc.statd
udp        0      0 127.0.0.1:917               0.0.0.0:*                               1589/rpc.statd
udp        0      0 0.0.0.0:791                 0.0.0.0:*                               1464/rpcbind
udp        0      0 0.0.0.0:6953                0.0.0.0:*                               51009/heartbeat: wr
udp        0      0 225.172.16.1:694            0.0.0.0:*                               51009/heartbeat: wr
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               1336/dhclient
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1464/rpcbind
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1618/cupsd
udp        0      0 :::791                      :::*                                    1464/rpcbind
udp        0      0 :::111                      :::*                                    1464/rpcbind
udp        0      0 :::35954                    :::*                                    1589/rpc.statd
[root@ZhongH100 /tmp]# ssh root@172.16.6.101 "ip addr"
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:14:0f:ea brd ff:ff:ff:ff:ff:ff
    inet 172.16.6.101/16 brd 172.16.255.255 scope global eth0
    inet 172.16.7.200/16 brd 172.16.255.255 scope global secondary eth0
    inet6 fe80::20c:29ff:fe14:fea/64 scope link
       valid_lft forever preferred_lft forever
3: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether f2:d1:dc:4f:50:3a brd ff:ff:ff:ff:ff:ff

Linux高可用(HA)之Heartbeat+Nginx+MySQL+NFS实现WEB+SQL服务高可用

lookback

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: