Linux高可用(HA)之分布式文件系统之MogileFS部署

评论
6,284 次

摘要

分布式文件系统（Distributed File System）是指文件系统管理的物理存储资源不一定直接连接在本地节点上，而是通过计算机网络与节点相连，也就是集群文件系统，可以支持大数量的节点以及PB级的数量存储。

一、基础知识

分布式存储系统的特性：

可扩展：分布式存储系统可以扩展到几百台至几千台的集群规模，且随着集群规模的增长，系统整体性能表现为线性增长

低成本：分布式存储系统的自动容错、自动负载均衡机制使其可以构建在普通PC机之上；另外，线性扩展能力也使得增加、减少机器非常方便，可以实现自动运维

高性能：无论是针对整个集群还是单台服务器，都要求分布式系统具备高性能

易用：分布式存储系统需要能够提供易用的对外接口；另外，也要求具备完善的监控、运维工具，并能方便的与其他系统集成，如从Hadoop云计算系统导入数据

MogileFS的特性：

工作于应用层：无需特殊的核心组件

无单点：三大组件(tracker，mogstore，database)皆可实现高可用

自动文件复制：复制的最小单位不是文件，而是class；基于不同的class，文件可以被自动的复制到多个有足够存储空间的存储节点上

传输中立，无特殊协议：可以通过NFS或HTTP协议进行通信

简单的命名空间：文件通过一个给定的key来确定，是一个全局的命名空间；没有目录，基于域实现文件隔离

不共享数据：无需通过昂贵的SAN来共享磁盘，每个存储节点只需维护自己所属的存储设备(device)即可

二、实验过程

1、环境介绍

配置store1 store2 SSH互相时间同步主机名互解

[root@Legion100 ~]# echo "172.16.6.100 Legion100.dwhd.org Legion100 node1" >> /etc/hosts
[root@Legion100 ~]# echo "172.16.6.101 Legion101.dwhd.org Legion101 node2" >> /etc/hosts
[root@Legion100 ~]# echo "*/5 * * * * /usr/sbin/ntpdate pool.ntp.org >/dev/null 2>&1" >> /var/spool/cron/root
[root@Legion100 ~]# ssh-keygen  -t rsa -f ~/.ssh/id_rsa  -P ''
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1c:c4:da:a5:de:1e:e5:8c:eb:40:cc:16:53:af:39:f1 root@Legion100.dwhd.org
The key's randomart image is:
+--[ RSA 2048]----+
|       .. .      |
|       .....     |
|       o+o. .    |
|      .+o+ =.    |
|       .S.+=E    |
|       o. +.o    |
|        .. o     |
|         .o      |
|         ..      |
+-----------------+
[root@Legion100 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@Legion101.dwhd.org
The authenticity of host 'legion101.dwhd.org (172.16.6.101)' can't be established.
RSA key fingerprint is a6:bb:83:e4:04:27:55:50:9a:e1:25:5a:7b:01:bc:8c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'legion101.dwhd.org,172.16.6.101' (RSA) to the list of known hosts.
root@legion101.dwhd.org's password:
Now try logging into the machine, with "ssh 'root@Legion101.dwhd.org'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

[root@Legion100 ~]#

[root@Legion101 ~]# echo "172.16.6.101 Legion101.dwhd.org Legion101 node2" >> /etc/hosts
[root@Legion101 ~]# echo "172.16.6.100 Legion100.dwhd.org Legion100 node1" >> /etc/hosts
[root@Legion101 ~]# echo "*/5 * * * * /usr/sbin/ntpdate pool.ntp.org >/dev/null 2>&1" >> /var/spool/cron/root
[root@Legion101 ~]# ssh-keygen  -t rsa -f ~/.ssh/id_rsa  -P ''
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
6b:d5:17:4d:04:80:c4:60:ad:a5:e3:71:e7:2a:2e:62 root@Legion101.dwhd.org
The key's randomart image is:
+--[ RSA 2048]----+
|        o=.....oo|
|       .  =    o |
|         +    . .|
|        = ...  . |
|       .S+.o. .  |
|        .o  ..   |
|        o  .     |
|    E ... .      |
|   . . o..       |
+-----------------+
[root@Legion101 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@Legion100.dwhd.org
The authenticity of host 'legion100.dwhd.org (172.16.6.100)' can't be established.
RSA key fingerprint is 27:4e:d4:ce:88:a2:21:42:32:c5:54:c5:00:d8:e9:d5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'legion100.dwhd.org,172.16.6.100' (RSA) to the list of known hosts.
root@legion100.dwhd.org's password:
Now try logging into the machine, with "ssh 'root@Legion100.dwhd.org'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

[root@Legion101 ~]#

2、安装和MariaDB

1)、安装MariaDB

[root@Legion103 ~]# rpm -ivh http://mirrors.aliyun.com/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@Legion103 ~]# yum clean all && yum makecache
[root@Legion103 ~]# yum update -y
[root@Legion103 ~]# yum groupinstall "Development tools" "Server Platform Development" -y
[root@Legion103 ~]# yum install cmake -y
[root@Legion103 ~]# cd /tmp
[root@Legion103 /tmp]# groupadd -g 1500 mysql && useradd -g mysql -u 1500 -s /sbin/nologin -M mysql
[root@Legion103 /tmp]# wget "https://downloads.mariadb.org/interstitial/mariadb-10.0.19/source/mariadb-10.0.19.tar.gz/from/http%3A//mirrors.opencas.cn/mariadb" -O mariadb-10.0.19.tar.gz
[root@Legion103 /tmp]# tar xf mariadb-10.0.19.tar.gz
[root@Legion103 /tmp]# cd mariadb-10.0.19
[root@Legion103 /tmp/mariadb-10.0.19]# cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DMYSQL_DATADIR=/data/mysql \
-DWITH_SSL=system \
-DWITH_INNOBASE_STORAGE_ENGINE=1 \
-DWITH_ARCHIVE_STORAGE_ENGINE=1 \
-DWITH_BLACKHOLE_STORAGE_ENGINE=1 \
-DWITH_SPHINX_STORAGE_ENGINE=1 \
-DWITH_ARIA_STORAGE_ENGINE=1 \
-DWITH_XTRADB_STORAGE_ENGINE=1 \
-DWITH_PARTITION_STORAGE_ENGINE=1 \
-DWITH_FEDERATEDX_STORAGE_ENGINE=1 \
-DWITH_MYISAM_STORAGE_ENGINE=1 \
-DWITH_PERFSCHEMA_STORAGE_ENGINE=1 \
-DWITH_EXTRA_CHARSETS=all \
-DWITH_EMBEDDED_SERVER=1 \
-DWITH_READLINE=1 \
-DWITH_ZLIB=system \
-DWITH_LIBWRAP=0 \
-DEXTRA_CHARSETS=all \
-DENABLED_LOCAL_INFILE=1 \
-DMYSQL_UNIX_ADDR=/tmp/mysql.sock \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci
[root@Legion103 /tmp/mariadb-10.0.19]# make -j $(awk '/processor/{i++}END{print i}' /proc/cpuinfo) && make install  && echo $?
[root@Legion103 /tmp/mariadb-10.0.19]# cd /usr/local/mysql/
[root@Legion103 /usr/local/mysql]# echo "export PATH=/usr/local/mysql/bin:\$PATH" > /etc/profile.d/mariadb10.0.19.sh
[root@Legion103 /usr/local/mysql]# . /etc/profile.d/mariadb10.0.19.sh
[root@Legion103 /usr/local/mysql]# sed -i "$(awk '$1=="MANPATH"{a=NR}END{print a}' /etc/man.config)a MANPATH\t/usr/local/mysql/man" /etc/man.config
[root@Legion103 /usr/local/mysql]# cp -a support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@Legion103 /usr/local/mysql]# \cp support-files/my-large.cnf /etc/my.cnf
[root@Legion103 /usr/local/mysql]# sed -i '/query_cache_size/a datadir = /data/mysql' /etc/my.cnf
[root@Legion103 /usr/local/mysql]# mkdir -p /data/mysql
[root@Legion103 /usr/local/mysql]# chown -R mysql.mysql /data/mysql
[root@Legion103 /usr/local/mysql]# /usr/local/mysql/scripts/mysql_install_db \
--user=mysql --datadir=/data/mysql/ --basedir=/usr/local/mysql
[root@Legion103 /usr/local/mysql]#

2)、配置

[root@Legion103 /usr/local/mysql]# chkconfig mysqld on
[root@Legion103 /usr/local/mysql]# chkconfig --list mysqld
mysqld          0:关闭  1:关闭  2:启用  3:启用  4:启用  5:启用  6:关闭
[root@Legion103 /usr/local/mysql]# service mysqld start
Starting MySQL.                                            [确定]
[root@Legion103 /usr/local/mysql]# ss -tnlp | grep 3306
LISTEN     0      150                      :::3306                    :::*      users:(("mysqld",23097,21))
[root@Legion103 /usr/local/mysql]#

[root@Legion103 /usr/local/mysql]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 10.0.19-MariaDB-log Source distribution

Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> USE mysql
Database changed
MariaDB [mysql]> UPDATE user set password=PASSWORD('lookback') WHERE USER='root';
Query OK, 4 rows affected (0.04 sec)
Rows matched: 4  Changed: 4  Warnings: 0

MariaDB [mysql]> DELETE FROM user WHERE User='';
Query OK, 2 rows affected (0.00 sec)

MariaDB [mysql]> grant all on mogdb.* to 'moguser'@'172.16.%.%' identified by 'mogpass';
Query OK, 0 rows affected (0.00 sec)

MariaDB [mysql]> GRANT all on *.* to 'root'@'172.16.%.%' identified by 'lookback';
Query OK, 0 rows affected (0.00 sec)

MariaDB [mysql]> SELECT USER,PASSWORD,HOST FROM user;
+---------+-------------------------------------------+--------------------+
| USER    | PASSWORD                                  | HOST               |
+---------+-------------------------------------------+--------------------+
| root    | *153CCFAEAA83407D8DBDBFAA3D17B1A95553E60C | localhost          |
| root    | *153CCFAEAA83407D8DBDBFAA3D17B1A95553E60C | legion103.dwhd.org |
| root    | *153CCFAEAA83407D8DBDBFAA3D17B1A95553E60C | 127.0.0.1          |
| root    | *153CCFAEAA83407D8DBDBFAA3D17B1A95553E60C | ::1                |
| root    | *153CCFAEAA83407D8DBDBFAA3D17B1A95553E60C | 172.16.%.%         |
| moguser | *528176B5D4BA99F614AA161C865930366820832E | 172.16.%.%         |
+---------+-------------------------------------------+--------------------+
6 rows in set (0.00 sec)

MariaDB [mysql]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [mysql]> \q
Bye
[root@Legion103 /usr/local/mysql]#

3、配置存储节点store1

[root@Legion100 ~]# wget http://www.dwhd.org/script/MogileFS_RPM_DownList && wget -i MogileFS_RPM_DownList
[root@Legion100 ~]# yum install *.rpm perl-IO-AIO perl-BSD-Resource \
perl-DBD-MySQL perl-Danga-Socket perl-IO-stringy \
perl-Sys-Syscall perl-common-sense -y
[root@Legion100 ~]# scp *.rpm 172.16.6.101:/tmp/
MogileFS-Server-2.46-2.el6_.noarch.rpm                                                                                           100% 1916     1.9KB/s   00:00
MogileFS-Server-mogilefsd-2.46-2.el6_.noarch.rpm                                                                                 100%  172KB 172.2KB/s   00:00
MogileFS-Server-mogstored-2.46-2.el6_.noarch.rpm                                                                                 100%   26KB  26.2KB/s   00:00
MogileFS-Utils-2.19-1.el6_.noarch.rpm                                                                                            100%   74KB  74.1KB/s   00:00
perl-MogileFS-Client-1.14-1.el6_.noarch.rpm                                                                                      100%   30KB  29.6KB/s   00:00
perl-Net-Netmask-1.9015-8.el6_.noarch.rpm                                                                                        100%   25KB  24.6KB/s   00:00
perl-Perlbal-1.78-1.el6_.noarch.rpm                                                                                              100%  262KB 262.3KB/s   00:00
[root@Legion100 ~]# cd /etc/mogilefs/
[root@Legion100 /etc/mogilefs]# mkdir -p /home/mogilefs
[root@Legion100 /etc/mogilefs]# chown -R mogilefs.mogilefs /home/mogilefs
[root@Legion100 /etc/mogilefs]# cp -a /etc/mogilefs/mogilefsd.conf /etc/mogilefs/mogilefsd.conf_backup_`date "+%Y%m%d%H%M%S"`
[root@Legion100 /etc/mogilefs]# cp -a /etc/mogilefs/mogstored.conf /etc/mogilefs/mogstored.conf_backup_`date "+%Y%m%d%H%M%S"`
[root@Legion100 /etc/mogilefs]# grep -v '^#' /etc/mogilefs/mogilefsd.conf
daemonize = 1
pidfile = /var/run/mogilefsd/mogilefsd.pid
db_dsn = DBI:mysql:mogdb:host=172.16.6.103
db_user = moguser
db_pass = mogpass
listen = 172.16.6.100:7001
conf_port = 7001
query_jobs = 10
delete_jobs = 1
replicate_jobs = 5
reaper_jobs = 1
[root@Legion100 /etc/mogilefs]# cat mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data
[root@Legion100 /etc/mogilefs]# mkdir -p /data/dev1
[root@Legion100 /etc/mogilefs]# chown -R mogilefs.mogilefs /data
[root@Legion100 /etc/mogilefs]# mogdbsetup --dbhost=172.16.6.103 --dbrootuser=root \
--dbrootpass=lookback --dbuser=moguser --dbpass=mogpass --dbname=mogdb --yes
[root@Legion100 /etc/mogilefs]# service mogilefsd start
Starting mogilefsd                                         [确定]
[root@Legion100 /etc/mogilefs]# service mogstored start
Starting mogstored                                         [确定]
[root@Legion100 /etc/mogilefs]#

4、配置存储节点store2

[root@Legion101 ~]# cd /tmp
[root@Legion100 /tmp]# yum install *.rpm perl-IO-AIO perl-BSD-Resource \
perl-DBD-MySQL perl-Danga-Socket perl-IO-stringy \
perl-Sys-Syscall perl-common-sense -y
[root@Legion101 /tmp]# cd /etc/mogilefs/
[root@Legion101 /etc/mogilefs]# mkdir -p /home/mogilefs
[root@Legion101 /etc/mogilefs]# chown -R mogilefs.mogilefs /home/mogilefs
[root@Legion101 /etc/mogilefs]# cp -a /etc/mogilefs/mogilefsd.conf /etc/mogilefs/mogilefsd.conf_backup_`date "+%Y%m%d%H%M%S"`
[root@Legion101 /etc/mogilefs]# cp -a /etc/mogilefs/mogstored.conf /etc/mogilefs/mogstored.conf_backup_`date "+%Y%m%d%H%M%S"`
[root@Legion101 /etc/mogilefs]# grep -v '^#' /etc/mogilefs/mogilefsd.conf
daemonize = 1
pidfile = /var/run/mogilefsd/mogilefsd.pid
db_dsn = DBI:mysql:mogdb:host=172.16.6.103
db_user = moguser
db_pass = mogpass
listen = 172.16.6.101:7001
conf_port = 7001
query_jobs = 10
delete_jobs = 1
replicate_jobs = 5
reaper_jobs = 1
[root@Legion101 /etc/mogilefs]# cat mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data
[root@Legion101 /etc/mogilefs]# mkdir -p /data/dev1
[root@Legion101 /etc/mogilefs]# chown -R mogilefs.mogilefs /data
[root@Legion101 /etc/mogilefs]# mogdbsetup --dbhost=172.16.6.103 --dbrootuser=root \
--dbrootpass=lookback --dbuser=moguser --dbpass=mogpass --dbname=mogdb --yes
[root@Legion101 /etc/mogilefs]# service mogilefsd start
Starting mogilefsd                                         [确定]
[root@Legion101 /etc/mogilefs]# service mogstored start
Starting mogstored                                         [确定]
[root@Legion101 /etc/mogilefs]#

5、给sotre1 和 sotre2 添加节点

[root@Legion100 ~]# mogadm --trackers=172.16.6.101:7001 host add 172.16.6.101 --ip=172.16.6.101 --status=alive
[root@Legion100 ~]# mogadm --trackers=172.16.6.101:7001 host list
172.16.6.101 [1]: alive
  IP:       172.16.6.101:7500

[root@Legion100 ~]#

[root@Legion101 /etc/mogilefs]# mogadm --trackers=172.16.6.100:7001 host add 172.16.6.100 --ip=172.16.6.100 --status=alive
[root@Legion101 /etc/mogilefs]# mogadm --trackers=172.16.6.100:7001 host list
172.16.6.101 [1]: alive
  IP:       172.16.6.101:7500

172.16.6.100 [2]: alive
  IP:       172.16.6.100:7500

[root@Legion101 /etc/mogilefs]#

6、添加设备

[root@Legion101 /etc/mogilefs]# mogadm --trackers=172.16.6.100:7001 device add 172.16.6.100 1
[root@Legion101 /etc/mogilefs]# mogadm --trackers=172.16.6.100:7001 device list
172.16.6.101 [1]: alive
                    used(G)    free(G)   total(G)  weight(%)

172.16.6.100 [2]: alive
                    used(G)    free(G)   total(G)  weight(%)
   dev2:   alive      0.000      0.000      0.000        100

[root@Legion101 /etc/mogilefs]#

[root@Legion100 ~]# mogadm --trackers=172.16.6.101:7001 device add 172.16.6.101 1
[root@Legion100 ~]# mogadm --trackers=172.16.6.101:7001 device list
172.16.6.101 [1]: alive
                    used(G)    free(G)   total(G)  weight(%)
   dev1:   alive      0.010      4.536      4.546        100

172.16.6.100 [2]: alive
                    used(G)    free(G)   total(G)  weight(%)
   dev2:   alive      0.000      0.000      0.000        100

[root@Legion100 ~]#

添加设备时所使用的设备ID号必须要/data/dev#中的编号一致

7、建立区域

[root@Legion100 /etc/mogilefs]# mogadm --trackers=172.16.6.101:7001 domain add images
[root@Legion100 /etc/mogilefs]# mogadm --trackers=172.16.6.101:7001 domain add text
[root@Legion100 /etc/mogilefs]# mogadm --trackers=172.16.6.101:7001 domain list
 domain               class                mindevcount   replpolicy   hashtype
-------------------- -------------------- ------------- ------------ -------
 images               default                   2        MultipleHosts() NONE   

 text                 default                   2        MultipleHosts() NONE   

[root@Legion100 /etc/mogilefs]#

8、在每一个区域中建立class

[root@Legion100 /etc/mogilefs]# mogadm --trackers=172.16.6.101:7001 class add images class1 --mindevcount=2
[root@Legion100 /etc/mogilefs]# mogadm --trackers=172.16.6.101:7001 class add images class2 --mindevcount=2
[root@Legion100 /etc/mogilefs]# mogadm --trackers=172.16.6.101:7001 class add text class1 --mindevcount=2
[root@Legion100 /etc/mogilefs]# mogadm --trackers=172.16.6.101:7001 class add text class2 --mindevcount=2
[root@Legion100 /etc/mogilefs]# mogadm --trackers=172.16.6.101:7001 class list
 domain               class                mindevcount   replpolicy   hashtype
-------------------- -------------------- ------------- ------------ -------
 images               class1                    2        MultipleHosts() NONE
 images               class2                    2        MultipleHosts() NONE
 images               default                   2        MultipleHosts() NONE   

 text                 class1                    2        MultipleHosts() NONE
 text                 class2                    2        MultipleHosts() NONE
 text                 default                   2        MultipleHosts() NONE   

[root@Legion100 /etc/mogilefs]#

domain :区域
class:复制文件的最小单位（最大为64M,如果一个单文件超出此大小将拆分为多个class存储）
mindevcount：最小复制文件的份数
replpolicy ：复制份数
hashtype：采用的hash的类型

9、上传文件进行测试

1)、图片区域

[root@Legion100 ~]# mogupload --trackers=172.16.6.101:7001 --domain=images --key='/images/2k.png' \
--file='/usr/share/backgrounds/wallpaper-six-2560x1240.png'
[root@Legion100 ~]# mogupload --trackers=172.16.6.101:7001 --domain=images --key='/images/2.png' \
--file='/usr/share/backgrounds/wallpaper-six-2048x1536.png'
[root@Legion100 ~]# moglistkeys --trackers=172.16.6.101:7001 --domain=images
/images/2.png
/images/2k.png
[root@Legion100 ~]# mogfileinfo --trackers=172.16.6.101:7001 --domain=images --key=/images/2.png
- file: /images/2.png
     class:              default
  devcount:                    2
    domain:               images
       fid:                    9
       key:        /images/2.png
    length:               608517
 - http://172.16.6.101:7500/dev1/0/000/000/0000000009.fid
 - http://172.16.6.100:7500/dev2/0/000/000/0000000009.fid
[root@Legion100 ~]# mogfileinfo --trackers=172.16.6.101:7001 --domain=images --key=/images/2k.png
- file: /images/2k.png
     class:              default
  devcount:                    2
    domain:               images
       fid:                    8
       key:       /images/2k.png
    length:               237895
 - http://172.16.6.101:7500/dev1/0/000/000/0000000008.fid
 - http://172.16.6.100:7500/dev2/0/000/000/0000000008.fid
[root@Legion100 ~]#

2)、文本区域

[root@Legion100 ~]# mogupload --trackers=172.16.6.101:7001 --domain=text --key='/text/fstab.html' --file='/etc/fstab'
[root@Legion100 ~]# mogupload --trackers=172.16.6.101:7001 --domain=text --key='/text/hosts.html' --file='/etc/hosts'
[root@Legion100 ~]# mogfileinfo --trackers=172.16.6.101:7001 --domain=text --key=/text/fstab.html
- file: /text/fstab.html
     class:              default
  devcount:                    2
    domain:                 text
       fid:                   10
       key:     /text/fstab.html
    length:                  863
 - http://172.16.6.101:7500/dev1/0/000/000/0000000010.fid
 - http://172.16.6.100:7500/dev2/0/000/000/0000000010.fid
[root@Legion100 ~]# mogfileinfo --trackers=172.16.6.101:7001 --domain=text --key=/text/hosts.html
- file: /text/hosts.html
     class:              default
  devcount:                    2
    domain:                 text
       fid:                   11
       key:     /text/hosts.html
    length:                  254
 - http://172.16.6.100:7500/dev2/0/000/000/0000000011.fid
 - http://172.16.6.101:7500/dev1/0/000/000/0000000011.fid
[root@Legion100 ~]#

10、测试web访问

11、安装Nginx

[root@Legion102 ~]# cd /tmp/
[root@Legion102 /tmp]# wget http://nginx.org/download/nginx-1.9.1.tar.gz
[root@Legion102 /tmp]# git clone https://github.com/vkholodkov/nginx-mogilefs-module.git
[root@Legion102 /tmp]# tar xf nginx-1.9.1.tar.gz
[root@Legion102 /tmp]# Username="www" && for i in `seq 1000 1500`;do [ -z "$(awk -F: '{print$3,$4}' /etc/passwd | grep "$i")" -a -z "$(awk -F: '{print$3}' /etc/group | grep "$i")" ] && UGID=$i && break;done && groupadd -g $UGID $Username && useradd -M -u $UGID -g $UGID -s /sbin/nologin $Username
[root@Legion102 /tmp]# yum install pcre-devel pcre -y
[root@Legion102 /tmp]# cd nginx-1.9.1
[root@Legion102 /tmp/nginx-1.9.1]# ./configure --prefix=/usr/local/nginx/ --user=www --group=www \
--error-log-path=/tmp/nginx/error.log --http-log-path=/tmp/nginx/access.log --pid-path=/var/run/nginx/nginx.pid \
--lock-path=/var/lock/nginx.lock --with-pcre --with-http_ssl_module --with-http_flv_module --with-http_spdy_module \
--with-http_gzip_static_module --with-http_stub_status_module --http-client-body-temp-path=/usr/local/nginx/client/ \
--http-proxy-temp-path=/usr/local/nginx/proxy/ --http-fastcgi-temp-path=/usr/local/nginx/fcgi/ --http-uwsgi-temp-path=/usr/local/nginx/uwsgi \
--http-scgi-temp-path=/usr/local/nginx/scgi --add-module=../nginx-mogilefs-module
[root@Legion102 /tmp/nginx-1.9.1]# make -j $(awk '{if($1=="processor"){i++}}END{print i}' /proc/cpuinfo) && make install && echo $?
[root@Legion102 /tmp/nginx-1.9.1]# wget http://www.dwhd.org/script/Nginx-init-CentOS -O /etc/rc.d/init.d/nginx
[root@Legion102 /tmp/nginx-1.9.1]# chmod +x /etc/rc.d/init.d/nginx
[root@Legion102 /tmp/nginx-1.9.1]# chkconfig nginx on
[root@Legion102 /tmp/nginx-1.9.1]# chkconfig --list nginx
nginx           0:关闭  1:关闭  2:启用  3:启用  4:启用  5:启用  6:关闭
[root@Legion102 /tmp/nginx-1.9.1]# echo "export PATH=/usr/local/nginx/sbin:\$PATH" > /etc/profile.d/nginx1.9.1.sh
[root@Legion102 /tmp/nginx-1.9.1]# . /etc/profile.d/nginx1.9.1.sh
[root@Legion102 /tmp/nginx-1.9.1]# cd /usr/local/nginx/conf/
#Nginx配置文件参考
[root@Legion102 /usr/local/nginx/conf]# grep -Ev '^(\s+)?#|^$' nginx.conf
worker_processes  1;
events {
	worker_connections  1024;
}
http {
	include       mime.types;
	default_type  application/octet-stream;
	sendfile        on;
	keepalive_timeout  65;

	upstream mogfs_cluster {
		server 172.16.6.100:7001;
		server 172.16.6.101:7001;
	}
	server {
		listen       80;
		server_name  localhost;
		location / {
			root   html;
			index  index.html index.htm;
		}
		location ~* ^(/images/.*)$ {
			mogilefs_tracker mogfs_cluster;
			mogilefs_domain images;
			mogilefs_noverify on;
			mogilefs_pass $1 {
				proxy_pass $mogilefs_path;
				proxy_hide_header Content-Type;
				proxy_buffering off;
			}
		}
		location ~* ^(/text/.*)$ {
			mogilefs_tracker mogfs_cluster;
			mogilefs_domain text;
			mogilefs_noverify on;
			mogilefs_pass $1 {
				proxy_pass $mogilefs_path;
				proxy_hide_header Content-Type;
				proxy_buffering off;
			}
		}
		error_page   500 502 503 504  /50x.html;
		location = /50x.html {
			root   html;
		}
	}
}
[root@Legion102 /usr/local/nginx/conf]# cd
[root@Legion102 ~]# service nginx start
正在启动 nginx：                                           [确定]
[root@Legion102 ~]# netstat -tnlp | grep nginx
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      5920/nginx
[root@Legion102 ~]#

12、测试Nginx+MogileFS

13、状态信息查看

[root@Legion100 ~]# mogstats --config=/etc/mogilefs/mogilefsd.conf
Fetching statistics... (all)

Statistics for devices...
  device     host                   files     status
  ---------- ---------------- ------------ ----------
  dev1       172.16.6.101          4      alive
  dev2       172.16.6.100          4      alive

Statistics for devices...
  device     host                   files     status
  ---------- ---------------- ------------ ----------
  dev1       172.16.6.101          4      alive
  dev2       172.16.6.100          4      alive
  ---------- ---------------- ------------ ----------

Statistics for file ids...
  Max file id: 11

Statistics for files...
  domain               class           files    size (m)  fullsize (m)
  -------------------- ----------- ---------- ----------- -------------
  images               default             2           0             1
  text                 default             2           0             0
  -------------------- ----------- ---------- ----------- -------------

Statistics for replication...
  domain               class        devcount      files
  -------------------- ----------- ---------- ----------
  images               default             2          2
  text                 default             2          2
  -------------------- ----------- ---------- ----------

Statistics for replication queue...
  status                      count
  -------------------- ------------
  -------------------- ------------

Statistics for delete queue...
  status                      count
  -------------------- ------------
  -------------------- ------------

Statistics for general queues...
  queue           status                      count
  --------------- -------------------- ------------
  --------------- -------------------- ------------

done
[root@Legion100 ~]#

[root@Legion100 ~]# mogstats --db_dsn="DBI:mysql:mogdb:host=172.16.6.103" --db_user="moguser"  --db_pass="mogpass"
Fetching statistics... (all)

Statistics for devices...
  device     host                   files     status
  ---------- ---------------- ------------ ----------
  dev1       172.16.6.101          4      alive
  dev2       172.16.6.100          4      alive
  ---------- ---------------- ------------ ----------

Statistics for file ids...
  Max file id: 11

Statistics for files...
  domain               class           files    size (m)  fullsize (m)
  -------------------- ----------- ---------- ----------- -------------
  images               default             2           0             1
  text                 default             2           0             0
  -------------------- ----------- ---------- ----------- -------------

Statistics for replication...
  domain               class        devcount      files
  -------------------- ----------- ---------- ----------
  images               default             2          2
  text                 default             2          2
  -------------------- ----------- ---------- ----------

Statistics for replication queue...
  status                      count
  -------------------- ------------
  -------------------- ------------

Statistics for delete queue...
  status                      count
  -------------------- ------------
  -------------------- ------------

Statistics for general queues...
  queue           status                      count
  --------------- -------------------- ------------
  --------------- -------------------- ------------

done
[root@Legion100 ~]#

[root@Legion100 ~]# mogstats --config=/etc/mogilefs/mogilefsd.conf --stats="domains"
Fetching statistics... (domains)

Statistics for domains...
  domain               class           files
  -------------------- ----------- ----------
  images               default             2
  text                 default             2
  -------------------- ----------- ----------

done
[root@Legion100 ~]#