搭建分布式文件系统FastDFS集群

FastDFS是为互联网应用量身定做的一套分布式文件存储系统,非常适合用来存储用户图片、视频、文档等文件。对于互联网应用,和其他分布式文件系统相比,优势非常明显。出于简洁考虑,FastDFS没有对文件做分块存储,因此不太适合分布式计算场景。

在生产环境中往往数据存储量比较大,因此会大部分会选择分布式存储来解决,主要解决以下几个问题

  • 海量数据存储
  • 数据高可用(冗余备份)
  • 较高读写性能和负载均衡
  • 支持多平台多语言
  • 高并发问题

常见分布式存储对比

image_1ebvdh20b1guo13gibct1bkcl5k9.png-343.2kB


FastDFS 相关组件及原理

FastDFS介绍

FastDFS是一个C语言实现的开源轻量级分布式文件系统,支持Linux、FreeBSD、AID等Linux系统,解决了大量数据存储和读写负载等问题,适合存储4KB~500MB之间的小文件,如图片网站,短视频网站,文档,APP下载站等,UC,京东,支付宝,迅雷,酷狗等都有使用,其中UC基于FastDFS向用户提供网盘,广告和应用下载的业务。FastDFS与MogileFS、HDFS、TFS等都不是系统级的分布式文件系统,而是应用级的分布式文件存储服务

FastDFS架构

FastDFS服务有三个角色: 跟踪服务(tracker server)、存储服务(storage server)和客户端(client)

tracker server: 跟踪服务,主要做调度工作,起到均衡的作用;负责管理所有的storage server和group,每个storage在启动后会连接Tracker,告知自己所属group等信息,并保持周期性心跳,tracker根据storage心跳信息,建立group --> [storage server list]的映射表;tracker管理的元数据很少,会直接存放在内存;tracker上的元信息都是由storage汇报的信息生成的,本身不需要持久化任何数据,tracker之间是对等关系,因此扩展tracker访问非常容器,之间增加tracker访问即可,所有tracker都接受storage心跳信息,生成元数据信息来提供读写访问(与其他master-slave架构的优势是没有单点,tracker也不会成为瓶颈,最终数据是和一个可用的storage server进行传输)

storage server: 存储服务器,主要提供容量和备份访问;以group为单位,每个group内可以包含多个storage server,数据互为备份,存储容量空间以group内容量最小的storage为准;建议group内的storage server配置相同;以group为单位组织存储能够方便的进行引用隔离、负载均衡和副本数定制;

缺点: group的容量受单机存储容量的限制,同时group内机器坏掉,数据恢复只能依赖group内其他机器重新同步(硬盘替换,重新挂载重启fdfs_storaged即可)

group存储策略

  • round robin (轮训)
  • load balance (选择最大剩余空间的组上传文件)
  • specify group (指定group上传)

group中storage存储依赖本地文件系统,storage可配置多个数据存储目录,磁盘不做raid,直接分别挂在到多个目录,将这些目录配置为storage的数据目录即可

storage接受写请求时,会根据配置好的规则,选择其中一个存储目录来存储文件;为避免单个目录下文件过多,storage第一次启动,会在每个数据存储目录里创建2级子目录,每级256个,总共65536个,新写的文件会以hash的方式被路由到其中某个子目录下,然后将文件数据直接作为一个本地文件存储到该目录中

image_1ebvil7hskr11tv1l9m1j7p1agjm.png-159.4kB


FastDFS工作流程

上传

image_1ebviv30o1v5p16geisa18gsp6f13.png-152.5kB

FastDFS提供基本的文件访问接口,如upload、download、append、delete等

选择tracker server

集群中tracker之间是对等关系,客户端在上传文件时可用任意选择一个tracker

选择存储group

当tracker接受到upload file的请求时,会为该文件分配一个可以存储的group,目前支持选择group的规则为

1.Round Robin (所有group轮训使用)

2.Specified group (指定某个确定的group)

3.Load balance (剩余存储空间较多的group优先)

选择storage server

当选定group后,tracker会在group内选择一个storage server给客户端,目前支持选择server的规则为

1.Round Robin (所有server轮训使用)默认规则

2.根据IP地质进行排序选择第一个服务器 (IP地址最小者)

3.根据优先级进行排序 (上传优先级由storage server来设置,参数为upload_priority)

选择storage path (磁盘或者挂载点)

当分配好storage server后,客户端将向storage发送写文件请求,storage会将文件分配一个数据存储目录,目前支持选择存储路径的规则为:

1.round robin (轮训)默认

2.load balance 选择使用剩余空间最大的存储路径

选择下载服务器

目前支持的规则为

1.轮训方式,可以下载当前文件的任一storage server

2.从源storage server下载

生成file_id

选择存储目录后,storage会生成一个file_id,采用Base64编码,包含字段包括: storage server ip、文件创建时间、文件大小、文件CRC32校验码和随机数;每个存储目录下有两个256*256个子目录,storage会按文件file_id进行两次hash,路由到其中一个子目录,然后将文件file_id为文件名存储在该子目录下,最后生成文件路径: group名称、虚拟磁盘路径、数据两级目录、file_id

group1 /M00/02/44/wkgDRe348wAAAAGKYJK42378.sh

其中,组名: 上传文件后所在的存储组的名称,在文件上传成功后由存储服务器返回,需要客户端自行保存

虚拟磁盘路径: 存储服务器配置的虚拟路径,与磁盘选项store_path*参数对应

数据两级目录: 存储服务器在每个虚拟磁盘路径下创建的两级目录,用于存储数据文件


同步机制

1.新增tracker服务器数据同步

由于storage server上配置了所有的tracker server,storage server和tracker server之间的通信是由storage server主动发起的,storage server为每台tracker server 启动一个线程进行通信;在通信过程中,若发现tracker server返回的本组storage server列表比本机记录少,就会将该tracker server上没有的storage server同步给该tracker,这样的机制使得tracker之间是对等的关系,数据保持一致

2.组内新增storage数据同步

若新增storage server或者其状态发生变化,tracker server都会将storage server列表同步给该组内所有storage server;以新增storage server为例,新加入storage server会主动连接tracker server,tracker server发现有新的storage server加入,就会将该组内所有的storage server返回给新加入的storage server,并重新将该组的storage server列表返回给该组内的其他storage server

3.组内storage数据同步

组内storage server之间是对等的,文件上传,删除等操作可以在组内任意一台storage server上进行。文件同步只能在同组内的storage server之间进行,采用push方式,即源服务器同步到目标服务器

A. 只在同组内的storage server之间同步

B. 源数据才需要同步,备份数据不再同步

C. 特例: 新增storage server时,由其中一台将已有的所有数据(包括源数据和备份数据)同步到新增服务器

storage server 7种状态

通过命令fdfs_monitor /etc/fdfs/client.conf可以查看ip_addr选项显示storage server当前状态

INIT       初始化,尚未得到同步已有数据的源服务器
WAIT_SYNC  等待同步,已得到同步已有数据的源服务器
SYNCING    同步中
DELETE     已删除,该服务器从本组中摘除
OFFLINE    离线
ONLINE     在线,尚不能提供服务
ACTIVE     在线,可以提供服务

组内增加storage server A状态变化过程:

1.storage server A主动连接tracker server,此时tracker server将storage serverA状态设置为INIT

2.storage server A向tracker server询问追加同步的源服务器和追加同步截止时间点(当前时间),若组内只有storage server A或者上传文件数为0,则告诉新主机不需要同步数据,storage serverA状态为ONLINE;若组内没有active状态及其,就返回错误给新机器,新机器重新尝试;否则tracker将其状态设置为WAIT_SYNC

3.假如分配了storage server B为同步源服务器和截止时间点,那么storage serverB将会截止时间点之前的所有数据同步给storage server A,并请求tracker设置storage server A状态为SYNCING;到了截止时间后,storage server B向storage server A的同步将由追加同步切换为正常binlog增量同步,当获取不到更多binlog时,请求tracker将storage server A同步完所有数据,暂时没有数据要同步时,storage server B请求tracker server将storage server A的状态设置为ONLINE

4.storage server B向storage server A同步完所有数据,暂时没有数据要同步时,storage server B请求tracker server将 storage server A的状态设置为ONLINE

5.当storage server A向tracker server发起心跳时,tracker server将其状态更改为ACTIVE,之后就是增量同步(binlog)

image_1ec2fv4og23cn1j1m58pvr12rk1g.png-155.3kB

注: 整个源同步班过程是源机器启动弄一个同步线程,将数据Push到新机器,最大达到一个磁盘的IO,不能并发;由于源同步截止条件是获取不到binlog,系统繁忙,不断有新数据写入的情况,将会导致一直无法完成源同步


下载

image_1ec2g5nompfd7e815vq1u0a1g491t.png-133.2kB

client发送下载请求给某个tracker,必须带上文件名信息,tracker从文件名中解析出文件的group、大小、创建时间等信息,然后为该请求选择一个storage用于读请求;由于group内的文件同步是异步进行,可能出现文件没有同步到其他storage server上或者延迟的问题,可以使用nginx_fastdfs_module模块解决

image_1ec2gftjjh28idtrvv1reqd792a.png-187.2kB

关于文件去重

由于FastDFS本身不能对重复上传的文件进行去重,而FastDHT可以做到去重。FastDHT是一个高性能的分布式哈希系统,它是基于键值对存储的,而且它需要依赖于Berkeley DB作为数据存储的媒介,同时需要依赖于libfastcommon

由于业务需要,目前不存在文件去重的时候,如果需要可以自己简单了解一下FastDHT

FastDFS-Nginx扩展模块源码分析


安装FastDFS集群

本次环境架构

image_1ec84b7b91s9hrhr1gdk1m0o1pg79.png-68.1kB

针对tracker.conf && storage.conf && mod_fastdfs.conf有一篇单独的文章介绍相关参数。有兴趣的可以看一下,也可以直接看默认的配置文件,对每个参数都有介绍

FastDFS 配置文件详解

1350

环境说明

#nginx这里可以部署2台,加上keepliveed作高可用,由于我这里机器不足,就使用单台nginx进行代理
nginx  192.168.31.100 nginx


tracker 节点
tracker 01:192.168.31.101  FastDFS,libfastcommon,nginx,ngx_cache_purge
tracker 02:192.168.31.102  FastDFS,libfastcommon,nginx,ngx_cache_purge
#其中tracker不提供存储


Storage 节点
[group1]
storage 01:192.168.31.103  FastDFS,libfastcommon,nginx,fastdfs-nginx-module
storage 02:192.168.31.104  FastDFS,libfastcommon,nginx,fastdfs-nginx-module

[group2]
storage 03:192.168.31.105  FastDFS,libfastcommon,nginx,fastdfs-nginx-module
storage 04:192.168.31.106  FastDFS,libfastcommon,nginx,fastdfs-nginx-module

1.所有的服务器都需要安装nginx,主要是用于访问和上传无关;

2.tracker安装nginx主要为了提供http反向代理、负载均衡以及缓存服务

3.每一台storage服务器部署Nginx及FastDFS扩展模块,主要用于对storage存储的文件提供http下载访问,仅当前storage节点找不到文件时会向源storage主机发送rediect或者proxy动作

所有节点安装

关闭防火墙,selinux

systemctl stop firewalld
systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat
iptables -P FORWARD ACCEPT
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

设置yum源

yum install -y wget
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

yum clean all 
yum makecache

温馨提示:除了nginx节点(192.168.31.100),其他节点都需要执行安装fastdfs和nginx

安装依赖包 (可解决99%的依赖问题)

yum -y install gcc gcc-c++ make autoconf libtool-ltdl-devel gd-devel freetype-devel libxml2-devel libjpeg-devel libpng-devel openssh-clients openssl-devel curl-devel bison patch libmcrypt-devel libmhash-devel ncurses-devel binutils compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel glibc glibc-common glibc-devel libgcj libtiff pam-devel libicu libicu-devel gettext-devel libaio-devel libaio libgcc libstdc++ libstdc++-devel unixODBC unixODBC-devel numactl-devel glibc-headers sudo bzip2 mlocate flex lrzsz sysstat lsof setuptool system-config-network-tui system-config-firewall-tui ntsysv ntp pv lz4 dos2unix unix2dos rsync dstat iotop innotop mytop telnet iftop expect cmake nc gnuplot screen xorg-x11-utils xorg-x11-xinit rdate bc expat-devel compat-expat1 tcpdump sysstat man nmap curl lrzsz elinks finger bind-utils traceroute mtr ntpdate zip unzip vim wget net-tools

下载依赖包 (除了nginx节点,其他节点都要安装)

mkdir /root/tools/
cd /root/tools
wget http://nginx.org/download/nginx-1.18.0.tar.gz
wget https://github.com/happyfish100/libfastcommon/archive/V1.0.43.tar.gz
wget https://github.com/happyfish100/fastdfs/archive/V6.06.tar.gz
wget https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.22.tar.gz


#为了保证文章可用性,本次软件包已经进行备份,下载地址如下
mkdir /root/tools/
cd /root/tools
wget http://down.i4t.com/fdfs/v6.6/nginx-1.18.0.tar.gz
wget http://down.i4t.com/fdfs/v6.6/V1.0.43.tar.gz
wget http://down.i4t.com/fdfs/v6.6/V6.06.tar.gz
wget http://down.i4t.com/fdfs/v6.6/V1.22.tar.gz



#解压
cd /root/tools
tar xf nginx-1.18.0.tar.gz 
tar xf V1.0.43.tar.gz 
tar xf V1.22.tar.gz 
tar xf V6.06.tar.gz 

安装libfastcommon (除了nginx节点,其他节点都要安装)

cd /root/tools/libfastcommon-1.0.43
./make.sh 
./make.sh install

安装FastDFS (除了nginx节点,其他节点都要安装)

cd /root/tools/fastdfs-6.06/
./make.sh 
./make.sh install

拷贝配置文件 (tracker01 02节点)

[root@tracker01 fastdfs-6.06]# cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf     #tracker节点
[root@01 fastdfs-6.06]# cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf      #客户端文件(测试使用)
[root@01 fastdfs-6.06]# cp /root/tools/fastdfs-6.06/conf/http.conf /etc/fdfs/      #nginx配置文件
[root@01 fastdfs-6.06]# cp /root/tools/fastdfs-6.06/conf/mime.types /etc/fdfs/     #nginx配置文件

配置tracker 01节点

这里可以先配置一台节点,没有问题在启动另外的节点

创建tracker数据存储及日志目录 (需要在tracker节点执行)

mkdir /data/tracker/ -p 

修改配置文件 (tracker 01节点执行)

cat >/etc/fdfs/tracker.conf <<EOF
disabled = false
bind_addr =
port = 22122
connect_timeout = 5
network_timeout = 60
base_path = /data/tracker
max_connections = 1024
accept_threads = 1
work_threads = 4
min_buff_size = 8KB
max_buff_size = 128KB
store_lookup = 0
store_server = 0
store_path = 0
download_server = 0
reserved_storage_space = 20%
log_level = info
run_by_group=
run_by_user =
allow_hosts = *
sync_log_buff_interval = 1
check_active_interval = 120
thread_stack_size = 256KB
storage_ip_changed_auto_adjust = true
storage_sync_file_max_delay = 86400
storage_sync_file_max_time = 300
use_trunk_file = false 
slot_min_size = 256
slot_max_size = 1MB
trunk_alloc_alignment_size = 256
trunk_free_space_merge = true
delete_unused_trunk_files = false
trunk_file_size = 64MB
trunk_create_file_advance = false
trunk_create_file_time_base = 02:00
trunk_create_file_interval = 86400
trunk_create_file_space_threshold = 20G
trunk_init_check_occupying = false
trunk_init_reload_from_binlog = false
trunk_compress_binlog_min_interval = 86400
trunk_compress_binlog_interval = 86400
trunk_compress_binlog_time_base = 03:00
trunk_binlog_max_backups = 7
use_storage_id = false
storage_ids_filename = storage_ids.conf
id_type_in_filename = id
store_slave_file_use_link = false
rotate_error_log = false
error_log_rotate_time = 00:00
compress_old_error_log = false
compress_error_log_days_before = 7
rotate_error_log_size = 0
log_file_keep_days = 0
use_connection_pool = true
connection_pool_max_idle_time = 3600
http.server_port = 8080
http.check_alive_interval = 30
http.check_alive_type = tcp
http.check_alive_uri = /status.html
EOF

启动tracker

[root@01 fastdfs-6.06]# /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start

配置tracker02节点

拷贝配置文件

scp -r /etc/fdfs/tracker.conf root@192.168.31.102:/etc/fdfs/
ssh root@192.168.31.102 mkdir /data/tracker/ -p 

tracker02启动tracker

[root@02 fastdfs-6.06]# /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start

检查启动状态

netstat -lntup|grep 22122
tcp        0      0 0.0.0.0:22122           0.0.0.0:*               LISTEN      108126/fdfs_tracker

如果启动失败可以查看tracker报错

tail -f /data/tracker/logs/trackerd.log

接下来编辑启动脚本

cat > /usr/lib/systemd/system/tracker.service <<EOF
[Unit]
Description=The FastDFS File server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
ExecStart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
ExecStop=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf stop
ExecRestart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart

[Install]
WantedBy=multi-user.target
EOF


$ systemctl daemon-reload
$ systemctl start tracker
$ systemctl enable tracker
$ systemctl status tracker

#需要先手动kill 掉tracker

storage 01-02节点配置

storage01节点和02节点配置相同,storage03和storage04配置相同

storage01和storage02节点属于group1组

创建storage 数据存储目录

mkdir /data/fastdfs_data -p

修改配置文件

cat >/etc/fdfs/storage.conf<<EOF
disabled = false
group_name = group1
bind_addr =
client_bind = true
port = 23000
connect_timeout = 5
network_timeout = 60
heart_beat_interval = 30
stat_report_interval = 60
base_path = /data/fastdfs_data
max_connections = 1024
buff_size = 256KB
accept_threads = 1
work_threads = 4
disk_rw_separated = true
disk_reader_threads = 1
disk_writer_threads = 1
sync_wait_msec = 50
sync_interval = 0
sync_start_time = 00:00
sync_end_time = 23:59
write_mark_file_freq = 500
disk_recovery_threads = 3
store_path_count = 1
store_path0 = /data/fastdfs_data
subdir_count_per_path = 256
tracker_server = 192.168.31.101:22122
tracker_server = 192.168.31.102:22122
log_level = info
run_by_group =
run_by_user =
allow_hosts = *
file_distribute_path_mode = 0
file_distribute_rotate_count = 100
fsync_after_written_bytes = 0
sync_log_buff_interval = 1
sync_binlog_buff_interval = 1
sync_stat_file_interval = 300
thread_stack_size = 512KB
upload_priority = 10
if_alias_prefix =
check_file_duplicate = 0
file_signature_method = hash
key_namespace = FastDFS
keep_alive = 0
use_access_log = false
rotate_access_log = false
access_log_rotate_time = 00:00
compress_old_access_log = false
compress_access_log_days_before = 7
rotate_error_log = false
error_log_rotate_time = 00:00
compress_old_error_log = false
compress_error_log_days_before = 7
rotate_access_log_size = 0
rotate_error_log_size = 0
log_file_keep_days = 0
file_sync_skip_invalid_record = false
use_connection_pool = true
connection_pool_max_idle_time = 3600
compress_binlog = true
compress_binlog_time = 01:30
check_store_path_mark = true
http.domain_name =
http.server_port = 80
EOF

#注意: 需要修改tracker_server地址,多个节点多复制几行,一个节点写一行就可以。 不建议单节点使用localhost

配置启动文件

cat >/usr/lib/systemd/system/storage.service <<EOF

[Unit]
Description=The FastDFS File server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
ExecStart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start
ExecStop=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop
ExecRestart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start storage
systemctl status storage
systemctl enable storage

检查启动状态

netstat -lntup|grep 23000

如果出现启动失败,可以到设置的目录查看一下log

tail -f /data/fastdfs_data/logs/storaged.log

storage 02-03节点配置

storage03和storage04节点属于group2组

基本流程不变这里只说明需要修改的地方

在storage03-storage-04节点同步执行

创建storage 数据存储目录
mkdir /data/fastdfs_data -p

修改配置文件
cat >/etc/fdfs/storage.conf<<EOF
disabled = false
group_name = group2
bind_addr =
client_bind = true
port = 23000
connect_timeout = 5
network_timeout = 60
heart_beat_interval = 30
stat_report_interval = 60
base_path = /data/fastdfs_data
max_connections = 1024
buff_size = 256KB
accept_threads = 1
work_threads = 4
disk_rw_separated = true
disk_reader_threads = 1
disk_writer_threads = 1
sync_wait_msec = 50
sync_interval = 0
sync_start_time = 00:00
sync_end_time = 23:59
write_mark_file_freq = 500
disk_recovery_threads = 3
store_path_count = 1
store_path0 = /data/fastdfs_data
subdir_count_per_path = 256
tracker_server = 192.168.31.101:22122
tracker_server = 192.168.31.102:22122
log_level = info
run_by_group =
run_by_user =
allow_hosts = *
file_distribute_path_mode = 0
file_distribute_rotate_count = 100
fsync_after_written_bytes = 0
sync_log_buff_interval = 1
sync_binlog_buff_interval = 1
sync_stat_file_interval = 300
thread_stack_size = 512KB
upload_priority = 10
if_alias_prefix =
check_file_duplicate = 0
file_signature_method = hash
key_namespace = FastDFS
keep_alive = 0
use_access_log = false
rotate_access_log = false
access_log_rotate_time = 00:00
compress_old_access_log = false
compress_access_log_days_before = 7
rotate_error_log = false
error_log_rotate_time = 00:00
compress_old_error_log = false
compress_error_log_days_before = 7
rotate_access_log_size = 0
rotate_error_log_size = 0
log_file_keep_days = 0
file_sync_skip_invalid_record = false
use_connection_pool = true
connection_pool_max_idle_time = 3600
compress_binlog = true
compress_binlog_time = 01:30
check_store_path_mark = true
http.domain_name =
http.server_port = 80
EOF


#配置启动文件
cat >/usr/lib/systemd/system/storage.service <<EOF

[Unit]
Description=The FastDFS File server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
ExecStart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start
ExecStop=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop
ExecRestart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start storage
systemctl status storage
systemctl enable storage


#检查启动状态
netstat -lntup|grep 23000

如果出现systemctl启动失败,可以使用命令启动,在根据日志进行查看。 大概启动时间为10s

#storage启动、停止、重启命令
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart

所有节点storage启动完毕后进行检查,是否可以获取到集群信息 (刚创建的集群比较慢,稍等一会。需要等待状态为ACTIVE即可)

#在任意节点storage节点执行命令都可以,获取结果应该如下
[root@storage01 fdfs]#  fdfs_monitor /etc/fdfs/storage.conf list

[2020-07-03 01:15:25] DEBUG - base_path=/data/fastdfs_data, connect_timeout=5, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=1, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=2, server_index=1   

tracker server is 192.168.31.102:22122     #tracker server处理本次命令的节点

group count: 2                            #group组数量

Group 1:                                   #group1组信息
group name = group1
disk total space = 17,394 MB
disk free space = 13,758 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 80
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

    Storage 1:                                   #storage1节点信息
        id = 192.168.31.103
        ip_addr = 192.168.31.103  ACTIVE         #storage节点状态
        http domain = 
        version = 6.06                           #fdfs 版本
        join time = 2020-07-03 01:08:29          #加入集群时间
        up time = 2020-07-03 01:08:29
        total storage = 17,394 MB
        free storage = 14,098 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 80
        current_write_path = 0
        source storage id = 192.168.31.104
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 1
        ...............省略号............................
        last_heart_beat_time = 2020-07-03 01:15:18
        last_source_update = 1970-01-01 08:00:00
        last_sync_update = 1970-01-01 08:00:00
        last_synced_timestamp = 1970-01-01 08:00:00 
    Storage 2:                                   #storage2节点信息
        id = 192.168.31.104                      #storage 节点IP
        ip_addr = 192.168.31.104  ACTIVE         #storage 节点状态
        http domain = 
        version = 6.06                          #storage 节点版本
        join time = 2020-07-03 01:08:26         #加入时间
        up time = 2020-07-03 01:08:26
        total storage = 17,394 MB
        free storage = 13,758 MB
        upload priority = 10
        store_path_count = 1
        ...............省略号............................
        last_heart_beat_time = 2020-07-03 01:15:17
        last_source_update = 1970-01-01 08:00:00
        last_sync_update = 1970-01-01 08:00:00
        last_synced_timestamp = 1970-01-01 08:00:00 

Group 2:                                         #group2集群信息
group name = group2
disk total space = 17,394 MB
disk free space = 15,538 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 80
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

    Storage 1:                                     #storage1节点信息
        id = 192.168.31.105
        ip_addr = 192.168.31.105  ACTIVE
        http domain = 
        version = 6.06
        join time = 2020-07-03 01:13:42
        up time = 2020-07-03 01:13:42
        total storage = 17,394 MB
        free storage = 15,538 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000                   #storage端口
        storage_http_port = 80
        current_write_path = 0
        source storage id = 
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 1
        ...............省略号............................
        last_heart_beat_time = 2020-07-03 01:15:22
        last_source_update = 1970-01-01 08:00:00
        last_sync_update = 1970-01-01 08:00:00
        last_synced_timestamp = 1970-01-01 08:00:00 
    Storage 2:
        id = 192.168.31.106
        ip_addr = 192.168.31.106  ACTIVE
        http domain = 
        version = 6.06
        join time = 2020-07-03 01:14:05
        up time = 2020-07-03 01:14:05
        total storage = 17,394 MB
        free storage = 15,538 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 80
        current_write_path = 0
        source storage id = 192.168.31.105
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 1
        connection.max_count = 1
        total_upload_count = 0
        ...............省略号............................
        total_file_write_count = 0
        success_file_write_count = 0
        last_heart_beat_time = 2020-07-03 01:15:10
        last_source_update = 1970-01-01 08:00:00
        last_sync_update = 1970-01-01 08:00:00
        last_synced_timestamp = 1970-01-01 08:00:00 

配置client

这里在tracker01节点配置celient客户端 (其他节点可不配置,client.conf为可选配置)

mkdir -p /data/fdfs_client/logs  #日志存放路径

cat >/etc/fdfs/client.conf <<EOF
connect_timeout = 5
network_timeout = 60
base_path = /data/fdfs_client/logs
tracker_server = 192.168.31.101:22122
tracker_server = 192.168.31.102:22122
log_level = info
use_connection_pool = false
connection_pool_max_idle_time = 3600
load_fdfs_parameters_from_tracker = false
use_storage_id = false
storage_ids_filename = storage_ids.conf
http.tracker_server_port = 80
EOF

#需要修改tracker_server地址

上传文件测试,这里的文件是init.yaml

[root@01 ~]# echo "test" >init.yaml
[root@01 ~]# fdfs_upload_file /etc/fdfs/client.conf init.yaml
group2/M00/00/00/wKgfaV7-GG2AQcpMAAAABTu5NcY98.yaml

Storage节点安装Nginx

所有storage节点mod_fastdfs.conf配置如下

cat >/etc/fdfs/mod_fastdfs.conf <<EOF
connect_timeout=2
network_timeout=30
base_path=/tmp
load_fdfs_parameters_from_tracker=true
storage_sync_file_max_delay = 86400
use_storage_id = false
storage_ids_filename = storage_ids.conf
tracker_server=192.168.31.101:22122
tracker_server=192.168.31.102:22122
storage_server_port=23000
url_have_group_name = true
store_path_count=1
log_level=info
log_filename=
response_mode=proxy
if_alias_prefix=
flv_support = true
flv_extension = flv
group_count = 2
#include http.conf

[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/data/fastdfs_data

[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/data/fastdfs_data
EOF

拷贝相关依赖 (以下是所有storage节点安装)

cp /root/tools/fastdfs-6.06/conf/http.conf /etc/fdfs/
cp /root/tools/fastdfs-6.06/conf/mime.types /etc/fdfs/

安装Nginx依赖包

yum install -y gcc glibc gcc-c++ prce-devel openssl-devel pcre-devel lua-devel libxml2 libxml2-devel libxslt-devel  perl-ExtUtils-Embed   GeoIP GeoIP-devel GeoIP-data zlib zlib-devel openssl  pcre pcre-devel gcc g++ gcc-c++ gd-devel

创建nginx用户

useradd -s /sbin/nologin nginx -M

编译nginx

cd /root/tools/nginx-1.18.0
./configure --prefix=/usr/local/nginx-1.18 --with-http_ssl_module --user=nginx --group=nginx  --with-http_sub_module  --add-module=/root/tools/fastdfs-nginx-module-1.22/src

make && make install

ln -s /usr/local/nginx-1.18 /usr/local/nginx

修改Nginx配置文件

cat > /usr/local/nginx/conf/nginx.conf <<EOF
worker_processes  1;
events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;


    server {
        listen       8888;
        server_name  localhost;

        location ~/group[0-9]/M00 {
            root /data/fastdfs_data;
            ngx_fastdfs_module;
        }
    }

}
EOF

启动nginx

/usr/local/nginx/sbin/nginx -t
/usr/local/nginx/sbin/nginx

在storage节点上8888的请求且有group的都转给ngx_fastdfs_module插件处理

接下来我们手动上传两张图片进行测试

[root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf abcdocker.png 
group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png

[root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf i4t.jpg 
group2/M00/00/00/wKgfal7-JySABmgLAABdMoE-LPo504.jpg

目前我们tracker属于轮训机制,会轮训group1和group2;具体使用参数可以参考下面的文章

接下来我们可以通过浏览器访问,不同的组对应不同的项目,FastDFS集群可以有多个组,但是每台机器只可以有一个storage

http://storage1节点:8888/group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png
http://storage1节点:8888/group2/M00/00/00/wKgfal7-JySABmgLAABdMoE-LPo504.jpg

经过我的测试,即使我们把图片上传到group1中,在group2上面直接访问也可以访问成功,但是在group2的存储目录并没有找到图片文件。原因如下

#Nginx日志
192.168.31.174 - - [03/Jul/2020:03:03:31 +0800] "GET /group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0"

为何我们其他storage节点也可以访问,原因是nginx中fastdfs-nginx-module模块可以重定向文件连接到源服务器取文件

补充: FastDFS常用命令参数

#查看集群状态
fdfs_monitor /etc/fdfs/storage.conf

#上传
fdfs_upload_file /etc/fdfs/client.conf abcdocker.png 
#下载
fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png
#查看文件属性
fdfs_file_info /etc/fdfs/client.conf group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png

#删除文件
fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png

#删除一个storage
/usr/local/bin/fdfs_monitor /etc/fdfs/storage.conf delete group2 192.168.31.105

Tracker配置高可用

在tracker上安装的nginx主要为了提供http访问的反向代理、负载均衡和缓存服务

这里我们tracker01 02同时进行安装即可

#下载nginx依赖包
mkdir /root/tools -p
cd /root/tools
wget http://down.i4t.com/ngx_cache_purge-2.3.tar.gz
wget http://down.i4t.com/fdfs/v6.6/nginx-1.18.0.tar.gz
tar xf ngx_cache_purge-2.3.tar.gz
tar xf nginx-1.18.0.tar.gz

#创建nginx用户
useradd -s /sbin/nologin -M nginx

#编译nginx
cd /root/tools/nginx-1.18.0
./configure --prefix=/usr/local/nginx-1.18 --with-http_ssl_module --user=nginx --group=nginx  --with-http_sub_module --add-module=/root/tools/ngx_cache_purge-2.3

make && make install

ln -s /usr/local/nginx-1.18 /usr/local/nginx

nginx安装完毕,接下来配置nginx.conf

tracker 中nginx节点可以不是80,我这里以80位代表

mkdir /data/nginx_cache -p

$ vim /usr/local/nginx/conf/nginx.conf
worker_processes  1;
events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server_names_hash_bucket_size 128;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 32k;
    client_max_body_size 300m;
    proxy_redirect off;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
    proxy_buffer_size 16k;
    proxy_buffers 4 64k;
    proxy_busy_buffers_size 128k;
    proxy_temp_file_write_size 128k;
    proxy_cache_path /data/nginx_cache keys_zone=http-cache:100m;

upstream fdfs_group1 {
    server 192.168.31.103:8888 weight=1 max_fails=2 fail_timeout=30s;
    server 192.168.80.104:8888 weight=1 max_fails=2 fail_timeout=30s;
}
upstream fdfs_group2 {
    server 192.168.31.105:8888 weight=1 max_fails=2 fail_timeout=30s;
    server 192.168.31.106:8888 weight=1 max_fails=2 fail_timeout=30s;
}


    server {
        listen       80;
        server_name  localhost;
        location /group1/M00 {
            proxy_next_upstream http_502 http_504 error timeout invalid_header;
            proxy_cache http-cache;
            proxy_cache_valid  200 304 12h;
            proxy_cache_key $uri$is_args$args;
            proxy_pass http://fdfs_group1;
            expires 30d;
        }

        location /group2/M00 {
            proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache http-cache;
            proxy_cache_valid 200 304 12h;
            proxy_cache_key $uri$is_args$args;
            proxy_pass http://fdfs_group2;
            expires 30d;
        }

    }
 }

/usr/local/nginx/sbin/nginx -t
/usr/local/nginx/sbin/nginx 

此时访问tracker01节点和tracker02节点应该都没有问题

http://192.168.31.101/group1/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg
http://192.168.31.101/group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg

http://192.168.31.102/group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg
http://192.168.31.102/group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg

效果图如下

image_1ec8ggt1o32s19besoncejeikm.png-820.7kB


Nginx代理安装

通过上面的步骤,已经可以使用storage节点和tracker节点进行访问,但是为了解决统一管理和tracker高可用,我们还需要使用nginx在去代理tracker

#nginx安装和上面一样,我这里就只更改nginx.conf文件,nginx代理不需要缓存模块,普通安装即可

$ vim /usr/local/nginx/conf/nginx.conf

worker_processes  1;
events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

upstream fastdfs_tracker {
        server 192.168.31.101:80 weight=1 max_fails=2 fail_timeout=30s;
        server 192.168.31.102:80 weight=1 max_fails=2 fail_timeout=30s;
}
    server {
        listen       80;
        server_name  localhost;

        location / {
        proxy_pass http://fastdfs_tracker/;
        }

    }
}

最后我们在tracker01节点上,测试nginx代理是否都可以访问成功

[root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf i4t.jpg 
group1/M00/00/00/wKgfZ17-Oa2AMuRqAABdMoE-LPo686.jpg

[root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf i4t.jpg 
group2/M00/00/00/wKgfaV7-Oa6ANXGLAABdMoE-LPo066.jpg

访问查看
[root@tracker01 ~]# curl 192.168.31.100/group1/M00/00/00/wKgfZ17-Oa2AMuRqAABdMoE-LPo686.jpg -I
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Thu, 02 Jul 2020 19:49:49 GMT
Content-Type: image/jpeg
Content-Length: 23858
Connection: keep-alive
Last-Modified: Thu, 02 Jul 2020 19:46:53 GMT
Expires: Sat, 01 Aug 2020 19:49:49 GMT
Cache-Control: max-age=2592000
Accept-Ranges: bytes

[root@tracker01 ~]# curl 192.168.31.100/group2/M00/00/00/wKgfaV7-Oa6ANXGLAABdMoE-LPo066.jpg -I
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Thu, 02 Jul 2020 19:50:17 GMT
Content-Type: image/jpeg
Content-Length: 23858
Connection: keep-alive
Last-Modified: Thu, 02 Jul 2020 19:46:54 GMT
Expires: Sat, 01 Aug 2020 19:50:17 GMT
Cache-Control: max-age=2592000
Accept-Ranges: bytes
「点点赞赏,手留余香」

    还没有人赞赏,快来当第一个赞赏的人吧!
abcdocker运维博客
links
2 条回复 A 作者 M 管理员
欢迎您,新朋友,感谢参与互动!欢迎您 {{author}},您在本站有{{commentsCount}}条评论
k8s分 享 面试宝典
加入我们
  • 站长QQ:1272204一键联系
  • abcdocker 微信公众号
    abcdocker QQ群