The ultimate Guide to a Clustered Redundant File Store

Leo Moll,

If you wish to setup a redundant clustered file server, you will find a lot of good articles that cover different aspects of the task. You will find also some step-by-step guides and how-tos but most of them do not include real life issues like selinux and firewall.

This "ultimate howto" aims to provide a step-by-step guide to a working clustered file store including every important aspect.

Prerequisites: You should have a decent understanding of networking, filesystems and be familiar with common tasks on Linux systems in order to understand this guide and perform the installation.

User Requirements

The following user requirements have been considered in the planning:

  1. The user shall be able to install the solution by using only supported packages from the used linux distribution
  2. The user shall be able to install the solution without disabling any security mechanism provided by the linux distribution (I stop reading any article as soon as I encounter phrases like "...disable selinux..." or "...disable the firewall...")
  3. The user shall be able to access the storage both from Windows and from Linux
  4. The user shall be able to use the installed system for a decent long period of time without having to plan major updates

As usual there are million ways to achieve the desired result. This guide explains one of them. Resulting from these requirements, I chose the following components for the system:

  • Distribution: CentOS 7
  • Clustered Filesystem: glusterfs

Why CentOS 7

  • Standard: CentOS is one of the most common distributions. Furthermore it is binary compatible to RHEL (Red Hat Enterprise Linux). You will likely achieve the same results with RHEL.
  • Long Term Support: CentOS 7 will be supported up to June 2024.
  • Modern Software: CentOS 7 ships with up to date technologies like systemd, firewalld, Samba 4, etc.

Why GlusterFS

  • Stability: GlusterFS is considered a reliable stable solution
  • Compatiblity: GlusterFS provides both native client access and support for NFS and CIFS (via native Samba integration)
  • Many Topologies: GlusterFS supports many storage topologies (distributed, striped, replicated and all combinations of them)
  • Simplicity: GlusterFS is easy to install and to manage
  • Modesty: A GlusterFS deployment can start with two machines

Assumptions in this guide

  1. Our two machines will have the fixed IP Addresses 10.42.0.51 and 10.42.0.52
  2. Our two machines will be named filenode1.example.local and filenode2.example.local
  3. The access to the resulting clustered file server will be provided by the virtual IP addresses 10.42.0.31 and 10.42.0.32
  4. The clustered file server will be accessed using the name fileserver.example.local

When you install and configure your machine, replace these data with the IP addresses and names that you plan for your network.

Name Resolution

Make sure that the DNS Server of your Network is able to resolve the names for your two machines. Both directions should work:

leo@bookpoldo ~ $ nslookup filenode1.example.local
Server:        10.42.8.13
Address:    10.42.8.13#53

Name:    filenode1.example.local
Address: 10.42.0.51

leo@bookpoldo ~ $ nslookup 10.42.0.51
Server:        10.42.8.13
Address:    10.42.8.13#53

51.0.42.10.in-addr.arpa    name = filenode1.example.local.

In addition you should register both virtual IP addresses in DNS for the name of the clustered file server. The DNS server will return the addresses in a round robin fashion.

leo@bookpoldo ~ $ nslookup fileserver.example.local
Server:        10.42.8.13
Address:    10.42.8.13#53

Name:    filenode1.example.local
Address: 10.42.0.31
Name:    filenode1.example.local
Address: 10.42.0.32

leo@bookpoldo ~ $ nslookup fileserver.example.local
Server:        10.42.8.13
Address:    10.42.8.13#53

Name:    fileserver.example.local
Address: 10.42.0.32
Name:    fileserver.example.local
Address: 10.42.0.31

Also here the addresses must resolve in both directions:

leo@bookpoldo ~ $ nslookup 10.42.0.31
Server:        10.42.8.13
Address:    10.42.8.13#53

31.0.42.10.in-addr.arpa    name = fileserver.example.local.

leo@bookpoldo ~ $ nslookup 10.42.0.32
Server:        10.42.8.13
Address:    10.42.8.13#53

32.0.42.10.in-addr.arpa    name = fileserver.example.local.

Installation of the Base System

Install the base system using the Minimal ISO. The operating system and the software does not require lots of space: 16GB should be enough. You can use a flash module or SSD for the operating system in case you install on a physical machine. On a virtual machine, create a properly sized volume. Make sure that only the first disk is used for the OS installation. If you chose automatic partitioning, the CentOS installer usually combines all found harddisks into one LVM-managed volume. We do not want this. If you want to be sure that the installer does not touch the storage disks, install/create them after the OS installation.

After the installation of the base system, you should be able to reach your machines via SSH.

Author:
This page was last updated on May 4, 2016, 19:17:07