Извольте:
=====
Howto to Configure High Availability (HA Cluster) OpenFiler and DRBD
Environment
This how-to will be dependent upon a specific hardware configuration, but differing hardware may be adaptable. The hardware selected was evaluated against price, performance, adaptability, and scalability.
Item Qty Detail
Chassis 1 NORCO RPC-2008 2U Rackmount Server Case
Motherboard 1 ASUS M2N32-WS Pro AM2
CPU 1 AMD Athlon 64 X2 5000+ Brisbane 2.6GHz Socket AM2 65W Dual-Core
Memory 2 Kingston HyperX 4GB (2 x 2GB) 240-Pin DDR2 SDRAM DDR2 800(PC2 6400)
CPU Cooler 1 Dynatron A48G 70mm 2 Ball CPU Cooler
Slim DVD 1 Sony Optiarc Slim Combo Black SATA Model CRX890S-10-OEM
PSU 1 Zippy
Hard Drive 8 WD AAKS
RAID Controller 1 3Ware 9550 SXu
Some of the above hardware specs are not necessarily designed for servers, such as the motherboard/processor, however they were salvaged from existing proven hardware to accommodate upgrades in other areas of the datacenter.
Depending on the requirements of the RAID level configuration, and other hardware the drives that participate in a LUN and their physcial location in the chassis can inadvertantly diminish redundancy. The NORCO RPC-2008 2U Rackmount Server Case hase two horizontal back planes. Given this configuration, in order to mitigate loss of a LUN, 4 RAID level 1 LUNs were configured in vertical fashion. Should one back plane fail, the array would not fall offline, but simply enter degraded mode, until the back plane is replaced.
Installation
The 3Ware 9550SXU is a supported controller for OpenFiler, however, upon installation, there was required the need to jockey the units the controller presented, both before and after installation of OpenFiler. Upon the first go, and a successful completion of the OF installation, OF will not boot. To fix this, reboot the system and in the controller bios, select the LUN with the <enter> key, then <page up>, then <enter>, then <F8> to save. This moves the LUN index so that the bootloader will be discovered upon powering on the system, and OF will start for the first time. This additional step is cumbersome, as the OF installation wizard will identify the device as sda, but after starting OF, performing cat /proc/partitions will show that the operating system was installed to sdb instead. If this is acceptable, proceed, otherwise, install to the counter partition, and switch the controller’s LUN index as needed.
Partitions & Volumes
The following table illustrates the partition scheme that will be used for the remainder of the cluster configuration.
Device Mount Type Size Start Point End Point
/dev/sda
- /dev/sda1 /boot ext3 4016 1 512
- /dev/sda2 / ext3 8032 513 1536
- /dev/sda3 swap 4016 1537 2048
- /dev/sda4 Extended 460755 2049 60786
- /dev/sda5 LVM PV 1012 2049 2172
- /dev/sda6 LVM PV 459735 2179 60786
/dev/sdb
- /dev/sdb1 LVM PV 476820 1 60786
Network Configuration
Also during installation, it is advised to configure networking, rather than after. If changes are required after installation, this can be done in the following manner:
BOOTPROTO=static
BROADCAST=192.168.10.255
HWADDR=00:18:F3:65:0E:3C
IPADDR=192.168.10.150
NETMASK=255.255.255.0
NETWORK=192.168.10.0
ONBOOT=yes
TYPE=Ethernet
Though not required, this installation uses eth1 on a back channel. The following table shows the network configuration that will be used for the remainder of the configuration:
Domain: iscsi00.web-wired.com
Interface IP Subnet
eth0 192.168.10.150 255.255.255.0
eth1 192.168.11.150 255.255.255.0
Domain: iscsi01.web-wired.com
Interface IP Subnet
eth0 192.168.10.160 255.255.255.0
eth1 192.168.11.160 255.255.255.0
Heartbeat will be configured later to operation on the 192.168.10.0/24 network, but this IP will be set at a later point.
Note: After installation, it may be recommended to update. However, this howto does not perform this step because of problems experienced. At a subsequent time, when performing an updateall, the code in the repository that is fetched, may be in a state differing to the time of this writing, nd therefore may work.
iscsi00.web-wired.com
DRBD Configuration
The remaining steps will be best performed using ssh. For Windows users, PuTTY is an excellent tool and can be downloaded here.
Shared Keys
Both node must be able to communicate with each other, and without requiring a password. To do this, shared keys must be setup on each node.
iscsi00.web-wired.com
~# ssh-keygen -t dsa
# scp .ssh/id_dsa.pub root@iscsi01:/.ssh/authorized_keys2
iscsi01.web-wired.com
~# ssh-keygen -t dsa
# scp .ssh/id_dsa.pub root@iscsi00:/.ssh/authorized_keys2
Backing up the original drbd .conf is a good idea. After this, create a new drbd .conf and provide it with the configuration below.
iscsi00.web-wired.com
global {
usage-count no;
}
common {
syncer { rate 100M; }
}
resource r0disk {
protocol C;
handlers {
pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";
pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";
local-io-error "echo O > /proc/sysrq-trigger ; halt -f";
}
startup {
degr-wfc-timeout 120; # 2 minutes.
}
disk {
on-io-error detach;
}
net {
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
}
syncer {
al-extents 257;
}
on iscsi00.web-wired.com {
device /dev/drbd0;
disk /dev/sda5;
address 192.168.11.150:7788;
meta-disk internal;
}
on iscsi01.web-wired.com {
device /dev/drbd0;
disk /dev/sda5;
address 192.168.11.160:7788;
meta-disk internal;
}
}
resource r1disk {
protocol C;
startup {
wfc-timeout 0; ## Infinite!
degr-wfc-timeout 120; ## 2 minutes.
}
disk {
on-io-error detach;
}
net {
}
syncer {
after "r0disk";
}
on iscsi00.web-wired.com {
device /dev/drbd1;
disk /dev/sda6;
address 192.168.11.150:7789;
flexible-meta-disk internal;
}
on iscsi01.web-wired.com {
device /dev/drbd1;
disk /dev/sda6;
address 192.168.11.160:7789;
meta-disk internal;
}
}
iscsi01.web-wired.com
iscsi00.web-wired.com
uname -n
Initialize DRBD
Initialize the metadata space on /dev/drbd0 (r0disk) and /dev/drbd1 (r1disk) on both systems. In essence, this defines the resources at the system block level.
First zero out the partitions.
iscsi00.web-wired.com
Note: if this is the first install, using dd is most likely not necessary, though it is a good idea provided there is no desireable data. Executing these statements will destroy data on the devices entered. This process can take awhile. Do not continue until it is complete.
iscsi00.web-wired.com
Writing meta data...
initialising activity log
NOT initialized bitmap
New drbd meta data block sucessfully created.
Starting DRBD
Perform the following on both OF nodes:
iscsi00.web-wired.com
drbd driver loaded OK; device status:
version: 8.2.7 (api:88/proto:86-88)
GIT-hash: 61b7f4c2fc34fe3d2acf7be6bcc1fc2684708a7d build by phil@fat-tyre, 2008-11-12 16:47:11
m:res cs ro ds p mounted fstype
0:r0disk Connected Secondary/Secondary Inconsistent/Inconsistent C
1:r1disk Connected Secondary/Secondary Inconsistent/Inconsistent C
The above indicates that the node is marked as inconsistent and secondary. This is true for both nodes, as neither have been set as the primary.
The following step will set iscsi00.web-wired.com as the primary Do this step ONLY ON THE INTENDED PRIMARY NODE (iscsi00.web-wired.com).
iscsi00.web-wired.com
drbd driver loaded OK; device status:
version: 8.2.7 (api:88/proto:86-88)
GIT-hash: 61b7f4c2fc34fe3d2acf7be6bcc1fc2684708a7d build by phil@fat-tyre, 2008-11-12 16:47:11
m:res cs ro ds p mounted fstype
0:r0disk Connected Primary/Secondary UpToDate/UpToDate C
... sync'ed: 1.5% (453959/460442)M
1:r1disk SyncSource Primary/Secondary UpToDate/Inconsistent C
Note: If desired, run the following for real time:
iscsi01.web-wired.com
Every 2.0s: service drbd status Sun Jul 12 18:19:20 2009
drbd driver loaded OK; device status:
version: 8.2.7 (api:88/proto:86-88)
GIT-hash: 61b7f4c2fc34fe3d2acf7be6bcc1fc2684708a7d build by phil@fat-tyre, 2008-11-12 16:47:11
m:res cs st ds p mounted fstype
0:r0disk Connected Secondary/Primary UpToDate/UpToDate C
... sync'ed: 6.4% (430585/459721)M
1:r1disk SyncTarget Secondary/Primary Inconsistent/UpToDate C
The remaining steps can be completed while DRBD continues to sync each block device.
On both nodes, enable DRBD to startup at boot.
iscsi00.web-wired.com
Creating DRBD Filesystem
Make a file system for drbd0 (dev/r0disk). Do this only for the primary node.
iscsi00.web-wired.com
iscsi00.web-wired.com
filter =
With this line:
filter = /dev/sda6
iscsi01.web-wired.com
filter =
With this line:
filter = /dev/sda6
Note: Remember to apply the above changes on both nodes. When the host scans for LVM partitions, it excludes the LVM partition, as this seems to cause problems with drbd consistency when this filter is not enabled.
Create a LVM PV. Do this only onthe primary node, as it will replicate to the secondary node via drbd ).
iscsi00.web-wired.com
Physical volume "/dev/drbd1" successfully created
Configuring Heartbeat
Heartbeat controls fail over between hosts. The two nodes run the Heartbeat service. This sends out a heartbeat pulse on eth1. If one node dies, Heartbeat roles over to the surviving node using scripts available in /etc/ha.d/resources.d.
Make modifications to /etc/ha.d/ha.cf and /etc/ha.d/authkeys. Make sure these changes are on both nodes.
Note: /etc/ha.d/authkeys does not exist in 2.3, so create it.
iscsi00.web-wired.com
Edit the following file.
auth 2
2 crc
iscsi01.web-wired.com
Edit the following file.
auth 2
2 crc
Restrict permission to authkeys for root access only.
iscsi00.web-wired.com
iscsi00.web-wired.com
Edit the following file.
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
bcast eth1
keepalive 5
warntime 10
deadtime 120
initdead 120
udpport 694
auto_failback off
node iscsi00.web-wired.com
node iscsi01.web-wired.com
iscsi01.web-wired.com
Edit the following file.
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
bcast eth1
keepalive 5
warntime 10
deadtime 120
initdead 120
udpport 694
auto_failback off
node iscsi00.web-wired.com
node iscsi01.web-wired.com
Enable Heartbeat to start up at boot. Make sure these changes are on both nodes.
iscsi00.web-wired.com
Note: It is important in the following steps to retain the permissions and ownership of files and folders when moving/copying, otherwise use of the web interface will fail to function and apply changes in some areas, such as iSCSI setup.
iscsi00.web-wired.com
~# mkdir /r0disk
~# mount /dev/drbd0 /r0disk
~# mv /opt/openfiler/ /opt/openfiler.local
~# mkdir /r0disk/opt
~# cp -r /opt/openfiler.local /r0disk/opt/openfiler
~# ln -s /r0disk/opt/openfiler /opt/openfiler
~# rm /r0disk/opt/openfiler/sbin/openfiler
~# ln -s /usr/sbin/httpd /r0disk/opt/openfiler/sbin/openfiler
~# rm /r0disk/opt/openfiler/etc/rsync.xml
~# ln -s /opt/openfiler.local/etc/rsync.xml /r0disk/opt/openfiler/etc/
Edit the /opt/openfiler.local/etc/rsync.xml file.
<?xml version="1.0" ?>
<rsync>
<remote hostname="192.168.11.160"/> ## IP address of Secondary node (eth1).
<item path="/etc/ha.d/haresources"/>
<item path="/etc/ha.d/ha.cf"/>
<item path="/etc/ietd.conf"/>
<item path="/etc/ldap.conf"/>
<item path="/etc/openldap/ldap.conf"/>
<item path="/etc/ldap.secret"/>
<item path="/etc/nsswitch.conf"/>
<item path="/etc/krb5.conf"/>
</rsync>
Additionally, create a space where the httpd service can live.
~# mkdir /r0disk
~# mv /opt/openfiler/ /opt/openfiler.local
~# ln -s /r0disk/opt/openfiler /opt/openfiler
Edit the /opt/openfiler.local/etc/rsync.xml file.
<?xml version="1.0" ?>
<rsync>
<remote hostname="192.168.11.150"/> ## IP address of Primary node (eth1).
<item path="/etc/ha.d/haresources"/>
<item path="/etc/ha.d/ha.cf"/>
<item path="/etc/ietd.conf"/>
<item path="/etc/ldap.conf"/>
<item path="/etc/openldap/ldap.conf"/>
<item path="/etc/ldap.secret"/>
<item path="/etc/nsswitch.conf"/>
<item path="/etc/krb5.conf"/>
</rsync>
Note: The rsync configuration is meant to synchronize changes between the two nodes, but this will not happen unless using the most recent build. Therefore, see the fix here.
Modify the /r0disk/opt/openfiler/etc/cluster.xml config file. This config file tells Heartbeat what to do in an event of a fail over. It is also where the cluster IP address is defined. All external resources should point to this IP in order to properly fail over. Make sure these changes are on iscsi00.web-wired.com only.
iscsi00.web-wired.com
<?xml version="1.0" ?>
<cluster>
<clustering state="on" />
<nodename value="iscsi00.web-wired.com" />
<resource value="MailTo::admin@web-wired.com::ClusterFailover"/>
<resource value="IPaddr::192.168.10.140/24" />
<resource value="drbddisk::" />
<resource value="LVM::r1disk" />
<resource value="Filesystem::/dev/drbd0::/r0disk::ext3::defaults,noatime" />
<resource value="MakeMounts" />
</cluster>
Note: The HA IP Address is declared here as 192.168.10.140. Heartbeat controls both the setup of the network interface and the mounting of our LVM VG and /dev/drbd0 (/r0disk).
Do not start heartbeat yet.
Samba and NFS Support
Modify Samba and NFS so it’s available on the /r0disk drbd resource.
iscsi00.web-wired.com
~# mkdir /r0disk/etc
~# mv /etc/samba/ /r0disk/etc/
~# ln -s /r0disk/etc/samba/ /etc/samba
~# mkdir -p /r0disk/var/spool
~# mv /var/spool/samba/ /r0disk/var/spool/
~# ln -s /r0disk/var/spool/samba/ /var/spool/samba
~# mkdir -p /r0disk/var/lib
~# mv /var/lib/nfs/ /r0disk/var/lib/
~# ln -s /r0disk/var/lib/nfs/ /var/lib/nfs
~# mv /etc/exports /r0disk/etc/
~# ln -s /r0disk/etc/exports /etc/exports
iscsi01.web-wired.com
~# rm -rf /etc/samba/
~# ln -s /r0disk/etc/samba/ /etc/samba
~# rm -rf /var/spool/samba/
~# ln -s /r0disk/var/spool/samba/ /var/spool/samba
~# rm -rf /var/lib/nfs/
~# ln -s /r0disk/var/lib/nfs/ /var/lib/nfs
~# rm -rf /etc/exports
~# ln -s /r0disk/etc/exports /etc/exports
iSCSI Support
iscsi00.web-wired.com
~# mv /etc/ietd.conf /r0disk/etc/
~# ln -s /r0disk/etc/ietd.conf /etc/ietd.conf
~# mv /etc/initiators.allow /r0disk/etc/
~# ln -s /r0disk/etc/initiators.allow /etc/initiators.allow
~# mv /etc/initiators.deny /r0disk/etc/
~# ln -s /r0disk/etc/initiators.deny /etc/initiators.deny
iscsi01.web-wired.com
Setup the symbolic links. They will appear broken from their origins until rsync does its job.
~# rm /etc/ietd.conf
~# ln -s /r0disk/etc/ietd.conf /etc/ietd.conf
~# rm /etc/initiators.allow
~# ln -s /r0disk/etc/initiators.allow /etc/initiators.allow
~# rm /etc/initiators.deny
~# ln -s /r0disk/etc/initiators.deny /etc/initiators.deny
FTP Support
iscsi00.web-wired.com
~# mv /etc/proftpd /r0disk/etc/
~# ln -s /r0disk/etc/proftpd/ /etc/proftpd
iscsi01.web-wired.com
~# rm -rf /etc/proftpd
~# ln -s /r0disk/etc/proftpd/ /etc/proftpd
Configure a VG for drbd1
Create a Volume group from /dev/drbd1.
iscsi00.web-wired.com
Volume group "r1disk" successfully created
Note: If planning on using Windows to connect to these iSCSI targets, do not use the “_†character or any other special characters when creating the volume group.
Heartbeat Configuration Final Steps
iscsi00.web-wired.com
Make sure these changes are on iscsi00.web-wired.com only.
Stopping openfiler: Re: Openfier+DRBD+Heartbeat
Starting openfiler: Re: Openfier+DRBD+Heartbeat
Log on the the web interface and select the Services tab. Enable the following services:
SMB / CIFS server
NFSv3 server
FTP server
iSCSI target server
Rsync server
This should automatically generate the /etc/ha.d/haresources file required for clustering. SCP this file to the secondary node.
Note: The /etc/ha.d/haresources file does not appear to be generated until entering the web interface, selecting the Services tab, and enabling rsync or iSCSI. This must occur after the cluster.xml file is configured and all other shared resource services are configured.
~# scp /etc/ha.d/haresources root@iscsi01.web-wired.com:/etc/ha.d/haresources
The /etc/ha.d/haresources file will look something like this.
iscsi00.web-wired.com MailTo::admin@web-wired.com::ClusterFailover IPaddr::192.168.10.140/24 drbddisk:: LVM::r1disk Filesystem::/dev/drbd0::/r0disk::ext3::defaults,noatime MakeMounts proftpd nfs smb iscsi-target rsync acpid openfiler
Note: Before starting Heartbeat, a volume must be created on /dev/drbd1.
~# lvcreate -L 512M -n lvm00r1disk r1disk
The rsync configuration is meant to synchronise changes between the two filers, but from observation this does not appear to happen. Do not modify /etc/ha.d/haresources directly, as it will get overwritten. Always use /r0disk/opt/openfiler/etc/cluster.xml.
Also it is a good idea to copy over the /etc/ha.d/haresources file everytime lvm volumes change as this can alter the haresources file and prevent the second filer from failing over.
- End.
====
Доброй охоты! 
Взято от сюда
=====
Chassis 1 NORCO RPC-2008 2U Rackmount Server Case
Motherboard 1 ASUS M2N32-WS Pro AM2
CPU 1 AMD Athlon 64 X2 5000+ Brisbane 2.6GHz Socket AM2 65W Dual-Core
Memory 2 Kingston HyperX 4GB (2 x 2GB) 240-Pin DDR2 SDRAM DDR2 800(PC2 6400)
CPU Cooler 1 Dynatron A48G 70mm 2 Ball CPU Cooler
Slim DVD 1 Sony Optiarc Slim Combo Black SATA Model CRX890S-10-OEM
PSU 1 Zippy
Hard Drive 8 WD AAKS
RAID Controller 1 3Ware 9550 SXu
/dev/sda
- /dev/sda1 /boot ext3 4016 1 512
- /dev/sda2 / ext3 8032 513 1536
- /dev/sda3 swap 4016 1537 2048
- /dev/sda4 Extended 460755 2049 60786
- /dev/sda5 LVM PV 1012 2049 2172
- /dev/sda6 LVM PV 459735 2179 60786
/dev/sdb
- /dev/sdb1 LVM PV 476820 1 60786
- vi /etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=static
BROADCAST=192.168.10.255
HWADDR=00:18:F3:65:0E:3C
IPADDR=192.168.10.150
NETMASK=255.255.255.0
NETWORK=192.168.10.0
ONBOOT=yes
TYPE=Ethernet
Though not required, this installation uses eth1 on a back channel. The following table shows the network configuration that will be used for the remainder of the configuration:
Interface IP Subnet
eth0 192.168.10.150 255.255.255.0
eth1 192.168.11.150 255.255.255.0
Interface IP Subnet
eth0 192.168.10.160 255.255.255.0
eth1 192.168.11.160 255.255.255.0
- conary updateall
- conary updateall
# scp .ssh/id_dsa.pub root@iscsi01:/.ssh/authorized_keys2
iscsi01.web-wired.com
# scp .ssh/id_dsa.pub root@iscsi00:/.ssh/authorized_keys2
Backing up the original drbd .conf is a good idea. After this, create a new drbd .conf and provide it with the configuration below.
- mv /etc/ drbd .conf /etc/ drbd .conf.bak.000
- vi /etc/ drbd .conf
usage-count no;
}
syncer { rate 100M; }
}
pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";
pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";
local-io-error "echo O > /proc/sysrq-trigger ; halt -f";
}
degr-wfc-timeout 120; # 2 minutes.
}
on-io-error detach;
}
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
}
al-extents 257;
}
device /dev/drbd0;
disk /dev/sda5;
address 192.168.11.150:7788;
meta-disk internal;
}
device /dev/drbd0;
disk /dev/sda5;
address 192.168.11.160:7788;
meta-disk internal;
}
}
wfc-timeout 0; ## Infinite!
degr-wfc-timeout 120; ## 2 minutes.
}
on-io-error detach;
}
}
after "r0disk";
}
device /dev/drbd1;
disk /dev/sda6;
address 192.168.11.150:7789;
flexible-meta-disk internal;
}
device /dev/drbd1;
disk /dev/sda6;
address 192.168.11.160:7789;
meta-disk internal;
}
}
iscsi01.web-wired.com
- mv /etc/ drbd .conf /etc/ drbd .conf.bak.000
- scp /etc/ drbd .conf root@iscsi01.web-wired.com:/etc/ drbd .conf
Initialize DRBD
- dd if=/dev/zero of=/dev/sda5
- dd if=/dev/zero of=/dev/sda6
- dd if=/dev/zero of=/dev/sda5
- dd if=/dev/zero of=/dev/sda6
- drbdadm create-md r0disk
- drbdadm create-md r1disk
- drbdadm create-md r0disk
- drbdadm create-md r1disk
initialising activity log
NOT initialized bitmap
New drbd meta data block sucessfully created.
Starting DRBD
- service drbd start
- service drbd start
- Starting DRBD resources: Re: Openfier+DRBD+Heartbeat.
- service drbd status
version: 8.2.7 (api:88/proto:86-88)
GIT-hash: 61b7f4c2fc34fe3d2acf7be6bcc1fc2684708a7d build by phil@fat-tyre, 2008-11-12 16:47:11
m:res cs ro ds p mounted fstype
0:r0disk Connected Secondary/Secondary Inconsistent/Inconsistent C
1:r1disk Connected Secondary/Secondary Inconsistent/Inconsistent C
The above indicates that the node is marked as inconsistent and secondary. This is true for both nodes, as neither have been set as the primary.
- drbdsetup /dev/drbd0 primary -o
- drbdsetup /dev/drbd1 primary -o
- service drbd status
version: 8.2.7 (api:88/proto:86-88)
GIT-hash: 61b7f4c2fc34fe3d2acf7be6bcc1fc2684708a7d build by phil@fat-tyre, 2008-11-12 16:47:11
m:res cs ro ds p mounted fstype
0:r0disk Connected Primary/Secondary UpToDate/UpToDate C
... sync'ed: 1.5% (453959/460442)M
1:r1disk SyncSource Primary/Secondary UpToDate/Inconsistent C
Note: If desired, run the following for real time:
- watch service drbd status
version: 8.2.7 (api:88/proto:86-88)
GIT-hash: 61b7f4c2fc34fe3d2acf7be6bcc1fc2684708a7d build by phil@fat-tyre, 2008-11-12 16:47:11
m:res cs st ds p mounted fstype
0:r0disk Connected Secondary/Primary UpToDate/UpToDate C
... sync'ed: 6.4% (430585/459721)M
1:r1disk SyncTarget Secondary/Primary Inconsistent/UpToDate C
The remaining steps can be completed while DRBD continues to sync each block device.
- chkconfig --level 2345 drbd on
- chkconfig --level 2345 drbd on
- mkfs.ext3 /dev/drbd0
- vi /etc/lvm/lvm.conf
With this line:
iscsi01.web-wired.com
- vi /etc/lvm/lvm.conf
With this line:
Note: Remember to apply the above changes on both nodes. When the host scans for LVM partitions, it excludes the LVM partition, as this seems to cause problems with drbd consistency when this filter is not enabled.
- pvcreate /dev/drbd1
Configuring Heartbeat
- vi /etc/ha.d/authkeys
2 crc
iscsi01.web-wired.com
- vi /etc/ha.d/authkeys
2 crc
Restrict permission to authkeys for root access only.
- chmod 600 /etc/ha.d/authkeys
- chmod 600 /etc/ha.d/authkeys
- vi /etc/ha.d/ha.cf
logfile /var/log/ha-log
logfacility local0
bcast eth1
keepalive 5
warntime 10
deadtime 120
initdead 120
udpport 694
auto_failback off
node iscsi00.web-wired.com
node iscsi01.web-wired.com
iscsi01.web-wired.com
- vi /etc/ha.d/ha.cf
logfile /var/log/ha-log
logfacility local0
bcast eth1
keepalive 5
warntime 10
deadtime 120
initdead 120
udpport 694
auto_failback off
node iscsi00.web-wired.com
node iscsi01.web-wired.com
Enable Heartbeat to start up at boot. Make sure these changes are on both nodes.
- chkconfig --level 2345 heartbeat on
- chkconfig --level 2345 heartbeat on
~# mount /dev/drbd0 /r0disk
~# mv /opt/openfiler/ /opt/openfiler.local
~# mkdir /r0disk/opt
~# cp -r /opt/openfiler.local /r0disk/opt/openfiler
~# ln -s /r0disk/opt/openfiler /opt/openfiler
~# rm /r0disk/opt/openfiler/sbin/openfiler
~# ln -s /usr/sbin/httpd /r0disk/opt/openfiler/sbin/openfiler
~# rm /r0disk/opt/openfiler/etc/rsync.xml
~# ln -s /opt/openfiler.local/etc/rsync.xml /r0disk/opt/openfiler/etc/
Edit the /opt/openfiler.local/etc/rsync.xml file.
- vi /opt/openfiler.local/etc/rsync.xml
<rsync>
<remote hostname="192.168.11.160"/> ## IP address of Secondary node (eth1).
<item path="/etc/ha.d/haresources"/>
<item path="/etc/ha.d/ha.cf"/>
<item path="/etc/ietd.conf"/>
<item path="/etc/ldap.conf"/>
<item path="/etc/openldap/ldap.conf"/>
<item path="/etc/ldap.secret"/>
<item path="/etc/nsswitch.conf"/>
<item path="/etc/krb5.conf"/>
</rsync>
Additionally, create a space where the httpd service can live.
- mkdir -p /r0disk/etc/httpd/conf.d
~# mv /opt/openfiler/ /opt/openfiler.local
~# ln -s /r0disk/opt/openfiler /opt/openfiler
Edit the /opt/openfiler.local/etc/rsync.xml file.
- vi /opt/openfiler.local/etc/rsync.xml
<rsync>
<remote hostname="192.168.11.150"/> ## IP address of Primary node (eth1).
<item path="/etc/ha.d/haresources"/>
<item path="/etc/ha.d/ha.cf"/>
<item path="/etc/ietd.conf"/>
<item path="/etc/ldap.conf"/>
<item path="/etc/openldap/ldap.conf"/>
<item path="/etc/ldap.secret"/>
<item path="/etc/nsswitch.conf"/>
<item path="/etc/krb5.conf"/>
</rsync>
Note: The rsync configuration is meant to synchronize changes between the two nodes, but this will not happen unless using the most recent build. Therefore, see the fix here.
- vi /r0disk/opt/openfiler/etc/cluster.xml
<cluster>
<clustering state="on" />
<nodename value="iscsi00.web-wired.com" />
<resource value="MailTo::admin@web-wired.com::ClusterFailover"/>
<resource value="IPaddr::192.168.10.140/24" />
<resource value="drbddisk::" />
<resource value="LVM::r1disk" />
<resource value="Filesystem::/dev/drbd0::/r0disk::ext3::defaults,noatime" />
<resource value="MakeMounts" />
</cluster>
Note: The HA IP Address is declared here as 192.168.10.140. Heartbeat controls both the setup of the network interface and the mounting of our LVM VG and /dev/drbd0 (/r0disk).
~# mv /etc/samba/ /r0disk/etc/
~# ln -s /r0disk/etc/samba/ /etc/samba
~# mkdir -p /r0disk/var/spool
~# mv /var/spool/samba/ /r0disk/var/spool/
~# ln -s /r0disk/var/spool/samba/ /var/spool/samba
~# mkdir -p /r0disk/var/lib
~# mv /var/lib/nfs/ /r0disk/var/lib/
~# ln -s /r0disk/var/lib/nfs/ /var/lib/nfs
~# mv /etc/exports /r0disk/etc/
~# ln -s /r0disk/etc/exports /etc/exports
iscsi01.web-wired.com
~# ln -s /r0disk/etc/samba/ /etc/samba
~# rm -rf /var/spool/samba/
~# ln -s /r0disk/var/spool/samba/ /var/spool/samba
~# rm -rf /var/lib/nfs/
~# ln -s /r0disk/var/lib/nfs/ /var/lib/nfs
~# rm -rf /etc/exports
~# ln -s /r0disk/etc/exports /etc/exports
iSCSI Support
~# ln -s /r0disk/etc/ietd.conf /etc/ietd.conf
~# mv /etc/initiators.allow /r0disk/etc/
~# ln -s /r0disk/etc/initiators.allow /etc/initiators.allow
~# mv /etc/initiators.deny /r0disk/etc/
~# ln -s /r0disk/etc/initiators.deny /etc/initiators.deny
iscsi01.web-wired.com
~# ln -s /r0disk/etc/ietd.conf /etc/ietd.conf
~# rm /etc/initiators.allow
~# ln -s /r0disk/etc/initiators.allow /etc/initiators.allow
~# rm /etc/initiators.deny
~# ln -s /r0disk/etc/initiators.deny /etc/initiators.deny
FTP Support
~# ln -s /r0disk/etc/proftpd/ /etc/proftpd
iscsi01.web-wired.com
~# ln -s /r0disk/etc/proftpd/ /etc/proftpd
Configure a VG for drbd1
- vgcreate r1disk /dev/drbd1
Note: If planning on using Windows to connect to these iSCSI targets, do not use the “_†character or any other special characters when creating the volume group.
- rm /opt/openfiler/etc/httpd/modules
- ln -s /usr/lib64/httpd/modules /opt/openfiler/etc/httpd/modules
- service openfiler restart
Starting openfiler: Re: Openfier+DRBD+Heartbeat
Log on the the web interface and select the Services tab. Enable the following services:
NFSv3 server
FTP server
iSCSI target server
Rsync server
This should automatically generate the /etc/ha.d/haresources file required for clustering. SCP this file to the secondary node.
The /etc/ha.d/haresources file will look something like this.
Note: Before starting Heartbeat, a volume must be created on /dev/drbd1.
The rsync configuration is meant to synchronise changes between the two filers, but from observation this does not appear to happen. Do not modify /etc/ha.d/haresources directly, as it will get overwritten. Always use /r0disk/opt/openfiler/etc/cluster.xml.
Взято от сюда
