Implementing a Linux Virtual Server [30 mn de lecture - paru le 9/19/2005 11:38:59 AM - Public : Expert]

Auteur

Vincent BEDNAREK Elève-Ingénieur Supinfo Paris Promotion SUPINFO 2005    Lui écrire    Tous les projets de cet auteur    Le mini-CV de cet auteur

3. Linux Virtual Server – Installation and configuration

3.1. Scenario

You have a Web site which generates a significant traffic. You decide to increase your availability, by distributing the treatment of requests on several Web servers. We would set up a NAT topology in round Robin mode. Once your Web servers are configured, you will adapt the configuration of LVS according to your needs and will activate for example the Round Robin option.

3.2. Network topology

Here the topology of the network such as we wish to conceive it.

The virtual server uses two network interfaces, one connected to the Internet for the clients to access and the other connected to the internal local area network (LAN), where all the real servers are placed. Scalability is brought by transparently adding or removing real servers from the internal LAN.

Obviously, in this type of configuration, it is necessary that each server has the same Web content. It is possible for that to use NFS shares or other, in order to facilitate replications with each update of the site.

3.3. Round Robin Algorithm

As the diagram shows it, the Round Robin process will do that requests of the first client will be treated by the 1st server of the pool, those of the second client by the second...

In particular cases, the administrator will be able to implement persistence (or sticky) on his architecture. This functionality is more met on trade Web site, requiring a maintenance of session, in order to store session datas for a specific user. This persistence functions either from the IP of the client, or from a cookie stored on the client machine.

This functionality is brought by LVS. Thus, a client having started a connection with a real server will preserve this connection so that the server will keep in memory the user session datas.

3.4. LVS installation

3.4.1. Rebuilding the kernel

Linux systems using kernel versions earlier than 2.4.28 do not have support for virtual server built into the kernel. Therefore, the first step involved in setting them up as a virtual server is to rebuild their kernel with the appropriate patch applied. Kernel versions 2.4.28 or later have LVS support built into them by default and, therefore, require no patching.

The patches can be downloaded from the LVS Web site. Various versions are available for different types of kernel. For this article, we will use a 2.6.10 kernel, so we have not to patch it.

To apply the patch to the kernel, move the patch file to the /usr/src directory and issue the following command as root:

# cd /usr/src/linux*
# gunzip ../linux-2.4.21-ipvs-1.0.10.patch.gz
# patch -p1 < ../linux-2.4.21-ipvs-1.0.10.patch


This will patch the kernel; after that you'll need to compile it.

# cd /usr/src/linux*
# make mrproper
# make oldconfig
# make menuconfig


This will bring up a screen with several subheadings. Select the “Networking Options” subhead, and then “IP:Virtual Server Configuration” in the following screen. Then select the following options:

virtual server support (EXPERIMENTAL)
[*] IP virtual server debugging
(16) IPVS connection table size (the Nth power of 2)
--- IPVS scheduler
round-robin scheduling
weighted round-robin scheduling
least-connection scheduling scheduling
weighted least-connection scheduling
locality-based least-connection scheduling
locality-based least-connection with replication scheduling
destination hashing scheduling
source hashing scheduling
shortest expected delay scheduling
never queue scheduling
--- IPVS application helper
FTP protocol helper


Save the current kernel configuration and exit from menuconfig. Then from the command prompt type :

# make dep
# make clean
# make bzImage
# make modules
# make modules_install


This will create a compressed kernel image (bzImage) in the /usr/src/linux*/arch/i386/boot directory and will also create and install all the modules for the new kernel. Now copy this new kernel image (bzImage) to the /boot directory.

Lastly, either edit your /etc/grub.conf or /etc/lilo.conf file or rename the new kernel image (/boot/bzImage) to the one being referred to in your bootloader configuration file, in order to make your system boot from the new kernel.

3.4.2. Installing IPTables and IPVsadm

After rebuilding the kernel you need to have the IPTables and IPVsadm packages installed on your system to configure it as virtual server. IPTables is used to build, maintain, and inspect IPv4 packet filtering and NAT (network address translation) rules in the Linux kernel. IPVsadm is the administrating utility for the Linux Virtual Server and will be used to set the scheduling algorithm and rules for forwarding client requests to the real servers.

The IPTables package comes bundled with most Linux distributions and can be easily installed from the installation CDs for your distribution. The source RPM for the IPVsadm utility can be obtained from the LVS Web site. For this example we'll use the ipvsadm-1.24-5.src.rpm SRPM package.

Once the packages have been installed you need to enable IP forwarding on the server. Open the file /etc/sysctl.conf in a text editor and set this value:

net.ipv4.ip_forward = 1

In the same time, think to modify the /proc/sys/net/ipv4/ip_forward file:

# echo 1 > /proc/sys/net/ipv4/ip_forward

Next, issue the following command to start the IPTables service on your system. This allows the virtual server to forward replies from the real servers to the clients:

# service iptables start

Warning, if you’re not locally connected to the machine, you take the risk to be disconnected.

3.4.3. Enabling IP masquerading

In order to enable masquerading for the real servers, we will assume that the external Internet interface on your Linux Virtual Server is eth0 and the internal LAN interface (connected to other real servers) is eth1. Therefore, on the server issue these commands:

# iptables -t nat -P POSTROUTING DROP
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

The first command sets up the default policy for IPTables to DROP, which means that if none of the specific rules match, the packet will be dropped. This ensures that not every packet is masqueraded by the server and, thus, provides an extra level of security. The second command enables NAT and masquerades the internal IP addresses of all the real servers to the IP address of the external Internet interface (eth0) of the virtual server. For more on IPTables, refer to official Web site.

3.4.4. Configuring the virtual server using IPVsadm

The next step is to configure the Linux Virtual Server using the IPVsadm utility. But before that, you must allocate proper IP addresses to all the machines on your network. Put the real servers in your internal LAN on a private IP address range, such as 192.68.0.0/255.255.255.0. Also, put the internal LAN interface of the virtual server on the same subnet. Assign the IP address of the internal LAN interface of the virtual server as the default gateway for all the real servers. For the external Internet interface of the virtual server, use a public IP address or the settings provided by your ISP.

In our example, we used 3 real servers running on different operating systems. One with the IP address 192.168.0.1 (providing HTTP service) and others with IP address 192.168.0.2 and .3 (providing both HTTP and FTP services), with the default gateway for both of them set as 192.168.0.254, which is the IP of the internal LAN interface of the virtual server. The external Internet interface of the virtual server had been assigned a public IP address 81.57.54.100.

Now add the virtual service and link a scheduler to it with these commands:

# ipvsadm -A -t 81.57.54.100:80 -s wlc
# ipvsadm -A -t 81.57.54.100:21 -s wrr

The above two commands add wlc (weighted least-connection scheduling) and wrr (weighted round robin scheduling) algorithms for HTTP (port 80) and FTP (port 21) traffic on the virtual server, respectively.

Next, add real servers on the virtual server to which the client requests will be forwarded:

# ipvsadm -a -t 81.57.54.100:80 -r 192.168.0.1:80 -m
# ipvsadm -a -t 81.57.54.100:80 -r 192.168.0.2:80 -m -w 2
# ipvsadm -a -t 81.57.54.100:21 -r 192.168.0.3:80 -m
# ipvsadm -a -t 81.57.54.100:21 -r 192.168.0.2:21 –m
# ipvsadm -a -t 81.57.54.100:21 -r 192.168.0.3:21 -m


This will cause all the HTTP traffic on the virtual server to be forwarded to 192.168.0.1, 192.168.0.2 and 192.168.0.3 according to the scheduling algorithm. All the FTP traffic will go to 192.168.0.2 and 192.168.0.3 only. The real server 192.168.0.2 is given a weight of 2 for HTTP traffic by the -w 2 switch. The default weight is 1.

3.5. Current installation problems

3.5.1. The Virtual IP Address (or VIP)

The virtual server carries on its external network interface an address named "VIP" (Virtual IP). Virtual because behind a single IP address are actually gathered several servers in a completely invisible way for Net surfers.

When the load balancer transmits a packet to a real server, parameters of the packet are not modified (it keeps the same IP address source and destination). It is thus necessary that the real server can accept packet on destination to the VIP. It will be necessary to activate this option on each real server (in the kernel). Thus, when the real server returns the packet to the net surfer, it will indicate the VIP as source IP address for the packet.

In this way, connections of the upper layers are not affected by the mechanisms of LVS.

Concerning ARP resolution, the VIP always returns the MAC address of the Load-Balancer so that packets coming from LVS server towards the real servers are returned towards load-balancer. That imposes a property on the real servers: the interface having VIP address should not solve on ARP level.

3.5.2. ARP problem

The cores Linux 2.2 and 2.4 show different characteristics from core 2.0 for ARP resolutions on virtual interfaces. This problem is regulated in LVS by the configuration script of the real servers specific to each core. In fact, it is not necessary to be concerned with this problem, but it is important to have understood the mechanism of "Virtual IP Address". For more information, defer to the official documentation on the Linux Virtual Server Web site.

3.6. Load balancer configuration

You just need to configure your load balancer with those command lines: I suggest you to copy it in a startup script.

3.7. Tests

After setting everything up, use a client machine to connect to the virtual server using its external IP address. Open multiple connections to the virtual server and check the status of the various connections on the real servers. You will notice that the incoming load is being equally distributed among the real servers. Thus, the virtual server is performing IP load balancing. On the load balancer, you can also control the evolution of your connections with the "ipvsadm" tool.