Fail-over Load Balancing of Zoned Apache Instances

Someone recently asked on zones-discuss how to implement load balanced fail-over of zoned apache instances.  It turns out that they were actually asking how to configure a scalable Sun/Oracle Solaris Cluster of zoned apache instances.  However, I am just going to show you how to use the Zone Manager [ Download here ] to create a set of three load balanced zoned apache instances using balance.

Balance is a flexible and very simple reverse proxy solution that gets the job done.  Certainly there are other viable commercial and open source projects available to facilitate apache load balancing with fail-over.  However, for the sake of brevity and simplicity, this example uses balance.

Here is all that you need to do to setup a load balanced set of zoned apache instances.

1. Add the following three entries into the /etc/hosts of the global zone so that the Zone Manager can resolve the automatically generated zone names to IP addresses.


192.168.0.96 zone0001
192.168.0.97 zone0002
192.168.0.98 zone0003

2. Add three zones to one or more servers.

# zonemgr -F -a add -o 'dCount|3' -I "ns|e1000g1|24|all" \
   -s jail -G balance -G apache2

This invocation of the Zone Manager creates three zones that have been hardened such that all un-necessary services have been disabled.   Each zone also has been configured with a unique IP address according to name resolution from /etc/hosts.  Lastly, balance and apache2 were installed from Blastwave.org.

Note that if I wanted all four of the these apache instances to point to a single set of content, I could have used -r /data/www to read-only mount a custom apache document root /data/www into each of the four zones.  Of course, you would need to re-configure each of the apache instances to point their document root directories to  /data/www.   There are two key benefits to this option.  First, the web server content cannot be altered from within the non-global zones.  So, if someone compromises any of the four apache instances, they cannot change the content unless of course point the apache server document root to an a different document root that they can change. The second benefit of this option is that the contents of the document root only needs to be cached into the ZFS filesystem once as opposed to once per apache instance.  This can result in much better caching efficiency for many apache instances.

3. Run balance in each zone on port 81
# zlogin zone0001 \
  "/opt/csw/sbin/balance 81 192.168.0.96:80 192.168.0.97:80 192.168.0.98:80"

# zlogin zone0002 \
  "/opt/csw/sbin/balance 81 192.168.0.96:80 192.168.0.97:80 192.168.0.98:80"

# zlogin zone0003 \
  "/opt/csw/sbin/balance 81 192.168.0.96:80 192.168.0.97:80 192.168.0.98:80"

There are many options with balance such as fail-over groups, sticky sessions, etc that you can take advantage of in the balance configuration.  See balance man page for full set of configuration options.

4. Make each apache instance distinct to see the load balancing by editing the index.html of each apache instance.

vi /zones/zone000*/root/opt/csw/apache2/share/htdocs/index.html

Within each index.html, find and replace "Apache HTTP Server Test Page" with the respective non-global zone name.  e.g. zone0001, zone0002, and zone0003.

5. The last step is simply to bring up a browser and point it to any of the four web pages and hit refresh every second or so to see the back end web server change.  Be sure to hold down the shift key or which ever key is necessary to force reading the content from web server instead of reading the content from the browser's cache.

   http://64.182.70.96:81/

   http://64.182.70.97:81/

   http://64.182.70.98:81/

As an optional last step, you can also configure DNS to do round robin load balancing across the three IP addresses so that you can point the browser to a single domain name instead of going to IP addresses. Further, could also configure the apache instances to use a non-standard and non-privileged port like 8000 so that you could configure balance to use port 80 rather than port 81.  Port 81 was just easier to use as in my example.

That is it.  Have a great day!

Brad

ZMU 101: How To Add And Delete A Zone

My foremost goal with this blog is to educate folks on how to harness the full potential of Solaris and Zones.  Toward that end, I am kicking off a series of blog posts that will walk you through the basics and some advanced uses of the Zone Manager.  I am titling the campaign series Zone Manager University (a.k.a. ZMU).  Before you get started, be sure to download the latest version of the Zone Manager from the Zone Manager Download Site.

This first post in the Zone Manager University series deals with adding and deleting zones.  I will first show you how to at delete zones because there are far fewer options to consider when deleting one  zones the compared to adding zones.  Plus, if you plan on running each of the "add examples", you will need to delete the zone(s) in between each "add example".  Before we get started with adding and deleting zones, lets first look at two very useful options.

List, Debug And Keeping Artifacts

There are three very useful items that enable you to understand how Solaris zones and the Zone Manager script works.  

Item 1 - List Zones

# zonemgr -F -a list

The first item is that you can use the "list" action to list the non-global zones that exist on the server.  This is very useful for example to make sure that the zone you are about to add doesn't already exist.

Item 2 - Enabling Debugging

# zonemgr -F -a list -o debug

The second item is the debugging option.  The "-o debug" option turns on debugging so that you can see what the Zone Manager is doing as it runs.  This option can be applied to any action.

Item 3 - Keep Artifacts

# zonemgr -F -a add -o keep_artifacts

The third item is the -o keep_artifacts option.   The Zone Manager creates several artifacts as it runs.  These artifacts can be useful for troubleshooting a problem or simply to understand how zones and the Zone Manager works.  By default, all Zone Manager artifacts are removed when the script finishes running.  However, this cleansing of artifacts can be disabled by passing the "-o keep_artifacts" option.

Lets look at some of the artifacts produced.

  zone-0-cfg - The zone configuration file used to add the zone.

  zone-0-sysidcfg  - The system configuration information used to configure Solaris on first boot.

  zone-0.hosts-local - The /etc/hosts file created for the zone.

  zone-0-out.log - The zone installation log file.

With those items out of the way, lets look at deleting zones.

Deleting Zones

In this section, you will learn through six examples how to delete one or more zones.  Note that when you get to adding zones, if you are running these examples as you go, you will probably need to remove the zone(s) before proceeding on to the next example.  These deletion methods will help you to delete the zones created by the Add examples.

Example 1 - Delete A Single Specified Zone

# zonemgr -F -a del -n z1

In this example, you delete a single zone named z1.

Example 2 - Delete Multiple Specified Zones

# zonemgr -F -a del -n "z1|z2"

In this example, you delete two zones named z1 and z2.

Example 3 - Delete All Non-Global Zones

# zonemgr -F -a del -n "allzones"

    or

# zonemgr -F -a del -n *

In this example, you delete all non-global zones on the server.

Example 4 - Delete All Zones That Begin With The Word "zone"

# zonemgr -F -a del -n "^zone*"

In this example, you delete all zones for which the zone name begins with the word "zone".

Example 5 - Delete All Zones Beginning With "zone" And Ending With "1"

# zonemgr -F -a del -n "^zone*1$"

In this example, you delete all zones for which the zone name begins with the word "zone" and that end with the number "1".

Example 6 - Delete All Zones Beginning That Include The Word "zone"

# zonemgr -F -a del -n "*zone*"

In this example, you delete all zones for which the zone name contains the word "zone" within the zone name.

Adding Zones

In this section you will learn through twenty-one examples how to add one or more zones.  Adding a zone with the Zone Manager is much easier than adding a zone by just about any other means.  Not only is it easy but it also enables significantly more possibilities as well. 

Example 1 - Add A Single Zone Using All Defaults

# zonemgr -F -a add

The first example adds a zone named zone0001 with a root zone directory of /zones/zone0001 in zfs filesystem rpool/zone0001 and sets the root user's password of zone0001 to the same password of the root user of the global zone.  

The zone root directory consists of a base directory plus the leaf directory.  The default zone base directory is /zones.  The default leaf directory is the zone name (e.g. zone0001).   Thus the combined default root zone directory would be /zones + /zone0001 which becomes /zones/zone0001.

If the filesystem containing the zone base directory (e.g. /zones) is a ZFS filesystem, the zpool of that ZFS filesystem will be used as the base for the filesystem containing the zone root directory.  In my case, the ZFS zpool containing /zones directory is rpool.  The default leaf filesystem name is the same as the zone name (e.g. zone0001).  The combined ZFS filesystem then is rpool/zone0001.

Note that if the zone name is not specified, the zone name is automatically generated based on available names.  If no generated zone names are taken, the first name is zone0001.  The second would be zone0002.  Successive zone names would increment through to zone9999.

Example 2 - Add A Single Specified Zone

# zonemgr -F -a add -n z1

This example varies from the first in that you specify the name of the zone instead of letting zonemgr figure out the next available automatically generated zone name.  This example also differs in that the root zone directory is in /zones/z1 and the corresponding zfs filesystem rpool/z1.

Example 3 - Custom Base Zone Root Directory

# zonemgr -F -a add -n z1 -Z /export

This example differs from example 2 in that you set the base directory of the zone root directory with the -Z flag.  The new base directory is now /export instead of the default of /zones.  Likewise, the zfs filesystem differs as well because the zpool containing /export is exportfs rather than rpool.  The zfs filesystem is exportfs/z1.

Example 4 - Custom Fully Qualified Zone Root Directory

# zonemgr -F -a add -n z1 -z /export/zone1

The -z flag is used to specify the fully qualified path of the zone root directory.  Thus, the zone root directory in this example is /export/zone1.  Note that the zfs filesystem containing the zone root directory is still derived from the zone root directory.  The zfs filesystem is exportfs/z1.

Example 5 - Custom Base Zone Root Directory ZFS Filesystem

# zonemgr -F -a add -n z1 -Z "/export|exportfs/zones"

In this example, a second parameter (e.g. exportfs/zones) was added to -Z.  This second parameter enables you to specify the base zfs filesystem to use when creating the / zfs filesystem.  Thus, the zfs filesystem in this example for the zone root directory is exportfs/zones/z1.

Example 6 - Custom And Customized Base Zone Root Directory ZFS Filesystem

# zonemgr -F -a add -n z1 -Z "/export|exportfs/zones|compress=gzip;recordsize=8k"

This example extends example 5 by customizing the zfs filesystem containing the zone root directory by adding some zfs options (e.g. compression=gzip;recordsize=8k).  In this case, you compressed with the gzip algorithm and reduced the recordsize from 128k to 8k of the zfs filesystem (e.g. exportfs/zones/z1).

Example 7 - Custom Fully Qualified Zone Root Directory ZFS Filesystem

# zonemgr -F -a add -n z1 -z "/export|exportfs/zones/zone1"

In this example, a second parameter (e.g. exportfs/zones/zone1) was added to -z.  This second parameter enables you to specify the fully qualified zfs filesystem to use for the zone root directory.

Example 8 - Custom And Customized Fully Qualified Zone Root Directory ZFS Filesystem

# zonemgr -F -a add -n z1 -z "/export|exportfs/zones/zone1| compress=gzip;recordsize=8k"

This example extends example 7 by customizing the zfs filesystem containing the zone root directory by adding some zfs options (e.g. compression=gzip;recordsize=8k).  In this case, you compressed with the gzip algorithm and reduced the recordsize from 128k to 8k of the zfs filesystem (e.g. exportfs/zones/z1).

Example 9 - Custom Password

# zonemgr -F -a add -n z1 -P mypwd

Note from example 1 that the default password of the non-global zone z1 is the same password as the root user of the global zone.  This example sets a specific password of mypwd with the -P flag.

Example 10 - Custom Encrypted Password

# zonemgr -F -a add -n z1 -E "2C5fE1n6hFA7E"

Like example 9, the -E flag is used to set the password of the root user of the non-global zone z1. The difference is that text passed to the -E flag is expected to be in the salted crypt form used by /etc/shadow.

Example 11 - Custom Password Via Input Password File

# zonemgr -F -a add -n z1 -P /.zonemgr/.pwfile

   or 

# zonemgr -F -a add -n z1 -E /.zonemgr/.pwfile

Like examples 9 and 10, this example sets the password of the root user of the non-global zone z1.  The difference though is that a file name is passed to the -P and -E flags rather than providing the password value in clear text as a text argument to the -P and -E flags.  Using a file is generally considered more secure than passing the text.

Example 12 - Custom Root User Home Directory

# zonemgr -F -a add -n z1 -R /root

In this example, the home directory of the root user of the non-global zone z1 is changed from the Solaris default of / to /root.

Example 13 - Custom Root User Shell

# zonemgr -F -a add -n z1 -R "/root|/usr/bin/ksh"

In this example, the home directory of the home directory and the shell of the root user of the non-global zone z1 are set to /root and /usr/bin/ksh respectively.

Example 14 - Enable Root User SSH Login

# zonemgr -F -a add -n z1 -R "/root|/usr/bin/ksh|on"

This example extends example 13 by enabling root login via ssh by setting PermitRootLogin to yes in /etc/ssh/sshd_config of the non-global zone z1.

Example 15 - Disable Autoboot

# zonemgr -F -a add -n z1 -o "comment|this is my zone" -o "autoboot|false"

In this example, you set the zone comment field to "this is my zone" and you disable autoboot. For reference, autoboot determines if the zone should attempt to boot when the global zone is booted or rebooted.

Example 16 - Add Multiple Zones With Specified Zone Names

# zonemgr -F -a add -n "z1|z2|z3"

In this example, you add three zones (e.g. z1, z2, and z3).

Example 17 - Add Multiple Zones With Dynamically Generated Zone Names

# zonemgr -F -a add -n -o "dCount|5"

In this example, you add five zones using the automated zone naming mechanism of the Zone Manager.  If no prior automated zone names existed, the zones created would be named zone0001, zone0002, zone0003, zone0004 and zone0005.

If automated zone names zone0001 and zone0003 were taken, the new zone names would be zone0002, zone0004, zone0005, zone0006 and zone0007.

Example 18 - Add Multiple Zones With Custom Dynamically Generated Zone Names

# zonemgr -F -a add -n -o "dCount|5" -o "dPrefix|z"

This example modifies example 17 by changing the prefix name from the default of "zone" to "z".  Thus the zone names might be z0001, z0002, z0003, z0004, and z0005.

Example 19 - Add Zone With Additional Inherited Directories

# zonemgr -F -a add -o "addDir|/opt/SUNWdsee7|/opt/MyApp"

Thus far, all zones have been of the sparse type (e.g. -t s).  This means that the /lib, /usr, /sbin, and /platform directories are inherited by the non-global zone z1 in read-only mode from the global zone.  The key benefit of the sparse zone type is that disk space is preserved by not duplicating the contents of those filesystems.  Further, less memory is used for the libraries opened from the shared filesystems. 

Example 20 - Add Zone With Less Inherited Directories

# zonemgr -F -a add -o "rmDir|/usr"

This example differs from example 19 in that instead of adding additional directories to be inherited, it removes one of the default sparse root inherited directories (e.g. /usr).  Note that this is not recommended but is possible.

Example 21 - Add Whole Root Zone

# zonemgr -F -a add -t w

As stated in example 19, thus far all zones have been of the sparse type.  This example differs from example 19 by changing the type from sparse to whole root.  A whole root zone does not inherit any directories from the global zone.  There are some cases where a whole root is necessary because the application or service to be run within the non-global zone requires placing one or more files in one or more of the inherited directories (e.g. /lib, /usr, /sbin, or /platform).  In most cases, the app tries to place files within the /usr directory tree.  A common example of this scenario is when you install any of the applications from SunFreeware.com because the apps are installed in /usr/local.

Conclusion

In this blog post you learned about the list action, the debug and keep_artifacts options, and how to add and delete one or more zones using the Zone Manager.  You also learned many of the customizations that can be applied when adding a zone.

The next blog post will focus on managing the state of one or more zones.

Until next time, have a great day!

Brad