DNS - Bind9/Named [Part 3]

Right now we have a really nice configuration of bind9. But we have no log about queries and actions. So what we're going to do now is to log in a file different queries and actions. To do we gonna add some lines in named.conf.local (or in options, if you prefer).

logging {
channel query.log
{
     file "/var/log/bind/query.log";
     //Set the severity to dynamic to see all the debug messages.
     severity debug 3;
};
     category default { default_syslog; default_debug; };
     category unmatched { null; };
     category queries { query.log; };
};

DNS - Bind9/Named [Part 2]

Now we have a DNS Server working correctly which serves our domain. We also can want to restrict access to this server. What we're going to see :

  • Restricting access
  • Serve domains according to their original network

So as for anything, we will talk about ACLs. That's very easy to add an ACL in bind. Just edit a file like "lan.acl" and in it you can create your acl rules :

# ACL for VMs on the cloud
acl "cloud_vms" { 10.0.0.0/24; };
# ACL for admin lans
acl "admin_lans" { 192.168.42.0/54; 192.168.150.0/27; };

Now thanks to this configuration, we can identify the origin of the clients. We also have to add the config file in named.conf like this :

include "/etc/bind/lan.acl";

So now, as we've seen just before, we allowed "any" to make queries. So, if you want to limit to the clients only, just edit named.conf.options

allow-query { cloud_vms; admin_lans; localhost; };

But this is very limited. You will serve the same domains to all your clients. And what we want is to serve the same domain with different IPs, we can use the "view" instruction. Just like this :

view "Cloud"
{
     match-clients { cloud_vms; };
     zone "somewhere.net"
     {
             type master;
             file "/etc/bind/db.somewhere.net.cloud";
     };
};
view "Admin"
{
     match-clients { admin_lans; };
     zone "somewhere.net"
     {
             type master;
             file "/etc/bind/db.somewhere.net.admin";
     };
};

If you get an error when restarting the server, comment the include of root-servers. On include in this way :

view "any"
{
     match-clients { any; };
     zone "."
     {
     type master;
     file "/etc/bind/db.root";
     };
};

DNS - Bind9/Named [Part 1]

The most popular DNS server is bind9 also called "named" (bind on Debian and Named on RedHat). There are a lot of tutorials on the internet to learn you how to do crazy things, you will find it easily. But you can spend hours to find what you want. So, here I'm going to see with you the setup and configuration of bind.

In this tutorial, I will be working on Debian 7, but all configuration files of bind will also work on RedHat (the paths will change).

What are we going to see ?

  • Setup DNS server
  • Basic zone configurations
  • Basic server configurations

Install DNS Server

# Debian
user@nameserver $> apt-get update ; apt-get install bind9 dnsutils -y
# RedHat
user@nameserver $> yum update ; yum install bind bind-utils -y

Configuration

Ok now bind is setup, we're going to configure it. In our examples, we will use the domain name "somewhere.net". In Debian, the config files are in /etc/bind and in RedHat the files are in /etc and /etc/named/. As ever, I do advice you to backup it somewhere to roll-back.

We will also find db. files, which are default or examples configuration files. We're going to copy the db.empty do db.<domain_name> db.somewhere.net. In this file, we modify localhost with our values :

; db for somewhere.net
;
$TTL    86400
@            IN    SOA       ns1.somewhere.net. root.somewhere.net. (
                             1         ; Serial
                             604800         ; Refresh
                             86400         ; Retry
                             2419200         ; Expire
                             86400 )       ; Negative Cache TTL
;
; Here we define the nameservers of the domain.
@            IN    NS        ns1.somewhere.net.
@            IN    NS        ns2.somewhere.net.
;
;Here we set the MX records for our domain
@            IN    MX    10  smtp.somewhere.net.
;
; Now we set the IP of the nameservers - Use yours
ns1          IN    A         192.168.1.1
ns2          IN    A         172.16.1.1
;
; Now, we set some zones
www          IN    A         192.168.1.10
smtp         IN    A         192.168.1.2

Here we have also set some little parameters in our db.file. We're now going to use it in bind to be able to resolve it.Open to edit named.conf.local. Here, we're going to add our zone.

zone "somewhere.net"
{
     type master;
     file "/etc/bind/db.somewhere.net";
};

Ok so there we have a very basic domain configuration. But our server is not going to serve this domain until you allowed it. So now let's see the bid configuration. We have a file called named.conf.options - In this file we can find some examples and default config. So here, what we gonna do is to allow clients to query the server, and thing which is really useful, use our DNS as a forwarder (for non-localy hosted domains) :

[...]
forwarders { 8.8.8.8; };
allow-query { any; };
allow-query-cache { localnet; };
allow-recursion { localhost; };
[...]

Here we said to our server to allow queries from anywhere. Maybe you want your server to answer only to your local machines. So to do this, we gonna use the ACLs, which we will see in next part. Common keywords are :

  • any
  • localnet
  • localhost

So now if we restart our server, you will be able to query it from clients.

Manage security groups

On the cloud, everything is acting as a service. That's also true for security. As this is a managed platform, you will also be able to manage firewall rules easily, via API.

Security groups - Definition

A security group is an object which represents a set of rules, incoming as outgoing. By default, any imcoming traffic is denied, and all outgoing is authorized. A security group is also a way to create complete security strategy, and represent each layer.

So here, let see what we can do.

Tip

There is not any specific best practices, but, I really recommend you to be carefull with cross-group security sets

Admin security groups

I never used the default group. So, I created a specific security group to administrate the VMs and have SSH remote access.

[admin@clc-0 ~] euca-create-group local-admin -d "Group to access VMs in Local"
[admin@clc-0 ~] euca-describe-groups
GROUP   sg-9BE43E0C     083277484068    local-admin     Group to access VMs in Local
GROUP   sg-7D063F3D     083277484068    default default group
[admin@clc-0 ~] euca-authorize sg-9BE43E0C -P tcp -p 22 -s a.b.c.d/subnet
GROUP   sg-9BE43E0C
PERMISSION      sg-9BE43E0C     ALLOWS  tcp     22      22      FROM    CIDR    a.b.c.d/subnet
# You now have created a new security group and allow SSH from you subnet.

Applicative security groups

This is the common way to use Security Groups. If you want to separate groups by their applications, you can create several groups and allow only specific groups to access to each others. Like maybe Apache and Database back-ends. So, there we are.

[admin@clc-0 ~] euca-create-group web-servers -d "Web Serves only"
GROUP   sg-EFCA418E     web-servers     Web Serves only
[admin@clc-0 ~] euca-create-group sql-backend -d "SQL Backend"
GROUP   sg-DBAB41E8     sql-backend     SQL Backend
[admin@clc-0 ~] euca-authorize web-servers -P tcp -p 80 -s 0.0.0.0/0
GROUP   web-servers
PERMISSION      web-servers     ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0
[admin@clc-0 ~] euca-authorize sql-backend -P tcp -p 3306 -o sg-EFCA418E
GROUP   sql-backend
PERMISSION      sql-backend     ALLOWS  tcp     3306    3306    GRPID   sg-EFCA418E     FROM    CIDR    0.0.0.0/0

Alias and targets

You may also want to use like "Aliases" to match a group of clients as being one. To do this, you can also create severs/clients security groups. Imagine, you have a master configuration machine, which all you security groups have to access. But it can be long to add each new security group you create to be authorized to access your main's. So, you create the security group of the server and you create the client one. Now, you can assign this client security group to all the VMs which have to access the configuation server, and in the security group of the server, you only have to open access from this security group. Easier to manage, and no overlap. But : do not forget to add it ;)

Generic SAN Adapter

As we all can not buy Eucalyptus with registration, some people contribute to community with packages and addons, like Nathanxu (https://github.com/nathanxu) who is in Eucalyptus team. There, he developed a generic san adapter I've tested on my old CX300 SAN. There is a network map

/galleries/eucalyptus/san-adapter/LAN-SAN.jpg

As you can see, the SC and NCs are connected to the SAN's LUN directly, using scsi via FiberChannel. The LUN appears on NCs and SC as /dev/sda. (Do not forget fdisk to set block alignement).

The main goal is to access the volumes created by eucalyptus and managed by the SC, from NC, directly on the LUN and not from the SC (this allow the SC fails but the NC still have access to volumes, what could not be in das or overlay configuration).

To use this SAN "plugin", you'll need to have CLC setup but not initialized yet, and all other components setup on your different servers. There, you will have to download and compile the SAN .jar file.

yum groupinstall "Development tools" -y
yum install axis axis2c ant java-1.7.0-openjdk-src java-1.7.0-openjdk-javadoc java-1.7.0-openjdk-devel java-1.7.0-openjdk rampartc rampartc-debuginfo libvirt-java-devel libvirt-java libvirt-devel libcurl-devel libcurl libxml2 libxml2-devel bea-stax-api libcap-devel libcap python-pip gcc make cmake python-devel libxml2-python python-lxml apache-ivy ant-nodeps git -y

Now, clone the git repository, get in and build

admin@clc-0 $> git clone https://github.com/nathanxu/gen-san-adapter.git
admin@clc-0 $> cd gen-san-adapter/
admin@clc-0 $> ./build.sh

It will take a little time depending on your server. If you prefer, you can also download tarballs.

Once it's done, on the SC :

admin@sc-0 $> ./install_sc.sh

On each Node Controller :

admin@nc-X $> ./install_nc.sh

Great. Now were close to be ready to do it. What I did too, is to copy the .jar from the SC to CLC in /usr/share/eucalyptus. Now, we can initialize our cloud and register elements (see here). Once you've registered your SC, set the block manager and device target :

admin@clc-0 $> euca-modify-property -p clustername.storage.blockstoragemanager=clvm
admin@clc-0 $> euca-modify-property -p EWS-FR-0.storage.sharedevice=/dev/sda

Now we have to configure the NC lvm config and ISCSI. So, edit lvm.conf (/etc/lvm/lvm.conf)

locking_type = 2
locking_library = "/lib/liblvm2eucalock.so"
volume_list = [ "sda","@*" ] # replace sda by your shared device
# add these lines at the end of your config filetags
{
             hosttags = 1
             @NODE-IP {} # replace by your NC's IP
}

Now, edit /etc/iscsi/initiatorname.iscsi

# Change IP by your NC's one
InitiatorName=iqn.1994-05.com.redhat:Node-IP

Once it's done, you should restart your SC to ensure settings. Now, to test if you can create a new volume.

admin@clc-o $> euca-create-volume -z <zone> --size 10 # for 10Go disk

If the drive come from creating to available, it means that everything is working properly. The EBS could be created. So now, let's do some tests with it :

  • Attach the drive to a VM, format and mount it
  • Unmount it and remount on a different VM
  • Run a bfEBS VM

If all these tests finished correctly, there you have a good "Generic San Adapter"

** Special thanks to Nathan Xu.**

Create a bf EBS with Eucalyptus

There are two types of instances : Instance-Store and EBS-Backend. As you will be able to see here (http://url-vers-explication-ebs-instancestrore) EBS give you the capacity to keep your datas on SAN, to stop / start a VM without loosing datas. But there is no "3 Clicks" way to have one today. So here we are going to see an easy and sure way to create your EBS EMIs.

Requirements

Of course, you will need to have a working Eucalyptus Cloud. It seems obviouis that you have enough compute capacity to run new instances and enough hard drive free space to create EBS and snapshots. Depending on your platform, some steps may be long.

Let's check some cloud-properties and status :

admin@clc-0 $>euca-describe-services -E # here we check that all the Eucalyptus components are running and registred
admin@clc-0 $>euca-describe-properties
admin@clc-0 $>euca-create-volume -s 5 -z cluster # here we test that EBS creation is working properly

The services must be all enabled and let's understand some cloud-properties for our needs :

  • cluster.storage.maxtotalvolumesizeingb : This is the size allocated for all EBS volumes
  • cluster.storage.maxvolumesizeingb : This is the maximum size of 1 EBS volume (default: 15Go)
    • You will have to change this property if you want to create more thant 15 Go EBS volumes
  • cluster.storage.shouldtransfersnapshots : Defines if your snapshots should also be transfered in S3 Walrus. (default: True)
    • This will allow you to "distribute" all your EMIs across regions easily. But This will take more time to create the snapshot.
  • walrus.storagemaxtotalsnapshotsizeingb : This is the maximum size allocated for all your snapshots

Original VM

Most of people want to do an EBS EMI from an Instance-Store one. I am not part of these people. I prefer to fully create an instance from my own. There are ways :

  • You have physical access to the NC, and the capacity to use and external medium (USB CD/DVD Reader - USB Key) to boot on and setup the OS
  • You do not have physical access but you have SSH access to it and your client has graphical X Server (You can also use XMing for Windows).
  • The computer you are working with supports virtualization.

First case

Here, we only have to create a volume, snapshot it and register an EMI. Then run a new instance. Obviously, no OS on the EBS. But thanks to the USB Device, you can add a PCI device on the fly, and declare to boot from it. Then, do your setup. When OS setup finished, you have your VM. Stop it, snapshot the root device and re-create an EMI.

Attention!

You will need to use the init scripts provided by Eucalyptus or create your own to have SSH-Key based access and other functions.

Before creating our VM

I advice you to enable graphical access (via VNC) to the VM created by Eucalyptus. To do so, uncomment vnc section in /etc/eucalyptus/libvirt.xsl :

<graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>

Restart the Node controller service (If you have running VMs on it, this will not stop them)

sudo service eucalyptus-nc restart

Second case

We now need an Instance (an instance-store one). This one will allow us to use our EBS in the future. Will eustore, download one and run it :

admin@clc-0 $>eustore-describe-images
admin@clc-0 $>eustore-install-image --help
admin@clc-0 $>euca-run-instances $EMI -n $number -k $KEY -t $instance-size -g $group

I do advice you to create specific Security Groups depending on your Instance usage.

Use virt-manager

I am a shell guy, mean I do everything in shell windows, but it is also true to say that GUI have a lot of benefits. Here, what we are going to do is to use virt-manager. It is a very powerful tool to manage KVM / XEN on remote and redirect all through SSH. So, here I am on a CentOS client with graphical interface. Be sure to have ssh access to your NC.

Use ssh-keys

For any action made by virt-manager, you will be prompted for your password. To avoid this and guarantee a full secured permanent access, use ssh-keys : generate a ssh-key for your current user, then, sync it for the root user on your NC :

user@client $>ssh-keygen -t rsa -b 4096
user@client $>ssh-copy-id -i $HOME/.ssh/id_rsa.pub root@node

Once it's done, download and setup virt-manager, then, launch it. If you plan to use your own computer (the one you are working with) to create this original VM, do so, otherwise, create a new connection to your NC.

Note

You will need the ISO of the Operating System you plan to install. On a NC, you should put this in /var/lib/libvirt/images with the correct access rights (libvirt:root)

Warning

If you plan to create a Windows EMI, get the KVM drivers and kernels. If you do not provide these to windows, you will not be able to detect virtio drives

Run our new EMI

Here we are ! That's the usual way to install a new operating system. You will be able to assign the values you want to your original VM.

Warning

In /etc/eucalyptus/eucalyptus.conf on your NC, you are using specific VM properties : virtio. Be sure that all devices on the VM you are creating are the same according to this configuration !

Tip

Here you will be able to organize your partitions on your will. There are tens way to do so. Here is the right moment to plan your instance according to its usage.

  • Classical VM ? :
    • Create a /boot, / using most of space and some for the swap
    • Create a /boot, / using most of space and NO swap : you will use a script which on startup will use part of your ephemeral as a swap device
  • Heavy storage VM ?
    • Create a /boot, and use LVM for the sub parts : in the future, if you need more space, add EBS volumes, and extend LVM ;)

Prepare our fresh VM for Eucalyptus

Our setup is finished, and after reboot we are glad to see our system running properly. But, it is a too specific one. So here we are going to "clean" this VM to be the most generic possible. We also are going to setup all our usual packages.

On linux

At startup, you system created some "rules" which come from your hardware configuration. We need to delete this, because all the instances you will create from this EMI will have different properties (i.e. the eth0 MAC address). So begin with deleting these rules :

rm -rfv /etc/udev/rules.d/* # rules usually are in the same directory for most common distros. Specifyt this path according to yours.

Warning

For CentOS or RedHat EMIs, remember to delete the HWADDR property in /etc/sysconfig/network-scripts/ifcfg-eth0 and set BOOTPROTO to dhcp and ONBOOT to yes

We are going to use SSH to access our VMs. So to provide a fully secured system, we are going to delete the host key files which have been generated by sshd. New ones will be generated on start by sshd.

rm -rfv /etc/ssh/ssh_host_*

On Windows

There are tools available on eucalyptus documentation website which will allow your to clean your EMI in addition to a sysprep

At that moment

Here, our VM is able to get connected to the network and you can log in with SSH. There, is a list of tools you must setup to use advanced EC2 scripts :

root@instance $>apt-get install curl bash-completion euca2ools ntp ntpdate python-boto
root@instance $>yum install ntp curl openssh-server openssh-clients rsync euca2ools

In addition to packages setup, I advice you to configure the NTP client to point onto your CLC, and set all your servers to be on UTC timezone.

root@instance $>chkconfig ntpd on
root@instance $>update-rc.d ntp defaults
root@instance $>unlink /etc/localtime && ln -s /usr/share/zoneinfo/UTC /etc/localtime
# /etc/ntp.conf on instance
server ntp.localdomain

Warning

If you do not have all the clocks synced, CLC will deny requests and none of your script calling http://169.254.169.254/ will work

Init scripts

This is no magic when you can log on your VMs with ssh private key. A script put the public key on the server for the user you wanted to be able to log on. So, now, several choices : do your own scripts, or use existing ones. You will be able to find some scripts on https://github.com/eucalyptus/Eucalyptus-Scripts . Once you get the script you want, run it manually to be sure everything is working properly.

Note

On some distros you are forced to set a root password. Reset it to none, and also add sudo rights to your user if you do not want to use the root one by default (usually used: ec2-user)

Final steps

Here our VM is ready. Be sure you comply with all requirements, followed my tips ;) and cleaned it. Your last command on it can also be halt, on shut it down thanks to euca-stop-instances. On the VM is stopped, we are back to our 2 cases :

First case - Second part

Your VM is already on an EBS. So, once it is stopped, just snapshot the volume, and register your new EMI with the snapshot id :

admin@clc-0 $>euca-register -n 'name' -d 'description' -b /dev/sda=snap-ID --root-device-name /dev/sda -a ARCH # short way
admin@clc-0 %>euca-register -n 'name' -d 'description' -b /dev/sda=snap-ID:size:true --root-device-name /dev/sda -a ARCH -b /dev/vdb=ephemeral0 # this specify EMI size and if the volume has to be deleted on termination and add an ephemeral on /dev/vdb

Second case - Second part

The hard drive raw file of your VM is on the Node-Controller (or on your computer) in /var/lib/libvirt/images/ and usually uses .img extension. Get this file. Now, create 2 EBS volumes on eucalyptus. One of the same size as the hard drive of your VM and another a little bigger. Then attach both to your instance-store instance. Connect to the Instance with your ssh-key, and see if the disks were correctly detected :

root@instance $>dmesg
root@instance $>fdisk -l
# Spot the biggest, and format it, to mount it as usual (on /mnt in our case)
root@instance $>mkfs.ext4 -m 0 /dev/vdb
root@instance $>mount /dev/vdb /mnt

Right now, go where your VM.img file is, and rsync it to /mnt on your MV :

root@nc-0$ >rsync -e 'ssh -i $HOME/cloud-user.pem' -avzt --progress --inplace /var/lib/libvirt/images/myfirstebs.img root@vm-ipL/mnt/
# Once it's done, **dd** the image file to the EBS of the correct size
root@instance $>dd if=/mnt/myfirstebs.img of=/dev/vdc bs=1M

Wait the dd to be finished. Then, detach the volume, create a snapshot from it, and register it like in the previous "First case" part.

The end

If your VM is running properly, you can reach it and log onto, you have your EBS backed instance !!! So now if you plan to make new EBS backed with any other tools on it (Web Server or anything else), run a VM from the EMI you've just done, do your stuff, register, snap it and register it (Do not forget to clean it !)

Frequently asked questions

Once you have this EMI, a lot of questions may come. Here I tried to summerise the most frequently asked.

Question

If I do changes on a VM from my EBS EMI, will all the others have the changes too ?

Answer

No. The EBS of your Instance has been copied from the snapshot, but you are not running your VM from the snapshot itsself. To keep any change on your VM for future VMs, clean, stop, snap and register it. Then, all the new instances you will run from this newly created EMI will have your changes.