Desktop cluster (The ganeti deployment)

So this will be a more short “walkthrough” of setting up a Ganeti cluster. And a mention of the pitfalls, will assume you are using standard hardware. If the hardware is conformed and everything is identical it’s great. But i will assume you arent having boxes with dual Ethernet cards and such. Building on chepast shit you could buy to get it up. Its the Desktop version of vmWare. Literally standing on the desktop.

This setup is intended for developers that need a more robust way of testing. That is not in the “cloooud”.. sure clouds is cool, but sucks sometimes or most of the times. And all the pretty deployment services is well pretty but mostly sucks as well..  So this is for those that needs the push down the rabbit hole and take control over their development environment. That said. this will not be crazy detailed.. I will assume you know linux and howto read INSTALL instructions.

So lets begin…

Step 1, aquire hardware. I use a setup of three Mini-ITX computer drawing 150W. The hardware config is Core i5, 16gb ram and 240Gb SSD disk. – Yes you can get more disk its your choice.

Step 2, Install Debian 7.4 on these bozes. Partion the drives accordingly

  • /boot = 200mb
  • / = 30gb
  • swap = 8gb
  • Leave the rest of the space partioned, not mounted, AND NOT FORMATTED!

Step 3, install it as a clean box. The only thing you want to get in during the initial install is SSH. Yes, only SSH. Nothing else. You will need a network connection at this stage.

Step 4, lets reconfigure your network. We will not use a VLAN client for but it will need to be installed later if you need. Lets install some stuff: apt-get install bridge-utils iproute iputils-arping make

Start by setting up a Bridge. This needs to be called xen-br0 (don’t argue it will save you time later).
config should look something like this. This would be the same for all your nodes. I lock the MAC address to a specific IP in my DHCP solution. You could set it to a static address if you wanted.

auto lo xen-br0
iface lo inet loopback
# The primary network interface
allow-hotplug eth0
allow-hotplug xen-br0
iface xen-br0 inet dhcp
bridge_ports eth0
bridge_stp off
bridge_fd 0

And reconfigure your /etc/sysctl.conf with this


Update it with: sysctl -p

Step 5, lets setup your LVM. Start by installing lvm (apt-get install lvm2) – i assume you have only one partion now but you could freely add disks, drives to this as you wanted.

  • This creates the physical volume: pvcreate /dev/sda7 (in my case)
  • Now create a volume group, this needs to be called xenvg: vgcreate xenvg /dev/sda7 (in my case)
  • Here is the trick – DON’T mount this, DON’T create a logical volume on this drive. Because Ganeti uses this volume group and creates its plain files in this logical volume automaticly. And if you use DRBD you will have distribution of these logical volumes across your network RAID1 – We will not use DRBD in this setup.

Step 6, lets make Ganeti shut the f**k up about DRBD. Start by installing DRBD (apt-get install drbd8-utils). Then configure it for Ganeti by issuing:
echo drbd minor_count=128 usermode_helper=/bin/true >> /etc/modules
depmod -a
modprobedrbd minor_count=128 usermode_helper=/bin/true

Yes boys and girls this is taken STRAIGT out of the common installation guide. ( You really should read it. This walkthrough is a crash course.

Step 6, now restart everything, nodes and all. Your boxes should have persistent bridged network and LVM drives.

Step 7, lets compile Ganeti. And this will create some headache.. i can only guarantee that. Not your success. First off you need to get the source.
Lets create a folder (mkdir /home/vendors) then you go ahead and find the latest ganeti and wget that shit in there. You find ganeti code here ( check the download section grab the latest stable version you can find.

Now, untar that file in place, you will end up with a ganeti-<version> something folder. Make your way into the folder and find the INSTALL file. Remember now, this is gonna have to be done on ALL your three nodes. I access all my nodes at once with putty connection manager and type into them simultaneously. You could make one and dd copy that first drive to the other drives as well. But this is just a reminder, this is gonna be done on THREE computers in this case.. or 2.. or 1.. whatever floats your boat..

Anyway, back.. In the folder you will find the file INSTALL.. Read that.. seriously don’t be a retard.. Every version will have different requirements. When writing this its 2.10.1 – that needed some shitload of Haskell libraries and shit.. You know what, the apt-get install commands is IN THE FILE…
So start by doing: apt-get install build-essential – this install the basic shit you would need.
Then go ahead: apt-get install m4 ndisc6 python python-openssl openssl python-pyparsing python-simplejson python-bitarray python-pyinotify python-pycurl python-ipaddr socat fping debootstrap dump kpartx qemu-utils gawk drbd-utils qemu-kvm

More shit is needed: apt-get install ghc libghc-json-dev libghc-network-dev libghc-parallel-dev libghc-deepseq-dev libghc-utf8-string-dev libghc-curl-dev libghc-hslogger-dev

And for good measure: apt-get install libghc-crypto-dev libghc-text-dev libghc-hinotify-dev libghc-regex-pcre-dev libpcre3-dev libghc-attoparsec-dev libghc-vector-dev libghc-snap-server-dev

Hopefully you have all the shit in place now and you can issue: ./configure && make && make install
Then you need to create some folders: mkdir -p /srv/ganeti/{os|export}
And make sure these exists : /etc/ganeti && /var/lib/ganeti && /var/log/ganeti

Step 8, hopefully you are not a total f**kup and managed to not f**k this up to badly. You will need to install some init.d scripts. Check out your /home/vendors/<ganeti version>/doc/examples/ and look for the file ganeti.initd – copy that t /etc/init.d/ganeti, chmod +x the file. And now update your rc.d by: update-rc.d ganeti defaults 20 80.
Then go ahead and look for the ganeti.cron file and copy it to /etc/cron.d/ganeti
Now restart your ganeti service; service ganeti restart

Step 9, You should now have 3 nodes, with ganeti running as single “localized” nodes without a cluster. You will need to create some changes in the /etc/hosts file.
For example your nodes need to know where the other ones are. So lets put in your nodes with IP numbers. You know howto do this. But.. You also need to add a unique IP for the cluster.
Welcome pitfall, add some thing like clust1 to your hosts file, so it has a unique ip number. This ip number will be attached to your bridge automaticly by ganeti during deployment.

Step 10, lets create your cluster by issuing: gnt-cluster –enabled-hypervisors=kvm -H kvm:kernel_path=” init clust1 – If you dont enable KVM it will assume XEN, and we dont have XEN. (PITFALL!)

Step 11, lets add your nodes to the cluster, assuming you initated the cluster on node1, this node will be attached automagicly to clust 1, now issue: gnt-node add node2 && gnt-node add node3 – where node2, node3 is in the hosts file as the name of the server. Noo you dont need a full domain. Really this is for your desktop. Lets check your nodes by issuing: gnt-node list – if everything went ok, and you havent put a partition on the LVM you will see 3 nodes with 188gb something space left on em. wooho.. Great.

Step 12, lets create your first instance. First make a shitload of vm1 through your accessible IP’s in your hosts file, or if you have a DNS server whatever. Commands:
gnt-instance add -t plain -s 8G -B memory=512M -o image+default -n node1 –no-start –no-install -H kvm:vnc_bind_address= vm1 – this creates a 512mb ram, 8gb large KVM instance that will not start right away. So now its time to make sure you have some kind of vnc client lying around.
Make sure somestuff is removed: gnt-instance modify -H serial_console=false vm1 && gnt-instance modify -H kernel_path=” vm1
We wanna install this machine from a ISO, so lets download your ISO to the vendors folder. I just copied the debian i used for the base system and use that. This is how you start the instance.
gnt-instance start -d -H boot_order=cdrom,cdrom_image_path=/home/vendors/debian-7.4.0-amd64-netinst.iso vm1
This will start your instance and it will sit in the “start menu” of any debian installer until you connect to it. So lets find out where that vnc connection is: gnt-instance info vm1 | grep vnc
you will see the connection information and can now connect to it with TightVNC or similar from your desktop – windows, mac whatever.

Step 13, Install your first instance, restart it check network configuration and all that shiet.. When done.. Format your three nodes, and DO THIS AGAIN..
I wrote this cuz i couldnt find any straight up and down “do this shiet” to get running on google. I tend to have a handson approach and figure out the other stuff when i have the main stuff in place.

Over and out. Yes this was it.. If you wanna know more details, google it.. My first attempt of this was done using: – this tutorial isnt really up to date but give you alot of information. The VLAN part in this tutorial fucked up everyting in my case, i have only one ethernet port, and dont need DRBD so.. just walk it off.. You will find rapid deployment information here tho.. Its a good read anyway.

You should read: