Building virtual machines with vmbuilder

After installing qemu-kvm, libvirt-bin, bridge-utils and ubuntu-vm-builder, set up a bridge that virtual machines can attach to:

auto lo
iface lo inet loopback

auto enp2s0
iface enp2s0 inet manual

auto br0
iface br0 inet static
      address 192.168.0.8
      netmask 255.255.255.0
      gateway 192.168.0.1
      nameserver 127.0.0.1
      bridge_ports enp2s0
      bridge_stp off
      bridge_fd 0
      bridge_maxwait 0

Then /etc/init.d/networking restart.

Install apt-cacher and give it the following config in /etc/apt-cacher/apt-cacher.conf:

group = www-data
user = www-data
daemon_addr = 192.168.0.8
path_map = ubuntu archive.ubuntu.com/ubuntu
allowed_hosts = 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
ubuntu_release_names = trusty

The vm build script, using the local cache:

#!/bin/sh

if [ $# -lt 2 ]; then
    echo "usage: HOSTNAME IP"
    exit 1
else
    HOSTNAME=$1
    IP=$2
fi

firstboot=`mktemp`
echo "#!/bin/bash" >>$firstboot
echo "rm -f /etc/resolvconf/resolv.conf.d/{original,base,head,tail}" >>$firstboot
echo "reboot" >>$firstboot
chmod +x $firstboot

qemu-img create -f qcow2 -o preallocation=falloc $HOME/vms/$HOSTNAME-rootdisk 4096M

vmbuilder kvm ubuntu \
  --suite trusty \
  --verbose \
  --libvirt qemu:///system \
  --destdir $HOME/vms/$HOSTNAME/ \
  --install-mirror http://192.168.0.8:3142/ubuntu \
  --mirror http://192.168.0.8:3142/ubuntu \
  --raw $HOME/vms/$HOSTNAME-rootdisk \
  --rootsize 4096 \
  --swapsize 0 \
  --mem 128 \
  --cpus 1 \
  --hostname $HOSTNAME \
  --bridge br0 \
  --ip $IP\
  --mask 255.255.255.0 \
  --gw 192.168.0.1 \
  --dns 192.168.0.8 \
  --lang en_US.UTF-8 \
  --timezone UTC \
  --user ubuntu \
  --name Ubuntu \
  --pass ubuntu \
  --ssh-user-key $HOME/.ssh/id_rsa.pub \
  --addpkg linux-image-generic \
  --addpkg openssh-server \
  --addpkg sudo \
  --firstboot $firstboot

rm $firstboot
rmdir $HOME/vms/$HOSTNAME/

virsh autostart $HOSTNAME
virsh start $HOSTNAME

Edit: To inspect and edit the virtual machine disk image, use guestfish, part of the libguestfs project:

$ sudo apt-get install libguestfs-tools
$ guestfish --rw --add testserver-rootdisk

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> run
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
><fs> list-filesystems
/dev/sda1: ext4
><fs> mount /dev/sda1 /
><fs> emacs /etc/resolv.conf
><fs> exit

Edit: Disk image pre-allocation can be done with QEMU-provided tools, which may or may not be a more kosher approach. Must investigate.

qemu-img create -f qcow2 -o preallocation=falloc $HOME/vms/$HOSTNAME-rootdisk 32768M

Edit: Fix nameserver enforcement in build script.

Tagged with:

Categorised as:


Salt Notes

I decided to go for Salt when picking a solution that would help me automate server management. Here are some things that required some figuring out.

Including keys in pillar data

Using Git as an example; deploy key is set in Github repo's settings:

sites:
  example.com:
    gitsource: git+ssh://git@github.com/you/your_repo.git
    gitidentity: |
      -----BEGIN RSA PRIVATE KEY-----
      <Deploy key goes here – mind the indentation!>
      -----END RSA PRIVATE KEY-----
    

Using the above in states:

{% if 'gitsource' in args and 'gitidentity' in args %}
/etc/deploy-keys/{{ site }}:
  file.directory:
    - makedirs: True
    - require:
      - pkg: nginx
    - watch_in:
      - service: nginx

/etc/deploy-keys/{{ site }}/identity:
  file.managed:
    - mode: 600
    - contents_pillar: sites:{{ site }}:gitidentity
    - require:
      - pkg: nginx
    - watch_in:
      - service: nginx

{{ args.gitsource }}:
  git.latest:
    - identity: /etc/deploy-keys/{{ site }}/identity
    - target: /var/www/{{ site }}
    - rev: master
    - force: True
    - require:
      - pkg: nginx
    - watch_in:
      - service: nginx
{% endif %}
    

Swap

Using a swap file here because DigitalOcean instances, at least the small ones that I've tested, don't include any swap.

/swapfile:
  cmd.run:
    - name: "fallocate -l 1024M /swapfile && chmod 600 /swapfile && mkswap /swapfile"
    - unless: test -f /swapfile
  mount.swap:
    - require:
      - cmd: /swapfile
    

Logentries

The "agent" of the excellent Logentries log gathering service doesn't use a config file, and instead relies on the le tool that is used to set thing up. After config changes, the Logentries daemon must be restarted (that last restart part can likely be streamlined but I couldn't get a hard service restart to work otherwise).

logentries:
  pkgrepo.managed:
    - name: deb http://rep.logentries.com/ trusty main
    - dist: trusty
    - file: /etc/apt/sources.list.d/logentries.list
    - keyid: C43C79AD
    - keyserver: pgp.mit.edu
  pkg:
    - latest

logentries_registered:
  cmd.run:
    - unless: le whoami
    - name: le register --force --account-key={{ pillar['logentries']['account_key'] }} --hostname={{ grains.id }} --name={{ grains.id }}-`date +'%Y-%m-%dT%H:%M:%S'`
    - require:
      - pkg: logentries
    - require_in:
      - pkg: logentries-daemon

logentries_follow:
  cmd.run:
    - name: |
        le follow /var/log/syslog
        le follow /var/log/auth.log
        le follow /var/log/salt/minion
{% for site, args in pillar.get('sites', {}).items() %}
        le follow /var/log/nginx/{{ site }}.access.log
        le follow /var/log/nginx/{{ site }}.error.log
{% endfor %}
    - require:
      - pkg: logentries
    - require_in:
      - pkg: logentries-daemon

logentries-daemon:
  pkg:
    - latest

logentries_daemon_stop:
  service.dead:
    - name: logentries
    - require:
      - pkg: logentries-daemon
    - require_in:
      - logentries_daemon_start

logentries_daemon_start:
  service.running:
    - name: logentries
    

Tagged with:

Categorised as:


Basic MRTG config

It's a mess, but making things work took some effort, and a lot of copypasting from numerous sources, so I'm recording this here.

/etc/mrtg.cfg:

WorkDir: /var/www/mrtg
WriteExpires: Yes
Language: english
Title[^]: Traffic Analysis for Yourserver
Options[_]: growright, bits

LoadMIBs: /usr/share/mibs/netsnmp/UCD-SNMP-MIB, /usr/share/mibs/ietf/TCP-MIB, /usr/share/mibs/ietf/UDP-MIB, /usr/share/mibs/ietf/HOST-RESOURCES-MIB

Target[localhost.cpu]:ssCpuRawUser.0&ssCpuRawUser.0:yoursnmpcommunity@localhost+ ssCpuRawSystem.0&ssCpuRawSystem.0:yoursnmpcommunity@localhost+ ssCpuRawNice.0&ssCpuRawNice.0:yoursnmpcommunity@localhost
RouterUptime[localhost.cpu]: yoursnmpcommunity@localhost
MaxBytes[localhost.cpu]: 100
Title[localhost.cpu]:  CPU Load
PageTop[localhost.cpu]: <H1> Active CPU Load %</H1>
Unscaled[localhost.cpu]: ymwd
ShortLegend[localhost.cpu]: %
YLegend[localhost.cpu]: %
Legend1[localhost.cpu]: %
Legend2[localhost.cpu]:
Legend3[localhost.cpu]:
Legend4[localhost.cpu]:
LegendI[localhost.cpu]: Active
LegendO[localhost.cpu]:
Options[localhost.cpu]: growright,nopercent

Target[localhost.mem]: .1.3.6.1.4.1.2021.4.6.0&.1.3.6.1.4.1.2021.4.6.0:yoursnmpcommunity@localhost
PageTop[localhost.mem]: <H1> Free Memory</H1>
Options[localhost.mem]: nopercent,growright,gauge,noo
Title[localhost.mem]:  Free Memory
MaxBytes[localhost.mem]: 1000000
kMG[localhost.mem]: k,M,G,T,P,X
YLegend[localhost.mem]: bytes
ShortLegend[localhost.mem]: bytes
LegendI[localhost.mem]: Free Memory:
LegendO[localhost.mem]:
Legend1[localhost.mem]: Free memory, not including swap, in bytes

Target[localhost.swap]: memAvailSwap.0&memAvailSwap.0:yoursnmpcommunity@localhost
PageTop[localhost.swap]: <H1> Swap Available</H1>
Options[localhost.swap]: growright,gauge,nopercent
Title[localhost.swap]:  Swap Available
MaxBytes[localhost.swap]: 1073741824
kMG[localhost.swap]: k,M,G,T,P,X
YLegend[localhost.swap]: bytes
ShortLegend[localhost.swap]: bytes
LegendI[localhost.swap]: Swap Available:
LegendO[localhost.swap]:
Legend1[localhost.swap]: Swap memory available

Target[localhost.rootdisk]: dskPercent.1&dskPercent.1:yoursnmpcommunity@localhost
PageTop[localhost.rootdisk]: <H1> Used Root Disk</H1>
Title[localhost.rootdisk]:  Used Root Disk
MaxBytes[localhost.rootdisk]: 100
Options[localhost.rootdisk]: nopercent,growright,gauge
Unscaled[localhost.rootdisk]: ymwd
ShortLegend[localhost.rootdisk]: %
YLegend[localhost.rootdisk]: %

Target[localhost.eth0]: /95.85.11.102:yoursnmpcommunity@localhost
MaxBytes[localhost.eth0]: 12500000
Title[localhost.eth0]:  eth0
PageTop[localhost.eth0]: <h1> eth0</h1>
ShortLegend[localhost.eth0]: b/s
YLegend[localhost.eth0]: b/s

Target[localhost.eth1]: /10.129.5.83:yoursnmpcommunity@localhost
MaxBytes[localhost.eth1]: 12500000
Title[localhost.eth1]:  eth1
PageTop[localhost.eth1]: <h1> eth1</h1>
ShortLegend[localhost.eth1]: b/s
YLegend[localhost.eth1]: b/s

Target[localhost.udpin]: udpInDatagrams.0&udpInDatagrams.0:yoursnmpcommunity@localhost
PageTop[localhost.udpin]: <H1> Incoming UDP pkts per minute</H1>
Title[localhost.udpin]:  Incoming UDP pkts per minute
MaxBytes[localhost.udpin]: 1000000
ShortLegend[localhost.udpin]: p/m
YLegend[localhost.udpin]: p/m
LegendI[localhost.udpin]: Incoming
LegendO[localhost.udpin]:
Options[localhost.udpin]: nopercent,growright,perminute

Target[localhost.udpout]: udpOutDatagrams.0&udpOutDatagrams.0:yoursnmpcommunity@localhost
PageTop[localhost.udpout]: <H1> Outgoing UDP pkts per minute</H1>
Title[localhost.udpout]:  Outgoing UDP pkts per minute
MaxBytes[localhost.udpout]: 1000000
ShortLegend[localhost.udpout]: p/m
YLegend[localhost.udpout]: p/m
LegendI[localhost.udpout]:
LegendO[localhost.udpout]: Outgoing
Options[localhost.udpout]: nopercent,growright,perminute

Target[localhost.tcpconns]: tcpCurrEstab.0&tcpCurrEstab.0:yoursnmpcommunity@localhost
Title[localhost.tcpconns]:  TCP Connections
PageTop[localhost.tcpconns]: <H1> TCP Connections</H1>
MaxBytes[localhost.tcpconns]: 10000000000
ShortLegend[localhost.tcpconns]: conns
YLegend[localhost.tcpconns]: conns
LegendI[localhost.tcpconns]: Incoming
LegendO[localhost.tcpconns]: Outgoing
Legend1[localhost.tcpconns]: Established incoming connections
Legend2[localhost.tcpconns]: Established outgoing connections
Options[localhost.tcpconns]: nopercent,gauge,growright

Target[localhost.tcpnewconns]: tcpPassiveOpens.0&tcpActiveOpens.0:yoursnmpcommunity@localhost
Title[localhost.tcpnewconns]:  New TCP Connections
PageTop[localhost.tcpnewconns]: <h1> New TCP Connections / minute</h1>
MaxBytes[localhost.tcpnewconns]: 1000000000
ShortLegend[localhost.tcpnewconns]: conns/min
YLegend[localhost.tcpnewconns]: conns/min
LegendI[localhost.tcpnewconns]: Incoming
LegendO[localhost.tcpnewconns]: Outgoing
Legend1[localhost.tcpnewconns]: New inbound connections
Legend2[localhost.tcpnewconns]: New outbound connections
Options[localhost.tcpnewconns]: growright,nopercent,perminute
    

Tagged with:

Categorised as: