Updating Linux tzdata for DST changes

So, you are a sysadmin living/managing servers in Egypt or in Egypt’s timezone. Or even a good faithful linux user. The government, in it’s infinite wisdom, decided that we should go back to DST. Are you sure you are ready for this?

index

Since I do run servers a lot of servers in the Africa/Cairo timezone, mostly Ubuntu LTS and Debian servers, I looked to see if there is an update for the tzdata package in Ubuntu that would include this, but couldn’t find any (bug report ?).

Although it’s not the best way to do this, I decided to create the timezone datafile myself. IANA is responsible for providing the datafiles. I downloaded the datafiles package, untared it, and checked the the africa file I was happy to see this:

Rule Egypt 2014 only - May 15 24:00 1:00 S

So, the file is up to date. No, using instructions from this awesome debian wiki page, I did the following to compile and test the datafile:

# cd /tmp/
# mkdir tzdata
# cd tzdata
# wget http://www.iana.org/time-zones/repository/releases/tzdata2014c.tar.gz
# tar zxvf tzdata2014c.tar.gz
# zic -d . africa
# TZ="/tmp/tzdata/Africa/Cairo" date --date="2014-05-15 01:00:00 UTC"
    Thu May 15 03:00:00 EET 2014
# TZ="/tmp/tzdata/Africa/Cairo" date --date="2014-05-16 01:00:00 UTC"
    Fri May 16 04:00:00 EEST 2014

As you can see, the time representation will change according the the timezone file. Now, all you need to do, is to copy it in place:

# cp /tmp/tzdata/Africa/Cairo /usr/share/zoneinfo/posix/Egypt
# cp /tmp/tzdata/Africa/Cairo /etc/localtime

Remember that you might need to do something similar if you have any Java applications, Java timezone files are provided by the package tzdata-java.

Now, to apply this on all of our servers, we used Ansible. A simple playbook should do the trick:

---
- hosts: all
  sudo: yes
  tasks:
    - name: Update timezone file
      copy: src=/path/to/data/file dest={{ item }}
      with_items:
        - /usr/share/zoneinfo/posix/Egypt
        - /etc/localtime
      when: ansible_os_family == "Debian"

This should be it! However, this is not meant to be a permanent solution. You should update your tzdata package as soon as the next update is released.

UPDATE: I just found out that the Ubuntu had released a critical update to fix this problem. You don’t need to perform these steps now, just make sure to get the latest tzdata package. Not sure about debian, yet.

Better handling of public SSH keys using Ansible..

Ansible has a dedicated module to manage public keys; the authorized_key module. It’s a very nice module, with enough flexibility to do almost anything I can think of.

However, it does have one very annoying thing. While I was migrating our automation scripts to ansible; I got to the point where I was working on the script that provisions our users. By default, we disabled all password authentication and root SSH access. Only key based access is allowed.

I found that I have to actually put the public SSH key strings inside the playbook vars. That’s just not cool. SSH keys are long, they might have specific options (although the authorized_key module allows you to configure that) and it’s harder to maintain the list of keys like this. So, I tried to work around this. My target was to add the public SSH keys for my users as static files in an ansible role. Basically, I will be populating my my group_vars files by reading files inside my roles.

  • First, I added the public key files in the ‘files‘ directory of the role I was using to configure the users.
  • Now, I have to find a way to “read” the key files and set them in the vars file. Fortunately, ansible provides Lookup plugins that allows me to do just that!
  • So, the related part of the vars file should look like this:


ssh_users:
  - name: user1
    key: "{{ lookup('file', 'user1.pub') }}"
  - name: user2
    key: "{{ lookup('file', 'user2.pub') }}"
  - name: user3
    key: "{{ lookup('file', 'user3.pub') }}"
  - name: user4
    key: "{{ lookup('file', 'user4.pub') }}"

  • Next, all we need to do is call the authorized_key module as usual

- name: Add ssh user keys
  authorized_key: user={{ item.name }} key="{{ item.key }}"
  with_items: ssh_users

Edit: Updated the variable name to avoid the deprecated syntax. Details in the first comment.

Here you go. Key files are neatly tucked in the files directory, easy to maintain and no wrapped lines and cluttered options missing up your var files.

Loops with lineinfile ansible module

I just started playing with ansible recently. And I love it!

I am working on a playbook for configuring Apache 2.4 for a complex application. The plan to run the application on an IaaS cloud(ish) platform. We need to control the Apache worker settings via the playbook since there will be several “flavors” of cloud instances with different sizes and configurations. I was considering using a template for the configuration file. But since I am playing..Screen-Shot-2013-03-30-at-6.07.36-PM

I decided to take a shot at using the lineinfile module, which I find really cool! And to make this a bit more interesting, I wanted to this this using ansible loops, not one configuration item at a time.

So, basically, this is the first shot at getting this done, there is a lot of room for improvement:

- name: Setting Apache Prefork MPM
  lineinfile:
    dest=/etc/apache2/conf.d/mpm_prefork.conf
    regexp="{{ item.key }}"
    line="{{ item.value }}"
with_items:
  - { key: "StartServers" , value: "StartServers {{ StartServers }}" }
  - { key: "MinSpareServers", value: "MinSpareServers {{ MinSpareServers }}" }
  - { key: "MaxSpareServers", value: "MaxSpareServers {{ MaxSpareServers }}" }
  - { key: "MaxClients", value: "MaxClients {{ MaxClients }}" }
  - { key: "MaxRequestsPerChild", value: "MaxRequestsPerChild {{ MaxRequestsPerChild }}"

Additionally, I added the values in the var file:

var:
  StartServers: 10
  MinSpareServers: 5
  MaxSpareServers: 10
  MaxClients: 300
  MaxRequestsPerChild: 0

This should do the trick!

The Mysteriously Slow sudo..

So, I was recently asked to check on an EC2 instance that started spitting Nagios plugin errors for no apparent reason for a few days.

Basically, almost all NRPE checks would time out randomly. There is no load on the server, no disk IO that would cause something similar. Also, several commands were pretty slow on the command line. However, the most notable ones were commands run with  sudo. Especially since the same commands when ran as root mostly worked fine.

Initially I tried to check dmesg for any file system (or disk ?) issues there. I found none of that. However, I did fine several traces of OOM kills. I checked and turned out that the application running on that instance had eaten all the memory and crashed a few days ago.

I tried to check the system logs for errors, and found out that all logs had their last around when the Nagios problem started, and that was also when the application on the instance crashed.

So, rsyslog was dead. There was a pid file and everything, but no process running. I went back to dmesg and found that it got waked by the OOM killer. By now I had a pretty good idea of what was the problem.

The theory is, most applications write to /dev/log (a UNIX socket) to send syslog messages to rsyslog. If rsyslog is dead, no one would read from that socket buffer and it will be filled pretty quickly. Once that happens, any process trying to that socket will have to block until the free buffer is freed or times out.

sudo was particularly sensitive to this because PAM write to the auth.log any time sudo is used. When rsyslog was dead, sudo had to wait till the log write attempt times out.

Simply restarting rsyslogd fixed the problem and everything was back to normal. I took a note-to-self that standard services such as rsyslogd should be monitored on any systems under our management, to avoid such situations.

Show all grants in MySQL

Recently I needed to migrate a database server to a new machine. The databases were reloaded on the new machine from backup. However, only the production databases were in backup.

I needed to copy all the privileges and users to the new database server quickly. This was a busy server with many users and hosts using it.

The following one-liner did the trick:

mysql -N -u root -p$passwd -e "select user, host from mysql.user" | (while read user host; do echo "$(mysql -N -u root -p$passwd -e "show grants for '$user'@'$host'")" ";"; done ) > privileges.sql


Once you did this on the original server, you can feed the privileges.sql to MySQL on the new server:

cat privileges.sql | mysql -u root -p

 

 

Address reuse in python’s SocketServer

Hello, World! This is my first technical post, I hope it’s useful to someone out there!

I am working on a very small tool that I need to for a proof of concept. It’s basically a small TCP server in python.
After creating a small skeleton using SocketServer, I found that the server it self works fine with no problem.

However, if I try to stop and start the server again to test any modifications, I get a random “socket.error: [Errno 98] Address already in use” error. This happens only if a client has already connected to the server.

Checking with netstat and ps, I found that although the process it self is no longer running, the socket is still listening on the port with status “TIME_WAIT”. Basically the OS  waits for a while to make sure this connection has no remaining packets on the way.

My good friend mux mentioned that I should probably set the socket option “SO_REUSEADDR” to avoid this issue.

The man socket(7) says about this:

SO_REUSEADDR

Indicates that the rules used in validating addresses supplied in a bind(2) call should allow reuse of local addresses.  For AF_INET sockets this means that a socket may bind, except when there is an active listening socket bound to the address.  When the listening socket is bound to INADDR_ANY with a specific port then it is not possible to bind to this port for any local address.  Argument is an integer boolean flag.

When using the pure socket module, you can simply set this option using:

import socket

s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

 

However, you don’t need to this in SocketServer. SocketServer.TCPServer allows you to set by setting a simple variable. Here is a working example:

#! /usr/bin/env python
import socket
import SocketServer

class MyServer(SocketServer.StreamRequestHandler):

def handle(self):

self.wfile.write("Hi! I am echo!\n")

while True:

data = self.rfile.readline().strip()
if not data: break
self.wfile.write(">>%s\n" % data)

if __name__ == "__main__":

host, port = "localhost", 10000
# Setting allow_resue_address to True.
SocketServer.TCPServer.allow_reuse_address = True
server = SocketServer.TCPServer((host, port), MyServer)
server.serve_forever()

 

This is just a simple example of how to use it. You can check the last link in the resources for a more complex example with threading support.

Resources:

http://linux.die.net/man/7/socket
http://docs.python.org/library/socket.html
http://docs.python.org/library/socketserver.html
http://code.activestate.com/lists/python-list/222584/
http://www.technovelty.org/code/python/socketserver.html