Lesser-known but extremely handy Linux tools

Kristóf Kovács has a fantastic post about some lesser-known Linux tools that can really come in handy in different situations.

If you haven't tried dstat (I hadn't until I saw Kristóf's post), this is a great one to try. You can keep a running tally on various server metrics including load average, network transfer, and disk operations.

Here is some sample output:

----total-cpu-usage---- ---paging-- ---load-avg--- ------memory-usage----- -net/total- ---procs--- --io/total- ---system-- ----tcp-sockets----
usr sys idl wai hiq siq|  in   out | 1m   5m  15m | used  buff  cach  free| recv  send|run blk new| read  writ| int   csw |lis act syn tim clo
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1314B  180B|  0   0   0|   0     0 |  70    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1779B 1004B|  0   0   0|   0     0 |  84    78 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M| 904B  362B|1.0   0 1.0|   0     0 |  75    86 | 13   9   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M|2203B 1559B|  0   0   0|   0     0 | 180   127 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M| 260B  130B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  114B|  0   0   0|   0     0 |  54    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|2271B  872B|  0   0   0|   0     0 |  94    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  54    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1126B 1254B|  0   0   0|   0  24.0 |  80    87 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1030B  130B|  0   0   0|   0     0 |  88    82 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 578B  114B|  0   0   0|   0     0 |  53    64 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1597B  890B|  0   0   0|   0     0 |  85    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 552B  114B|  0   0   0|   0     0 |  63    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1624B 1254B|  0   0   0|   0     0 |  81    75 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 478B  114B|  0   0   0|   0     0 |  67    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 418B  114B|  0   0   0|   0     0 |  59    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1265B  874B|  0   0   0|   0     0 |  82    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 758B  114B|  0   0   0|   0     0 |  60    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1236B 1255B|  0   0   0|   0  4.00 |  93    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  71    70 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 214B  114B|  0   0   0|   0     0 |  55    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1201B  890B|  0   0   0|   0     0 |  80    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 108B  114B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1344B 1254B|  0   0   0|   0  10.0 | 119    85 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 172B  130B|  0   0   0|   0  8.00 |  80    82 | 13   7   0   0   5

Learn more about dstat on Dag Wieërs' site.

Lesser-known but extremely handy Linux tools is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Posted in Blog Posts, command line, Linux, network, performance, Security, sysadmin | Comments Off

Standards-based data encryption in PHP

Let’s say you are a good sysadmin, and you back up your database regularly. It contains all kinds of configration data, some of which is sensitive (like passwords), and stored in plain text. For most modern software, you can generate a one-way hash and just store that rather than the clear-text data. However, there might be times when different components or applications simply won’t support any hashing comparison or other alternatives. In these cases, you might have to pass in some clear text.

Since you are a good sysadmin, you are concerned about storing clear text. The general concern is that any physical storage where this data lives might not be safe 100% of the time (removable media, for instance). If someone obtains a copy of your data, you don’t want any sensitive information to be compromised without at least putting up a fight.

Many encryption methods use a “key” during encryption that acts as the secret sauce to converting the encrypted data back into readable, usable data. It is always important to be very cautious about where this key is stored and who/what has access to it. I’ve seen a few proprietary encryption methods that use hard-coded keys, which makes any compromise catastrophic to multiple data sets.

The following algorithm performs the task of encrypting data using 256-bit AES encryption. It provides a mechanism for you to input a custom key on each instance, eliminating the concern for “one key unlocks all the doors”. As an example of how this can be handy, you might provision one key for each of your individual customers. This way, customer B will never be able to decrypt customer A’s top secret documents.

One interesting/debatable topic is whether or not 2-way encryption algorithms are even necessary. This guy has an interesting blog post with his stance, and I tend to agree with him on most of it.

A debatable item in my implementation is the assemble() and disassemble() methods. A knowledgeable colleague of mine suggested that it is not necessary to do my own padding / prefixing length etc. In PHP’s openssl_encrypt() and openssl_decrypt() functions, it appears that this padding and its removal is handled for you if you simply pass in a string of arbitrary length. I didn’t want to get rid of the code just yet, but opinions (especially excerpts from RFC’s) are always appreciated. I’d be interested to know if perhaps the data passed to openssl_encrypt is already of correct size, it is left alone, or if the padding method will still pad an extra 16 bytes for AES-256. Maybe a follow-up post will come some future day.

<?php

/* vim: set expandtab tabstop=4 shiftwidth=4 softtabstop=4: */

/**
 * StandardDataEncryption - At least it's better than clear text.
 *
 * PHP Version 5.3+
 *
 * This program is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <http://www.gnu.org/licenses/>.
 *
 * @category   Encryption
 * @package    StandardDataEncryption
 * @author     Ryan Uber <ryan@blankbmx.com>
 * @copyright  2012 Ryan Uber <ryan@blankbmx.com>
 * @link       http://www.ryanuber.com/standard-data-encryption-in-php.html
 */

/**
 * Provides a standards-base encryption algorithm for safe storage of
 * sensitive data. This algorithm is not designed to provide over-the-
 * wire security, but rather protecting data saved to non-volatile
 * memory and stored some place where the decryption algorithm (or at
 * least the cipher password) is not present.
 *
 * This algorithm is mostly useful when you want/need to have access
 * to clear-text data (maybe you have to store a clear-text password
 * for use in a script somewhere), but dont want to / can't store
 * without encryption.
 *
 * This algorithm implements the following features:
 *  - Data padding to nearest cipher block size
 *  - Randomly generated initialization vectors
 *  - Settable cipher password
 *  - Hexadecimal data returned for easy storage
 *  - Configurable cipher method
 */
class StandardDataEncryption
{
    /**
     * Name of encryption cipher
     *
     * @const string $cipher_name
     */
    const CIPHER_NAME = 'AES-256-CBC';

    /**
     * Block size of cipher (in bytes)
     *
     * @const integer $cipher_blocksize
     */
    const CIPHER_BLOCKSIZE = 16;

    /**
     * Cipher password
     *
     * @var string $cipher_password
     */
    private static $cipher_password = null;

    /**
     * Cipher Initialization Vector
     *
     * @var integer cipher_iv
     */
    private static $cipher_iv = null;

    /**
     * Debugging mode (enabled=TRUE, disabled=FALSE)
     *
     * @var bool $debug
     */
    private static $debug = FALSE;

    /**
     * Simple constructor to make it possible to encrypt / decrypt string
     * data without making explicit calls to set the password and
     * initialization vector.
     *
     * @param string $cipher_password  The cipher password
     * @param string $cipher_iv  The cipher initialization vector
     * @returns bool
     */
    public function __construct( $cipher_password=null )
    {
        if(!is_null($cipher_password))
        {
            self::set_cipher_password($cipher_password);
        }
    }

    /**
     * Method to enable debug messages. Encryption is generally difficult to
     * troubleshoot if you can't see what is going on behind the scenes, but
     * it is equally annoying to receive debug messages when you don't want
     * them. This allows us to set debug mode on an instance of this class.
     * Non-printable characters will show up as a star (*).
     *
     * @param string $Message  The message to output to the debug channel
     * @returns bool
     */
    private static function debug( $Message )
    {
        if(self::$debug)
        {
            $out='';
            foreach(str_split($Message) as $c)
            {
                $out .= ctype_print($c)?$c:'*';
            }
            print "\n*** DEBUG ***\n[".time()."] ".$out."\n";
        }
    }

    /**
     * Prepare data to be encrypted. This manipulates the data and pads
     * it up to the next full block-size with zeros. It will prepend the
     * length of the actual string data before returning.
     *
     * @param string $Data  Arbitrary string data to encrypt
     * @returns string
     */
    private static function assemble( $Data )
    {
        $padlen = self::CIPHER_BLOCKSIZE - (strlen($Data) % self::CIPHER_BLOCKSIZE);
        $result = str_pad($padlen, self::CIPHER_BLOCKSIZE, 0, STR_PAD_LEFT) . $Data . self::pad($padlen);
        self::debug('Assembled '.mb_strlen($result).'-byte string for encryption: '.$result);
        return $result;
    }

    /**
     * After decrypting data, we are left with a string in "prepared"
     * state, basically the return of the assemble() method. We need to convert
     * this into a usable text value and chop the padding off.
     *
     * @param string $Data  Prepared string from original assemble() method
     * @returns string
     */
    private static function disassemble( $Data )
    {
        self::debug('Assembled '.mb_strlen($Data).'-byte string found in decrypted data: '.$Data);
        $padlen = abs((int)substr($Data, 0, self::CIPHER_BLOCKSIZE));
        $result = substr($Data, self::CIPHER_BLOCKSIZE, (strlen($Data)-self::CIPHER_BLOCKSIZE-$padlen));
        return $result;
    }

    /**
     * Convert hex data into an ASCII string. In PHP >= 5.4.0, this method
     * could be replaced by hex2bin(), but alas in 5.3.x, it is absent.
     *
     * @param string $Data  Hexadecimal data
     * @returns string
     */
    private static function fromhex( $Data )
    {
        self::debug('Decoding from hexadecimal value: '.$Data );
        $result = '';
        for($i=0; $i<strlen($Data)-1; $i+=2 )
        {
            $result .= chr(hexdec($Data[$i].$Data[$i+1]));
        }
        return $result;
    }

    /**
     * Convert an ASCII string to hexadecimal format.
     *
     * @param string $Data  ASCII string to convert
     * @returns string
     */
    private static function tohex( $Data )
    {
        return bin2hex($Data);
    }

    /**
     * Set the initialization vector
     *
     * @param string $IV  The initialization vector
     * @returns bool
     */
    private static function set_cipher_iv( $IV=null )
    {
        if(is_null($IV))
        {
            self::debug('Null initialization vector passed - Auto-generating');
            $IV = openssl_random_pseudo_bytes(self::CIPHER_BLOCKSIZE);
        }
        self::$cipher_iv = $IV;
    }

    /**
     * Generates fixed-length padding with repeating values, implemented per
     * RFC 2315 Section 10.2. Characters will likely be non-printable and will
     * not be displayed in debugging output.
     *
     * @param integer $len  The length of padding to return
     * @returns string
     */
    private static function pad( $len=0 )
    {
        return str_repeat(chr('0x'.str_pad($len, strlen(self::CIPHER_BLOCKSIZE), 0, STR_PAD_LEFT)), $len);
    }

    /**
     * Set the secret cipher password for use in encryption
     *
     * @param string $Password  The cipher password
     * @returns bool
     */
    public function set_cipher_password( $Password )
    {
        self::$cipher_password = $Password;
    }

    /**
     * Set debugging mode. Valid states are TRUE of FALSE (enabled or disabled).
     * While debugging mode is set, you will see extra messages that could be
     */
    public function set_debug( $State=FALSE )
    {
        self::$debug = $State;
    }

    /**
     * Using OpenSSL and the supplied parameters, prepare the arbitrary
     * string and encrypt it, returning a value suitable for safe storage.
     * During encryption, the initialization vector is randomly generated
     * so that two encryptions of the same data do not yield the same hash.
     *
     * @param string $Data  Prepared string data as returned by assemble()
     * @returns mixed
     */
    public function encrypt( $Data )
    {
        self::set_cipher_iv();
        $encrypted = self::tohex(self::$cipher_iv . openssl_encrypt(
            self::assemble( $Data ), self::CIPHER_NAME, self::$cipher_password,
            TRUE, self::$cipher_iv
        ));
        if(!$encrypted)
        {
            self::debug('Failed while encrypting data: '.$Data);
            return false;
        }
        return $encrypted;
    }

    /**
     * Decrypt a string and convert the result from its prepared state to a
     * readable, usable string.
     *
     * @param string $Data  Encrypted data
     * @returns mixed
     */
    public function decrypt( $Data )
    {
        $raw = self::fromhex($Data);
        if((($raw?strlen($raw):1) % self::CIPHER_BLOCKSIZE) != 0)
        {
            self::debug('Provided hexadecimal data was invalid.');
            return false;
        }
        self::set_cipher_iv(mb_strcut($raw, 0, self::CIPHER_BLOCKSIZE));
        $decrypted = self::disassemble(openssl_decrypt(
            mb_strcut($raw, self::CIPHER_BLOCKSIZE), self::CIPHER_NAME,
            self::$cipher_password, TRUE, self::$cipher_iv
        ));
        if(!$decrypted)
        {
            self::debug('Failed while decrypting data: '.$Data);
            return false;
        }
        return $decrypted;
    }
}

/* EOF */
?>

Some example usage is below:

<?php
include 'StandardDataEncryption.php';
$e = new StandardDataEncryption('My Encryption Key Here');
$e->set_debug(TRUE);
$encrypted = $e->encrypt('Hack Me!');
print "\nEncrypted value is: ".$encrypted."\n";
$decrypted = $e->decrypt($encrypted);
print "\nDecrypted value is: ".$decrypted."\n";
?>

And it will yield some output like:

*** DEBUG ***
[1335560630] Null initialization vector passed - Auto-generating

*** DEBUG ***
[1335560630] Assembled 32-byte string for encryption: 0000000000000008Hack Me!********

Encrypted value is: ce5b31f1144c93094ffbb805e5d053b04fd9f27821b7f91475ef47688c7495df02681e498933dee6d997ade66506c363b71682277738f1983899935b0c604aa6

*** DEBUG ***
[1335560630] Decoding from hexadecimal value: ce5b31f1144c93094ffbb805e5d053b04fd9f27821b7f91475ef47688c7495df02681e498933dee6d997ade66506c363b71682277738f1983899935b0c604aa6

*** DEBUG ***
[1335560630] Assembled 32-byte string found in decrypted data: 0000000000000008Hack Me!********

Decrypted value is: Hack Me!
Posted in PHP, Security | Comments Off

Performance and redundancy boost for icanhazip.com

It's been a few years since I started a little project to operate a service to return your IPv4 and IPv6 address. Although there are a bunch of other sites that offer this service as well, I've been amazed by the gradually increasing traffic to icanhazip.com.

Here's a sample of the latest statistics:

  • Hits per day: 1.8 million (about 21 hits/second)
  • Unique IP addresses per day: 25,555
  • Hits per day from IPv6 addresses: 1,069 (a little sad)
  • Bandwidth used per day: ~ 400MB

The site is now running on multiple Cloud Servers at Rackspace behind a load balancer cluster. In addition, the DNS records are hosted with Rackspace's Cloud DNS service.

This should allow the site to reply more quickly and reliably. If you have suggestions for other improvements, let me know!

Performance and redundancy boost for icanhazip.com is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Posted in Blog Posts, networking, web | Comments Off

DIY Home Security Part 1 – Setup A Red5 RTMP Live Stream

The goal of this guide is setup an RTMP live stream from your home or office.  Once the stream is setup, I’m going to show you how to play your stream on a webpage or on your android phone.  It’s surprisingly simple if you have a basic knowledge of linux to install red5.  You can see an example of this work on the bar to the right.  I’ve set up a few features like ‘privacy mode’ and I’ll do my best to cover these in a later guide.  For now, get booted into your CentOS server and get ready to install and compile some packages! Click here to check out the full guide!

Posted in flash, Flash Media Encoder, FME, Guide, install, jwplayer, Linux, oflaDemo, Red5, Server, setup, stream, streaming | Comments Off

Getting a Technical Job at Rackspace

You've probably noticed that the blog has slowed down a bit recently. Part of the slowdown is due to an uptick in work required to get OpenStack Nova and its related software up and running at Rackspace for Cloud Servers and another part of it is a severe case of writer's block. I threw out some questions on Twitter about the topics people would like to see covered in some new posts and a commonly requested topic was employment at Rackspace.

Boromir - One Does Not Simply Get a Job at RackspaceFirst things first, getting a job at Rackspace isn't easy. We don't intentionally make the process difficult. It's just that the work we do is unique and demanding.

We work in a fast-paced, extremely dynamic team-centric environment. While some people in the company work in extremely small teams or sometimes all by themselves, that's pretty few and far between. We look for people who can survive and flourish in this atmosphere and we look for people who can do it all while working as a team. Even with all of this hustle and bustle, we still remember why we're doing it: the pursuit of Fanatical Support for our customers.

Another thing to keep in mind is that there's no true secret for making it through the application process. There's no magic combination of skills or "silver bullet" that will scoot you through. Every candidate is reviewed individually for each position. There have been several times at the end of an interview where we've gotten together and said: "Wow, this candidate is solid, but they're just not right for this position. Let's find the right spot and see if there's a spot open." We look for the right candidate for the right position at the right time.

One of the best ways to get ahead in the screening or interview process is to do a little homework about Rackspace and the products we offer. Much of this is covered in a post I wrote in 2011. You'll go into the interviews with more confidence and it will be much more obvious that you're really interested in the position.

Don't be discouraged if the process takes a little longer than you expected. When I was hired in 2006, I went through two phone pre-screens and then three back-to-back interviews in person. Things have changed a little since then and I've heard of some candidates receiving two to three pre-screens via telephone and then one or two interviews in person. The additional screening and interviews may be due to Rackers trying to find the right fit for a particular applicant. As I said previously, we look for the right fit for each applicant. We may consider you for a different position than you applied for if we feel like your skill set or personality fits that role better.

A very common question is what to wear to a Rackspace interview. It's confusing to know exactly what's expected since we have Rackers in the building wearing everything from suits to flip-flops. This is where you really have to go with your gut. Interviewing for a customer-facing sales position while wearing a hoodie and shorts is probably going to bring a suboptimal result. Keep in mind that there's really nothing negative about overdressing (but keep your tuxedo in the closet, seriously). I wore a shirt and tie for my interviews in 2006 but my tie got caught in the car door and was shredded. After a lot of cursing, I took off the tie and decided to wing it with my dress shirt. Nobody ever said a word about it.

Remember to be flexible during the interviews. You might be asked to draw a solution on a whiteboard or think through a really complicated situation. Roll with it and keep your confidence up. When you don't know something, admit it, but then talk about how you'd research an answer.

There's one last thing to keep in mind and it's really critical. If you're ever asked about how you would solve a problem or how you solved a problem in the past, don't divulge any information which is confidential or proprietary to your current company. Just tell the interviewers that you've solved the solution in the past but you'll need to keep things vague to maintain confidentiality. We will definitely understand and we will encourage you to maintain that confidentiality.

Leave your comments if you have any! I'll be glad to answer any questions you have.

Getting a Technical Job at Rackspace is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Posted in Blog Posts, general advice, interview, rackspace | Comments Off

Mysterious puppet agent hangs, stale puppetdlock

Recently, there has been some strangeness happening with the puppet agent process (puppetd), where during a run, the agent gets stuck and does not finish removing the lock file. This results in a useless puppet agent, until it is restarted. This can become a big problem for many reasons, mainly that nobody wants to have to repeatedly log in to every machine to restart the puppetd service.

This issue is discussed at length in this thread:

https://projects.puppetlabs.com/issues/12185#change-56530

There is more than one way to solve this problem, so below I’ll detail my solution. Keep in mind that this is not intended to be a permanent fix, but merely a way of keeping the machines running until a more elegant solution is implemented.

My solution is a simple cron/script type. Every 5 minutes, I run a script to check the last modified time of the puppetdlock file, and compare it to the current time. If it is older than 5 minutes, I call a puppetd restart. That’s it!

#!/bin/bash
# File Name:     puppetd-hang-fix.sh
# Author:        Ryan Uber <ryan@blankbmx.com>
#
# Description:   This is a *TEMPORARY* fix to the common puppetdlock issue.
#                Running this script checks if the lock file is more than 5
#                minutes old, and if it is, restarts the puppet process.
#
# See Also:      https://projects.puppetlabs.com/issues/12185#change-56530

# Path to the puppetdlock file
FILE=/var/lib/puppet/state/puppetdlock

# Number of minutes since last modification that indicate a stale lock file
STALEMIN=5

# UNIX timestamp of last puppetdlock modification
MODIFIED=$(stat -c %Y ${FILE})

# Current UNIX timestamp
NOW=$(date +%s)

# Check if file is stale or not and restart if it is.
if [ $((${NOW}-${MODIFIED})) -gt $((${STALEMIN}*60)) ]
then
    /sbin/service puppet restart
fi

# EOF

See one of my other posts if you are interested in how I went about pushing out the cron job.

Posted in Bourne-Again Shell (bash), Puppet | Comments Off

Why technical people should blog (but don’t)

I originally wrote this post for the Rackspace Blog but I decided to post it here in case some of my readers might have missed it. Please feel free to leave your comments at the end of the post.


Sometimes people talk to me about posts I've written on my blog, or posts they wish I would write. At some point during the discussion, I'll almost always ask the person why they don't start up their own blog or contribute to someone else's. Very few people actually seem interested when I probe them about writing posts on technical topics.

My mother was always the one who told me (and her students) that everyone has a story. She said that writing could be therapeutic in ways you probably won't consider until you've written something that someone else enjoys. Just as software developers exist to write software for their users, writers exist to write stories for their readers. There's nothing that says technical people can't become excellent writers who inspire others to learn and share their knowledge with others.

The goal of this post is to encourage technical people to enjoy writing, write efficiently and feel comfortable doing it. I'll roll through some of the most common responses I've received about why technical people don't blog about what they know.

I don't think I'm really an expert on anything. I'm not an authority on any topic I can think of.

I'm leading off with this response because it's the most critical to refute. If you don't take away anything else from this post, let it be this: you don't need to be an expert on a topic to write about it.

You can find examples of this by rolling through some of the posts on my blog. I'd consider myself to be an expert on one, maybe two topics, but I've written over 450 posts in the span of just over five years. I certainly didn't write all of those about the one or two topics I know best.

Write about what you know and don't be afraid to do a little research to become an authority on something. A great example of this was my post, entitled "Kerberos for haters." I had almost no expertise in Kerberos. In fact, I couldn't even configure it properly for my RHCA exam! However, I did a ton of research and began to understand how most of the pieces fit together. Many other people were just as confused and I decided to pack all of the knowledge I had about Kerberos into a blog post. Positive and negative feedback rolled in and it was obvious that my post taught some readers, inspired some others and angered a few.

What a great way to lead into the next response:

What if I say something that isn't correct? I'll look like an idiot in front of the whole internet!

Been there, done that. Every writer makes errors and comes up with bad assumptions at least once. Readers will call you out on your mistakes (some do it delicately while others don't) and it's your duty to correct your post or correct the reader. I've written posts with errors, and I've gotten a little lazy on my fact-checking from time to time. As my middle school journalism teacher always reminded me, the most important part of a mistake is what you do to clean it up and learn from it.

In short: you'll make mistakes. As long as you've done your due diligence to minimize them and respond to them promptly, your readers should forgive you.

Speaking of errors:

I'm great at a command prompt but my spelling and grammar are awful. I write terribly.

This is easily fixed. If you're one of those folks who live the do-it-yourself type of lifestyle, pick up a copy of The Elements of Style by Strunk & White. There are free PDF versions online or you can borrow one from your nearest journalist. No matter the situation you're in, this book has details about where punctuation should and shouldn't be, how to structure sentences and paragraphs, and how to properly cite your sources (really vital for research posts).

Hauling around a copy of an ultra-dry reference book may not be your thing. If that's the case, find someone you know who has a knack for writing. You can usually find helpful folks in marketing or corporate communications in most big companies who will take your post and return it covered in red ink ready for corrections (thanks, Garrett!). I've even spotted some folks on Fiverr who will do this for as low as $5.

I'll wrap up with the second most common response:

I don't know who I'm writing for? What if I write about something simple and the really technical folks think I'm a noob? What if I write something crazy complex and it goes over most people's heads?

I've done both of these. Most Linux system administrators worth their salt know how to add and remove iptables rules, and they'd consider it to be pretty trivial work. Would it surprise you to know that out of over 450 posts, my post about deleting a single iptables rule is in the top five most accessed posts per month? I receive just over 11 percent of my monthly hits to this post. People are either learning from it or they can't remember how to delete the rule and they want to use the post as a quick reference. Either way, the post is valuable to many people even if I think it's the simplest topic possible.

On the flip side, I went nuts and wrote up a complete how-to for a redundant cloud hosting configuration complete with LVS, glusterfs, MySQL on DRBD, memcached, haproxy and ldirectord. I thought it would be valuable knowledge to a few folks but that it might sail over the heads of most of my readers. Again, I was wrong. The post is constantly in the top 10 most visited posts on the blog and I've probably received more feedback via comments, email and IRC about that post than any other. Once again, a post I thought would be mostly useless turned into a real conversation starter.

Let's conclude and wrap up. Keep these things in mind if you feel discouraged about writing:

  • Write about what interests you whether you're an expert on it or not
  • Don't be afraid to fail
  • Be responsive to your readers
  • Even if you think nobody will read your post, write it
  • Always ensure your voice shines through in your writing — this is what makes it special and appealing

Why technical people should blog (but don't) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Posted in blog, Blog Posts, fedora, general advice, red hat, rhca, sysadmin, writing | Comments Off

mysql-json-bridge: a simple JSON API for MySQL

My quest to get better at Python led me to create a new project on GitHub. It's called mysql-json-bridge and it's ready for you to use.

Why do we need a JSON API for MySQL?
The real need sprang from a situation I was facing daily at Rackspace. We have a lot of production and pre-production environments which are in flux but we need a way to query data from various MySQL servers for multiple purposes. Some folks need data in ruby or python scripts while others need to drag in data with .NET and Java. Wrestling with the various adapters and all of the user privileges on disparate database servers behind different firewalls on different networks was less than enjoyable.

That's where this bridge comes in.

The bridge essentially gives anyone the ability to talk to multiple database servers across different environments by talking to a single endpoint with easily configurable security and encryption. As long as the remote user can make an HTTP POST and parse some JSON, they can query data from multiple MySQL endpoints.

How does it work?
It all starts with a simple HTTP POST. I've become a big fan of the Python requests module. If you're using it, this is all you need to submit a query:

import requests
payload = {'sql': 'SELECT * FROM some_tables WHERE some_column=some_value'}
url = "http://localhost:5000/my_environment/my_database"
r = requests.post(url, data=payload)
print r.text

The bridge takes your query and feeds it into the corresponding MySQL server. When the results come back, they're converted to JSON and returned via the same HTTP connection.

What technology does it use?
Flask does the heavy lifting for the HTTP requests and Facebook's Tornado database class wraps the MySQLdb module in something a little more user friendly. Other than those modules, PyYAML and requests are the only other modules not provided by the standard Python libraries.

Is it fast?
Yes. I haven't done any detailed benchmarks on it yet, but the overhead is quite low even with a lot of concurrency. The biggest slowdowns come from network latency between you and the bridge or between the bridge and the database server. Keep in mind that gigantic result sets will take a longer time to transfer across the network and get transformed into JSON.

I found a bug. I have an idea for an improvement. You're terrible at Python.
All feedback (and every pull request) is welcome. I'm still getting the hang of Python (hey, I've only been writing in it seriously for a few weeks!) and I'm always eager to learn a new or better way to accomplish something. Feel free to create an issue in GitHub or submit a pull request with a patch.

mysql-json-bridge: a simple JSON API for MySQL is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Posted in Blog Posts, development, github, json, mysql, python, rackspace, web | Comments Off

Compare commits between two git branches

I found myself stuck in a particularly nasty situation a few weeks ago where I had two git branches with some commits that were mixed up. Some commits destined for a branch called development ended up in master. To make matters worse, development was rebased on top of master and the history was obviously mangled.

My goal was to find out which commits existed in development but didn't exist anywhere in master. From there, I needed to find out which commits existed in master that didn't exist in development. That would give me all of the commits that needed to be in the development branch.

I constructed this awful looking bash mess to figure out which commits were in development but not in master:

I had a list of commits that existed in development but not in master:

965cf71 Trollface
acda854 Some patch 2
bf1f3e2 Some patch 1
db1980c Packaging

From there, I could swap MASTER and DEV to figure out which commits existed in master but not in development. Only a couple of commits showed up and these were the ones which were committed and pushed to master inadvertently. After a couple of careful cherry picks and reversions, my branches were back to normal.

Compare commits between two git branches is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Posted in Bash, Blog Posts, command line, development, git, github, Shell | Comments Off

New Fedora and EPEL package: httpry

A fellow Racker showed me httpry about five years ago and I've had in my toolbox as a handy way to watch HTTP traffic. I'd used some crazy tcpdump arguments and some bash one-liners to pull out the information I needed but I never could get the live look that I really wanted.

Here's an example of what httpry's output looks like on a busy site like icanhazip.com:

2012-03-13 23:29:39 186.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	186.x.x.x < -	-	-	HTTP/1.1	200	OK
2012-03-13 23:29:39 187.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.0	-	-
2012-03-13 23:29:39 192.x.x.x	187.x.x.x < -	-	-	HTTP/1.0	200	OK
2012-03-13 23:29:39 188.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	188.x.x.x < -	-	-	HTTP/1.1	200	OK
2012-03-13 23:29:39 189.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	189.x.x.x < -	-	-	HTTP/1.1	200	OK

You can watch the requests come in and the responses go out in real time. It even allows for BPF-style packet filters which allow you to narrow down the source and/or destination IP addresses and ports you want to watch. You can run it as a foreground process or as a daemon depending on your needs.

It's now available as a RPM package for Fedora 15, 16, 17 (and rawhide) as well as EPEL 6 (for RHEL/CentOS/SL 6).

New Fedora and EPEL package: httpry is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Posted in Blog Posts, centos, command line, development, fedora, Linux, red hat, rpm, scientific linux | Comments Off