Nerderati

You're probably not nerdy enough.

Simplify Your Life With an SSH Config File

If you’re anything like me, you probably log in and out of a half dozen remote servers (or these days, local virtual machines) on a daily basis. And if you’re even more like me, you have trouble remembering all of the various usernames, remote addresses and command line options for things like specifying a non-standard connection port or forwarding local ports to the remote machine.

Shell Aliases

Let’s say that you have a remote server named dev.example.com, which has not been set up with public/private keys for password-less logins. The username to the remote account is fooey, and to reduce the number of scripted login attempts, you’ve decided to change the default SSH port to 2200 from the normal default of 22. This means that a typical command would look like:

$ ssh fooey@dev.example.com -p 22000
password: *************

Not too bad.

We can make things simpler and more secure by using a public/private key pair; I highly recommend using ssh-copy-id for moving your public keys around. It will save you quite a few folder/file permission headaches.

$ ssh fooey@dev.example.com -p 22000
# Assuming your keys are properly setup...

Now this doesn’t seem all that bad. To cut down on the verbosity you could create a simple alias in your shell as well:

$ alias dev='ssh fooey@dev.example.com -p 22000'
$ # To connect:
$ dev

This works surprisingly well: Every new server you need to connect to, just add an alias to your .bashrc (or .zshrc if you hang with the cool kids), and voilà.

~/.ssh/config

However, there’s a much more elegant and flexible solution to this problem. Enter the SSH config file:

# contents of $HOME/.ssh/config
Host dev
    HostName dev.example.com
    Port 22000
    User fooey

This means that I can simply  $ ssh dev, and the options will be read from the configuration file. Easy peasy. Let’s see what else we can do with just a few simple configuration directives.

Personally, I use quite a few public/private keypairs for the various servers and services that I use, to ensure that in the event of having one of my keys compromised the dammage is as restricted as possible. For example, I have a key that I use uniquely for my github account. Let’s set it up so that that particular private key is used for all my github-related operations:

Host dev
    HostName dev.example.com
    Port 22000
    User fooey

Host github.com
    IdentityFile ~/.ssh/github.key

The use of IdentityFile allows me to specify exactly which private key I wish to use for authentification with the given host. You can, of course, simply specify this as a command line option for “normal” connections:

 $ ssh -i ~/.ssh/blah.key username@host.com

but the use of a config file with IdentityFile is pretty much your only option if you want to specify which identity to use for any git commands. This also opens up the very interesting concept of further segmenting your github keys on something like a per-project or per-organization basis:

Host github-project1
    User git
    HostName github.com
    IdentityFile ~/.ssh/github.project1.key

Host github-org
    User git
    HostName github.com
    IdentityFile ~/.ssh/github.org.key

Host github.com
    User git
    IdentityFile ~/.ssh/github.key

Which means that if I want to clone a repository using my organization credentials, I would use the following:

$ git clone git@github-org:orgname/some_repository.git

Going further

As any security-conscious developer would do, I set up firewalls on all of my servers and make them as restrictive as possible; in many cases, this means that the only ports that I leave open are 80/443 (for webservers), and port 22 for SSH (or whatever I might have remapped it to for obfuscation purposes). On the surface, this seems to prevent me from using things like a desktop MySQL GUI client, which expect port 3306 to be open and accessible on the remote server in question. The informed reader will note, however, that a simple local port forward can save you:

$ ssh -f -N -L 9906:127.0.0.1:3306 coolio@database.example.com
# -f puts ssh in background
# -N makes it not execute a remote command

This will forward all local port 9906 traffic to port 3306 on the remote dev.example.com server, letting me point my desktop GUI to localhost (127.0.0.1:9906) and have it behave exactly as if I had exposed port 3306 on the remote server and connected directly to it.

Now I don’t know about you, but remembering that sequence of flags and options for SSH can be a complete pain. Luckily, our config file can help alleviate that:

Host tunnel
    HostName database.example.com
    IdentityFile ~/.ssh/coolio.example.key
    LocalForward 9906 127.0.0.1:3306
    User coolio

Which means I can simply do:

$ ssh -f -N tunnel

And my local port forwarding will be enabled using all of the configuration directives I set up for the tunnel host. Slick.

Homework

There are quite a few configuration options that you can specify in ~/.ssh/config, and I highly suggest consulting the online documentation or the ssh_config man page. Some interesting/useful things that you can do include: change the default number of connection attempts, specify local environment variables to be passed to the remote server upon connection, and even the use of * and ? wildcards for matching hosts.

I hope that some of this is useful to a few of you. Leave a note in the comments if you have any cool tricks for the SSH config file; I’m always on the lookout for fun hacks.

Ideas of March

Four years ago when I started using a micro-blogging service, I revelled in the sheer simplicity and low barrier to communicating my thoughts. Blogging, in that era, seemed like a historical vestige on the verge of being consumed and overtaken by the rapid fire, real-time tweets of my peers. I jumped on that bandwagon, and never looked back. Until now.

Like my friend Chris (and many others), I am reminded of the time when we used to have productive and lengthy discussions in and around blog posts. Moreover, I am reminded of the fact that when I truly need to find some snippet of information or tutorial on a particular web-related topic, the solution is rarely found in multiples of one hundred forty characters.

Now, I’m not ditching Twitter. Micro-blogging does serve a very important role (and I was reminded of this during the recent tsunami/earthquake in Japan) in the effective transmission of near real-time information, but I miss the asynchronous nature of blogs and the less ephemeral nature of the community that surrounds them. It’s about time that I do something to remedy that.

For the rest of the month of March, I will be posting at least two articles per week on various different tech-related topics, with the intent on starting a habit of regular, informative blogging, and I hope that you will join me in this endeavour. I hope and believe that the long term rewards will far outweigh the short term investment in time and effort.

Let’s try and move the conversations we have from being limited by the bits and bytes of tweets to sentences and paragraphs.

Code Your Art Out: If Non-Profits and Tech Had Babies

Ever since I had the pleasure of keynoting at Make Web Not War: For The Web in 2010, I’ve been eagerly awaiting the announcement of the 2011 edition, which was announced just a few days ago.

The title of this year’s competition is Code Your Art Out, and the theme is that of “helping non-profits harness the power of technology in order to better serve its communities and members”. Oddly enough, this is a topic that I have discussed many times over the years with @katiestardust, and is thus very close to my heart.

The competition itself is rather simple: You register, think of an application that might be useful for non-profits (there’s a list of ideas available if you’re stuck), and the architect/code/design/innovate to your heart’s content. As an added incentive, there’s a non-negligible chunk of cash given out to the teams that place in first and second. The only technological requirements are that you need to use either the Windows Azure clould platform, or deploy on Windows IIS as your webserver.

While there’s no requirement that the applications developed be made available under an open source license, I’m hopeful that many submissions will choose to go this route.

The use of modern web technology stacks and applications in most non-profits is woefully lacking, and the space is ripe for innovation. I’m very excited to see what applications people develop!

A few resources:

Twitter account
Facebook Event page
Facebook Page

Perpetual IRC – The Proxy Edition

Last time, we looked at how one could maintain a persistent session in IRC through the use of a terminal multiplexer (such as screen or tmux) and SSH. While this has the advantage of being very easy to setup, there are a few very obvious disadvantages and trade-offs:

  1. You must be on a machine that is able to SSH into your remote server.
  2. You are confined to using the command-line IRC client running in the remote session.
  3. Any scripts that attempt to interact with your ‘local’ desktop (e.g. Growl notifications), are painful to setup, if not impossible.

For many, including myself, the first two are deal-breakers. I use a variety of computing devices on a daily basis, and asking me to SSH into a remote server from my smartphone to then attach a screen session is not reasonable. Luckily, there does exist a method for establishing a ‘perpetual’ IRC session that does not suffer from any of the disadvantages brought on by the SSH + multiplexer approach.

Enter Bip

Bip is an IRC proxy by Arnaud Cornet and Loïc Gomez, and is released under the GPLv2 license. From their about page:

[Bip keeps you] connected to your preferred IRC servers, can store the logs for you, and even send them back to your IRC client(s) upon connection. You may want to use bip to keep your logfiles (in a unique format and on a unique computer) whatever your client is, when you connect from multiple workstations, or when you simply want to have a playback of what was said while you were away.

Strangely, Bip does not seem to be very well known amongst people I would call “heavy” IRC users, despite it’s very lengthy list of useful features and the relatively simple and straightforward setup.

Remote Server Setup

It may be possible to install Bip through a package manager, but we’ll go right to building from source to ensure that everyone is on the same page. Feel free to skip this part if you are able to install bip through your distribution’s package manager.

Note: You will need to have git, make, gcc, lex and yacc (the latter two are for lexing/parsing the bip configuration file) to compile from source.

$ mkdir src && cd src
$ git clone http://rcs-git.duckcorp.org/projects/bip/bip.git
$ cd bip
$ ./bootstrap
$ ./configure # Use the --prefix option to install somewhere other than /usr/local
$ make && make install

Assuming we have the proper dependencies installed and there were no errors, we should have two binaries — bip and bipmkpw — installed in /usr/local/bin/ (or whatever path was specified for the ‐‐prefix option during the configure phase. In case you decided to skip the make install part of the above instructions, the compiled binaries are available in the src folder of your bip source directory. Place these binaries in an appropriate location, preferably reachable via $PATH.

Configuration

Note: Nearly all of this configuration information can be found in the README file included with bip.

For this configuration file to work, we need to create a directory to store a few things:

$ mkdir -p ~/.bip

Let’s start off with the most bare-bones configuration file possible. Create a file named bip.conf in your $HOME/.bip/ directory with the following contents:

Alright, let’s go through what these options actually do.

ip = "0.0.0.0";
port = 6667;

The ip string indicates from which IP bip should accept incoming connections, and port is self-explanatory. By setting it to 0.0.0.0, bip will accept a connection from any IP, which is most likely what you want. Additionally, restricting incoming connections to certain IP addresses for particular ports is something that can (and should) be pushed to your firewall.

Note: Be sure that you have left an opening in your firewall for connecting to your proxy!

log_level = 2;
log_root = "$HOME/.bip/logs";
log_sync_interval = 5;

A log_level of 2 indicates that we want to log all errors and warnings, but ignore info and debug messages. The log_root indicates where logs should be stored, and log_sync_interval dictates at what time interval the log file is written to disk.

network {
    name = "freenode";
    server { host = "irc.freenode.net"; port = 6667; };
};

Next, we define a network that we would like to connect to. You can specify an arbitrary name — I could have just as easily called this network opensource — and the relevant connection information in the server key.

user {
    name = "joel";
    password = "<some_long_password_hash>";
    default_nick = "jperras";
    default_user = "jperras";
    default_realname = "Joël";
        # ...

Here, we define a bip user named “joel” (this name is only used when you connect from your local client to your proxy), and a password. The password that you insert here should be created through the bipmkpw binary, which will salt & hash an input string to produce the value that needs to be placed in the above user {} block. The other three options of default_nick, default_user, and default_realname should be self-evident if you have ever used IRC before.

    backlog = true;
    backlog_lines = 200;
    backlog_always = false;
    backlog_reset_on_talk = false;
    backlog_reset_connection = false;
    backlog_msg_only = true;

Now we start getting into the more interesting configuration options for bip. backlog = true enables backlogging, which means that when your local client (e.g. smartphone, desktop client, web client) reconnects to your proxy, all the messages that have been logged since your last connection will be replayed to you. This is, in my opinion, the best reason to consider using an IRC proxy such as bip; you’ll never miss a message, conversation or context again.

backlog_lines = 200 sets an upper limit to the number of messages per channel that will be sent to your client upon reconnection.

backlog_always = false indicates that you don’t want to receive backlog messages that you have already seen. By setting this option to true, every time you connect a client to your proxy you will receive backlog_lines of backlog messages for each channel. If you do enable this option (which comes in handy when you regularly connect via two or more devices), make sure the value for backlog_lines is something reasonable.

backlog_reset_on_talk and backlog_reset_connection indicate that the backlog marker should be reset every time you talk, scoped to the particular channel or entire connection, respectively.

By setting backlog_msg_only = true, we only log messages and ignore all IRC events, including joins/parts/quits/nick changes, and the like.

On to the connection definition:

    connection {
        name = "freenode";              # used by bip only
        network = "freenode";           # which IRC network to connect to

        # Autojoined channels:
        channel { name = "#awesome-channel"; };
        channel { name = "#secret-channel"; key = "passw0rd"; };
        channel { name = "#unimportant"; backlog = false; };
    };

First, we give a name to the connection (which will be useful in the following segment), and specify which network (defined previously) that we wish to connect to.
Finally, we specify the channels that we wish to automatically connect to when first starting up the bip proxy. Note the use of key for password protected channels, and the backlog = false usage for channels that we never want to receive a backlog for.

Note that these channels are auto-joined when the bip daemon starts, and not each time your local client connects to the proxy. Once bip has started up and we connect to it, we are free to join/part channels at will, and these will be persisted between client connections.

Starting The Server

Alright, let’s start the bip process:

$ /usr/local/bin/bip -f /path/to/bip.conf

Note: If things don’t seem to be working as expected, tail’ing the bip log file may yield some clues as to what is going wrong.

The bip process will daemonize itself by default; use the -n option to have it remain in the foreground, if so desired.

When you start the bip process, it initiates the IRC handshake & dance that normally occurs when you connect a local client to an IRC network. The channels that you defined in the bip configuration file are now connected to and being kept open by bip.

Connecting to the Proxy

Now that we’ve started the bip proxy on our remote server, let’s fire up an IRC client and configure the server connection information.

Examples

I’ve included a few examples of how the server connection information should look like (given the example bip configuration that I’ve provided above), in both a GUI-based client as well as a text-based one.

Important: The password that must be used when connecting to your proxy must follow this pattern:

username:unHashedPassword:network

Where the username portion corresponds to the value of the name field in the user { } block in your bip configuration, the unHashedPassword portion is the un-hashed version of the password that you generated (using bipmkpw) and pasted in your config, and network is the name of the IRC network (also defined in your configuration) that you wish to connect to.

An interesting thing to note is that, with this particular scheme, it’s quite simple to connect to a different network, or to add a new user to the proxy.

Colloquy

Irssi
servers = (
  {
    address = "my-proxy-host.com";
    chatnet = "Proxy";
    port = "7778";
    password = "joel:myHashedPassword:freenode";
    autoconnect = "yes";
  }

Conclusion

And that’s it! We are now perpetually connected to IRC, and can connect to our proxy using an unlimited number of devices in a completely transparent and seamless manner. Moreover, the logs for all channels that we are connected to are saved and automatically rotated on our remote server, making for an easy to maintain archive in the absence of another external channel logging mechanism. Additionally, we can now generate an OpenSSL certificate for our remote server, and configure bip as well as our local clients to communicate with our proxy over a secured connection. This, along with many other features and options that I did not cover, are all documented in the README and man pages that are installed alongside the bip binary; Please take a look for more information on using bip to it’s full potential.

Perpetual IRC – The Multiplexer Edition

As is the case with most people involved in Open Source, I’m on IRC all day long. I can help people from around the world use some of the projects I’ve helped create, as well as some of the software that I use on a daily basis.

Additionally, IRC is also the preferred form of group communication for the company I work for. One of the major advantages that IRC has over your ‘traditional’ instant messenger clients is that, with a minimum amount of effort and hardware, you can create a setup that will remain perpetually* connected, even when you’re not online. That means that you can keep logs of conversations, receive messages, catch up on what the current topic of discussion is when starting your day, and still be able to shut down your computer at night if you so choose.

There are a few ways of achieving the status of perpetual IRC denizen. Note, however, that almost any method requires that you have access to a remote server in addition to your local machine. It doesn’t need to be anything special – a cheap VPS will do just fine, as long as you can install software packages and open the necessary ports in the firewall.

Simplest. If you want to expend the least amount of effort, at the cost of some flexibility, the easiest method of achieving IRC immortality is with the use of a terminal multiplexer, such as screen or tmux (my personal favourite), and a console-based IRC client, such as Irssi.

Don’t know what a terminal multiplexer is? Here’s a description, right from the screen homepage:

Screen is a terminal multiplexer: it takes many different running processes and manages which of them gets displayed to the user. Think of it as a window manager for your console or terminal emulator. With screen, you can have interactive processes running on your home computer and connect to them from anywhere else to check on their progress.

Let’s assume that you’ve installed screen and Irssi on your remote server. What we want to do is start Irssi inside a screen session:

$ screen -S irc irssi # name the session `irc` and start `irssi` in it

Digression. At this point, feel free to browse the documentation on how to properly setup your preferred servers, channels and other configuration parameters for the IRC client. For a quick test, you can try the following at the Irssi prompt:

/set nick YourNickName
/connect irc.freenode.net
/join #some-interesting-channel

You can also automate the process, so that upon startup Irssi will auto-connect to the servers and channels you specify:

/server add -auto -network Freenode irc.freenode.net 6667
/network add -nick YourNickName Freenode
/channel add -auto #some-interesting-channel Freenode

If you have a registered nickname, you can have it auto-identify as well:

/network add -autosendcmd "/msg nickserv identify your_pasword_here ;wait 2000" Freenode

And you’ll probably want to cut down on the noise:

/ignore #some-interesting-channel ALL -PUBLIC -ACTIONS

At this point, you should have a fully functioning IRC client inside of a screen session. You can detach from the current session by pressing control-a d, and re-attach with:

$ screen -r

See where this is going? With this setup, you can simply maintain your IRC connection within a screen session on your remote server. When you want to “log on”, you simply SSH into your remote server, screen -r, and voilà! Perpetual IRC.

This approach, while simple, has some very obvious drawbacks:

  1. You must be on a machine that is able to SSH into your remote server.
  2. You are confined to using the command-line IRC client running in the remote session.
  3. Any scripts that attempt to interact with your ‘local’ desktop (e.g. Growl notifications), are painful to setup, if not impossible.

Okay, so it’s not completely ideal. But it does the job – as long as your remote screen session remains operational, you’ll be logged in to IRC.

In my next post, we’ll look at how you can achieve the same results, but without any of the aforementioned drawbacks, through the use of a specialized IRC proxy daemon.


*: Of course, if the remote server running the terminal multiplexer session goes down, it’s not really ‘perpetual’.