I figured it was easy to have a reverse tunnel for more application and especially the one i wanted to quickly show my friend.

Perhaps a plugin is a bit of overkill for just a rake task but what the heck here it is.

In order to install the plugin in your rails app just run

$ ./script/plugin install http://svn.stevenkroon.com/plugins/tunnel

Now you should have access the specific tunnel rake tasks, you can run

$ rake -T

From your rails root, and discover new tunnel specific tasks.

rake tunnel:create_config            # Create reverse tunnel default config
rake tunnel:start                    # Starts a reverse tunnel
rake tunnel:status                   # Check if reverse tunnel is running

First thing todo is to create the tunnel config file, the easiest way is to run the create_config rake tastk.

$ rake tunnel:create_config

Now you will have a config file called ./config/tunnel.yml, change this file according to your needs.

remote_user: steven
remote_host: rails.stevenkroon.com
remote_port: 9999
local_port: 3000

The remote_user should be your ssh login name on the remote_host. If you just change the remote_user and remote_host configs, the only thing left todo is.

$ rake tunnel:start
(in /Users/steven/svn/railstt)
* Tunneling rails.stevenkroon.com:9999 to 127.0.0.1:3000
* press ctrl-c to quit the tunnel

If you haven’t setup ssh authorization keys you will be prompted for the password of your remote_user. As the message states, a reverse tunnel on the remote_host on port remote_port is started, meaning that if you would connect against this url, in my case http://rails.stevenkroon.com:9999 you will get the output of whatever is running on 127.0.0.1:3000, in my case a ruby on rails project.

So if you launch your Rails application you can now see this application when going to the external hosts url.

Now this is just for quick test and should not be seen as a deployment for your rails app, just a easy way to have a external entry point into your application running on your localhost also this is needed when developing facebook application since the call back url need to point to your running application.

There are a couple of commands which we are going to use in order the schedule the tasks, lets walk through all of them one at a time.

at

At alllows a fairly complex time format, but lets stick with the obvious format (at least which i find obvious).

$ at now + 5 minutes

now at will accept commands from standard input, you will get a shell like prompt where you can type in your commands.

at> date > /tmp/date
at>

Now press CTRL+D, after that you will be dropped back to the shell and at will report to you at which time the command will be executed.

at> <EOT>
job 9 at Mon Mar 25 10:43:00 2007

Here you see that the command will be executed at 10:43:00, now that’s a coincident, it look like we typed our at command exactly within the first second of the 38th minute.

Let’s double check if it really works like this.

$ echo 'echo "scheduling tasks in linux" > /tmp/banner.txt' > /tmp/schedule; \
date; \
at now + 1 minute -f /tmp/schedule
Mon Mar 26 10:49:54 CEST 2007
warning: commands will be executed using /bin/sh
job 12 at Mon Mar 26 10:50:00 2007

Here we use at a little different instead of letting at read our command from standard input, we use the -f command line option to let at read our command from a file. As you can see we create this file in the beginning then we print the date and then run at.

I’m consolidating this 3 separate commands on one line in order to be sure that the three almost happen instantly, so we don’t have much time skew here.

As you can see in the output we enter our at schedule at “10:49:54″ you might have expected the execution of the at schedule to run at “10:50:54″ since we did the “now + 1 minute”. However it runs the at schedule at “10:50:00″, so be aware that it will execute it at the full minute.

Also there is another thing to note in our output “warning: commands will be executed using /bin/sh”, what this means is that. When at invokes the commands in our at schedule it will spawn a new shell for this. So don’t expect to see output come bouncing back to your shell where you invoked the at schedule. However standard output and standard error will be e-mailed to shell user invoking the at schedule. These mail message will have a subject line like

Output from your job 42

In this case the mail body will contain the Output form at job 42.

Sometimes however you would also like to get notified if a job has run, when there is no output given you can do this with the -m parameter, e.g.

$ echo 'date > /dev/null' > /tmp/jobs
$ at 10pm tomorrow -f /tmp/jobs -m

Even though this job will not output anything, at least it will be redirected to /dev/null, it will e-mail you about it.

Also you can see an other time format “10pm tomorrow”. Tomorrow is a easy way of specifying that a task as to run tomorrow, instead of looking at the exact date. There is also “today” if you want to specify today.

Lets show some other time formats, they are actually really simple to memorize because the almost read like a normal sentence.

$ at march 29 2007 -f /tmp/jobs

We just tell it the month, day and year. If you specify it like this, the task will be executed on march 29 2007 and at the hour and minute you run at. If you want a different time you can specify this as well.

$ at 6pm january 1 2008 -f /tmp/jobs

There is also a syntax which i use pretty often and that i using offsets. So let’s say you started a benchmark today and what to run it for a week you could do something like

$ at 4pm + 7 days -f /tmp/stop-benchmark

This will run the commands in /tmp/stop-benchmark, however you can even write this much better.

$ at 4pm + 1 week -f /tmp/stop-bencmark

atq

Okay now we have a tasks scheduled, now how do we see which tasks are actually scheduled? The atq command lists all the pending scheduled tasks. You could also use “at -l” to see the same results. Lets see it in action.

$ atq
25      Mon Mar 26 23:05:00 2007 a steven
26      Mon Apr  2 18:00:00 2007 a steven

Here we see i’ve got two scheduled tasks pending, the first scheduled for March 26 at 23:05 and the later on April 2 at 18:00. Now i can’t even remember what the first task was, so lets look at what it actually does.

$ at -c 25
#!/bin/sh
# atrun uid=1000 gid=1000
# mail steven 0
umask 22
USER=steven; export USER
MAIL=/var/mail/steven; export MAIL
PATH=/usr/sbin:/usr/bin:/sbin:/bin:export PATH
PWD=/home/steven; export PWD
LANG=en_US.UTF-8; export LANG
HISTCONTROL=ignoredups; export HISTCONTROL
SHLVL=1; export SHLVL
HOME=/home/steven; export HOME
LOGNAME=steven; export LOGNAME
cd /home/steven || {
         echo 'Execution directory inaccessible' >&2
         exit 1
}
date > /dev/null

Here you can see that at records all the SHELL ENVIRONMENT variables at the time when at was invoked. So you can set environment variables when invoking at and it will record them so they are still available when the job start running.

atrm

All the way at the bottom of the previous output we see the actual command “date > /dev/null”. This was just a bogus job, since i used it only to test out some stuff. Lets remove it right away

$ atrm 25

And it should be gone, lets verify

$ atq
26      Mon Apr  2 18:00:00 2007 a steven

Yep job 25 is gone.

batch

Lets say we need to run some maintenance scripts todo some house keeping, but currently the system is under high load, so we don’t want to stress the server out even more with our maintenance jobs. We can however schedule this commands to be run whenever the load average drops below 1.5.

$ batch
warning: commands will be executed using /bin/sh
at> updatedb
at> /usr/local/sbin/clean-database.rb
at> <EOT>
job 32 at Mon Mar 26 14:02:00 2007

This 1.5 is the compiled in value, and you can change this value with the -l paramater of atd. You might want to change this value if you are in SMP environment.

Let’s look at the pending jobs.

$ atq
32      Mon Mar 26 14:02:00 2007 b steven
26      Mon Apr  2 18:00:00 2007 a steven

There we see it, job number 32, notice that there is a “b” instead of a “a”, this column specifies in which Queue the job is. Queue a is the default queue “at” and Queue b is the default queue for “batch”.

There is also a special Queue called “=” this is queue is reserved for jobs which are currently running.

$ at now + 1 minute
warning: commands will be executed using /bin/sh
at> sleep 300
at> <EOT>
job 33 at Mon Mar 26 14:08:00 2007

Now if we wait a little and check the running queue we can see it running.

$ atq
32      Mon Mar 26 14:02:00 2007 b steven
26      Mon Apr  2 18:00:00 2007 a steven
33      Mon Mar 26 14:08:00 2007 = steven

That’s about it. There are some more advantaged features however these are the once i use the most and which seems to be the most practical once.

There should be a shortcut to this command if you ask me, nice thing is we can create it ourselfs. It would be nice just to be able to type in something simple like search-apt or even simpler sapt packagename, in fact it is pretty easy to make something like this.

There are a couple of ways doing the same thing. Here are a few ways of doing it.

Alias

The first method is using the aliasing mechanism. You can put this alias in your /etc/profile or perhaps in your ~/.profile or your what ever your shell reads upon login e.g. ~/.bashrc, what ever suites you best.

$ alias sapt='apt-cache search $*'

$ sapt irb
irb - Interactive Ruby (irb)
irb1.6 - Interactive Ruby (irb) 1.6.x
lg-issue88 - Issue 88 of the Linux Gazette.
irb1.8 - Interactive Ruby (for Ruby 1.8)

As we can see it works perfectly, now lets put it in the /etc/profile file.

$ echo -e "\nalias sapt='apt-cache search \$*'" |sudo tee -a /etc/profile

# Alias 'apt-cache search' into a single sapt command
alias sapt='apt-cache search $*'

Bash Script

You could also create a bash script, and place it in something like /usr/local/sbin/sapt, ofcourse make sure that /usr/local/sbin is in your $PATH environment variable.

$ sudo tee /usr/local/sbin/sapt << EOF
> #!/bin/sh
> apt-cache search \$*
> EOF
#!/bin/sh
apt-cache search $* 
$ sudo chmod +x /usr/local/sbin/sapt

Now you can do:

$ sapt irb
irb - Interactive Ruby (irb)
irb1.6 - Interactive Ruby (irb) 1.6.x
lg-issue88 - Issue 88 of the Linux Gazette.
irb1.8 - Interactive Ruby (for Ruby 1.8)

While I’m a pretty fast typer, but this still saved me a lot of time.

This tool can do more then just resolve domain names to ip addresses or visa versa. Lets look more closely into the world of domain name resolution and explorer the possibilites with dig. We will quickly run through the basics and at the end look at some nice features of dig.

Basic query

Let’s get straight to work, first we will do a basic ip address lookup. Lets say we would like to know which ip address is associated with www.stevenkroon.com. I’ve got a local name server with ip address 192.168.7.254, so i’ll be performing the query on this machine.

$ dig @192.168.7.254 www.stevenkroon.com | cat -b

 1  ; <<>> DiG 9.2.4 <<>> @192.168.7.254 www.stevenkroon.com
 2  ;; global options:  printcmd
 3  ;; Got answer:
 4  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55392
 5  ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 2

 6  ;; QUESTION SECTION:
 7  ;www.stevenkroon.com.           IN      A

 8  ;; ANSWER SECTION:
 9  www.stevenkroon.com.    86376   IN      CNAME   stevenkroon.com.
10  stevenkroon.com.        86376   IN      A       80.247.216.241

11  ;; AUTHORITY SECTION:
12  stevenkroon.com.        86376   IN      NS      ns1.bitrey.nl.
13  stevenkroon.com.        86376   IN      NS      ns2.bitrey.nl.

14  ;; ADDITIONAL SECTION:
15  ns1.bitrey.nl.          86376   IN      A       80.247.216.241
16  ns2.bitrey.nl.          86376   IN      A       80.247.216.247

17  ;; Query time: 2 msec
18  ;; SERVER: 192.168.7.254#53(192.168.7.254)
19  ;; WHEN: Sun Sep 10 21:30:13 2006
20  ;; MSG SIZE  rcvd: 144

As you can see i’ve run dig through cat so we get some line numbering, which makes it easier to discuss certains parts of the output. These line numbers are not included when using dig. If you would like to read all about cat please read dog is better than cat.

As you can see the basic syntax is

dig @server hostname type

Where server is the name or ip address of the name server you want to query and name is the resource record that is to lookup and finally the type, which can be ANY, A, MX, SIG, etc. You can omit the type, in this case it will look up a “A” record. “A” stands for “Address” which is just what we are looking for, so we dont need to specify it.

Sections

DNS packets are composed of five sections

The HEADER SECTION is present in every query and response, we will talk about this section later on.

The QUESTION SECTION tell us what we are doing, if you look at line 6 and 7 of the output, you see that in our case we are looking for the A record of www.stevenkroon.com. Maybe a good way to read this is query is “We are looking for the address of www.stevenkroon.com”.

On line 8 to 10 we have the ANSWER SECTION, this tells us what we asked for. There is a twist on line 9 though, the name server looked up the address but found out that it was not a “A” record, instead it was a CNAME to a other domain name. In this case the CNAME refers to stevenkroon.com. A CNAME record maps an alias to its canonical name, so line 9 can be read as “www.stevenkroon.com maps to the ip address belonging to stevenkroon.com”. Because we were looking for the address associated with *www.stevenkroon.com the name server also tries to resolve the A record of stevenkroon.com, this is what happens on line 10. So the answers to our query is “www.stevenkroon.com has the same ip address as the domain stevenkroon.com which in turn has an ip address of 80.247.216.241″.

The AUTHORITY SECTION tells us which name servers can give is a authoritative answers to our questions. The name servers that can give us these for “www.stevenkroon.com” are listed on line 12 and 13. So this means that if we query these name servers directly we can assume the this data is correct and not outdated because of caching on local name server. We will get to this issue later on.

And the ADDITIONAL SECTION adds information that may complete information included in other sections. For instance name servers that are listed in the authority section.

After a while of head scratching we thought lets try resolving the Virtual Control Center from one of the ESX nodes.

$ dig vcc.test.stevenkroon.com
;; Truncated, retrying in TCP mode.
;; Connection to 192.168.7.254#53(192.168.7.254) for 
   vcc.test.stevenkroon.com failed: connection refused.

Aha! That’s strange, it can’t resolve the Virtual Control Centers ip address, probably ESX tries to resolve the hostname of the Virtual Control Center machine machine and can’t resolve it. But why is that?

If you know a little bit about DNS (if not you might consider reading Dig it! which gives a basic overview of DNS) you know that queries are send over UDP per default however, if there tend to be too much data to send back, resolvers drop the UDP packet and retry in TCP mode. Lets just ignore the truncation, and just receive the UDP packet and blindly accept its data was not mangled in transit.

$ dig vcc.test.stevenkroon.com +ignore

; <<>> DiG 9.2.4 <<>> vcc.test.stevenkroon.com +ignore
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42395
;; flags: qr aa tc rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 23, ADDITIONAL: 0

;; QUESTION SECTION:
;vcc.test.stevenkroon.com.      IN      A

;; ANSWER SECTION:
vcc.test.stevenkroon.com. 86400 IN      A       192.168.7.2

;; AUTHORITY SECTION:
test.stevenkroon.com.   86400   IN      NS      ns10.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns11.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns12.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns13.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns14.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns15.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns16.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns17.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns18.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns19.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns20.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns21.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns22.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns23.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns24.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns25.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns26.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns27.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns28.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns29.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns30.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns31.test.stevenkroon.com.
test.stevenkroon.com.   86400   IN      NS      ns32.test.stevenkroon.com.

;; Query time: 5 msec
;; SERVER: 192.168.7.254#53(192.168.7.254)
;; WHEN: Tue Sep 12 20:11:57 2006
;; MSG SIZE  rcvd: 495

Ah so that does work, so the problem is related to DNS query that are done in TCP mode. Because this zone has so many NS records the size of the reply probably gets beyond the threshold to put this in a UDP packet, so our resolver tries to ask for the same query but then in TCP mode. Lets look at the network traffic that gets send. From the nameserver I’ll be doing a tcpdump, just to look at what is happening out there.

$ sudo tcpdump -qnei eth1 port 53 and host 192.168.7.3
 1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 2 listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
 3 14:32:40.339322 00:50:56:49:aa:45 > 00:10:5a:37:15:ae, IPv4, length 84: 
 . IP 192.168.7.3.32769 > 192.168.7.254.53: UDP, length: 42
 4 14:32:40.343649 00:10:5a:37:15:ae > 00:50:56:49:aa:45, IPv4, length 550: 
 . IP 192.168.7.254.53 > 192.168.7.3.32769: UDP, length: 508

While the tcpdump was running I did a normal query for vcc.test.stevenkroon.com, as you can see on line number 3 and 4, these packets are UDP. However we don’t get a retry in TCP mode, we only see UDP packets on our nameserver. So why isn’t ESX sending us the TCP packets?

Firewalling could prevent us from sending out TCP DNS packets. Since this is a testing environment we can without troubles turn of the firewall.

$ sudo /etc/init.d/firewall stop
Stopping firewall                                          [  OK  ]

With the firewall turn off lets try again.

$ dig vcc.test.stevenkroon.com +short
;; Truncated, retrying in TCP mode.
192.168.7.2

Got ‘em! So there is something with this firewalling that prevents TCP DNS packets going out, we know this because we ware not receiving any TCP packets on our nameserver. Lets start the firewall again and look at the firewalling rules.

$ sudo /etc/init.d/firewall start
Starting firewall                                          [  OK  ]
$ sudo /sbin/iptables -nL OUTPUT
Chain OUTPUT (policy DROP)
target     prot opt source          destination         
ACCEPT     all  --  0.0.0.0/0       0.0.0.0/0          
valid-tcp-flags tcp  --  0.0.0.0/0  0.0.0.0/0          
icmp-out   icmp --  0.0.0.0/0       0.0.0.0/0          
ACCEPT     udp  --  0.0.0.0/0       0.0.0.0/0  udp spts:1024:65535 dpt:53 
ACCEPT     all  --  0.0.0.0/0       0.0.0.0/0  state RELATED,ESTABLISHED 
ACCEPT     tcp  --  0.0.0.0/0       0.0.0.0/0  tcp dpt:902 state NEW 
ACCEPT     udp  --  0.0.0.0/0       0.0.0.0/0  udp spts:67:68 dpts:67:68 
ACCEPT     tcp  --  0.0.0.0/0       0.0.0.0/0  tcp dpts:2050:5000 state NEW 
ACCEPT     udp  --  0.0.0.0/0       0.0.0.0/0  udp dpts:2050:5000 state NEW 
ACCEPT     tcp  --  0.0.0.0/0       0.0.0.0/0  tcp dpts:8042:8045 state NEW 
ACCEPT     udp  --  0.0.0.0/0       0.0.0.0/0  udp dpts:8042:8045 state NEW 
ACCEPT     udp  --  0.0.0.0/0       0.0.0.0/0  udp spt:427 
ACCEPT     tcp  --  0.0.0.0/0       0.0.0.0/0  tcp spt:427 state NEW 
ACCEPT     tcp  --  0.0.0.0/0       0.0.0.0/0  tcp dpt:27000 state NEW 
ACCEPT     tcp  --  0.0.0.0/0       0.0.0.0/0  tcp dpt:27010 state NEW 
ACCEPT     tcp  --  0.0.0.0/0       0.0.0.0/0  tcp dpt:21 ctstate NEW,RELATED 
ACCEPT     udp  --  0.0.0.0/0       0.0.0.0/0  udp dpt:902 state NEW 
REJECT     all  --  0.0.0.0/0       0.0.0.0/0  reject-with icmp-port-unreachable

Line number 6 on the output says:

ACCEPT     udp  --  0.0.0.0/0        0.0.0.0/0  udp spts:1024:65535 dpt:53

This is the rule that allows UDP DNS packets going out, however there is no such rule for TCP packets, so this must be the root cause of why we are not getting any TCP DNS packets on our nameserver. Lets fix it!

sudo /usr/sbin/esxcfg-firewall -o 53,tcp,out,dnsClientTCP

This will add the following line to the OUTPUT firewall chains:

ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0          tcp dpt:53

That’s the theory, lets see if it actually works. The firwall is still turned on so we are good to go.

$ dig vcc.test.stevenkroon.com +short
;; Truncated, retrying in TCP mode.
192.168.7.2

There we go! To wrap it up lets check the tcpdump output this time.

$ sudo tcpdump -qnei eth1 port 53 and host 192.168.7.3
 1 15:06:38.178860 00:50:56:49:aa:45 > 00:10:5a:37:15:ae, IPv4, length 84:
 . IP 192.168.7.3.32769 > 192.168.7.254.53: UDP, length: 42
 2 15:06:38.183241 00:10:5a:37:15:ae > 00:50:56:49:aa:45, IPv4, length 549:
 . IP 192.168.7.254.53 > 192.168.7.3.32769: UDP, length: 507
 3 15:06:38.184584 00:50:56:49:aa:45 > 00:10:5a:37:15:ae, IPv4, length 74:
 . IP 192.168.7.3.33003 > 192.168.7.254.53: tcp 0
 4 15:06:38.184630 00:10:5a:37:15:ae > 00:50:56:49:aa:45, IPv4, length 74:
 . IP 192.168.7.254.53 > 192.168.7.3.33003: tcp 0
 5 15:06:38.184758 00:50:56:49:aa:45 > 00:10:5a:37:15:ae, IPv4, length 66:
 . IP 192.168.7.3.33003 > 192.168.7.254.53: tcp 0
 6 15:06:38.184813 00:50:56:49:aa:45 > 00:10:5a:37:15:ae, IPv4, length 110:
 . IP 192.168.7.3.33003 > 192.168.7.254.53: tcp 44
 7 15:06:38.184846 00:10:5a:37:15:ae > 00:50:56:49:aa:45, IPv4, length 66:
 . IP 192.168.7.254.53 > 192.168.7.3.33003: tcp 0
 8 15:06:38.189732 00:10:5a:37:15:ae > 00:50:56:49:aa:45, IPv4, length 1235:
 . IP 192.168.7.254.53 > 192.168.7.3.33003: tcp 1169
 9 15:06:38.190741 00:50:56:49:aa:45 > 00:10:5a:37:15:ae, IPv4, length 66:
 . IP 192.168.7.3.33003 > 192.168.7.254.53: tcp 0
10 15:06:38.191177 00:50:56:49:aa:45 > 00:10:5a:37:15:ae, IPv4, length 66:
 . IP 192.168.7.3.33003 > 192.168.7.254.53: tcp 0
11 15:06:38.191905 00:10:5a:37:15:ae > 00:50:56:49:aa:45, IPv4, length 66:
 . IP 192.168.7.254.53 > 192.168.7.3.33003: tcp 0
12 15:06:38.192035 00:50:56:49:aa:45 > 00:10:5a:37:15:ae, IPv4, length 66:
 . IP 192.168.7.3.33003 > 192.168.7.254.53: tcp 0

As you can see the first 2 lines are the orignal UDP packets, and after that are the TCP packets.

We talkd to the guys of VMware and agreed that this was indeed a bug, and it will be resolved in the upcoming versions of ESX 3.

NOTE: output of “ls -l” is edited in order to reduce the width on this page. Also command line commands might be truncated via the backslash character.

Java

Lets dive right in and start looking at the problems described above. The first problem we encountered was the java compiler. If you installed the sun java package via “java-package” package from debian then your all set, debian already configured the alternatives for you.

$ java -version            
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode, sharing)

This tells us that currently we are running the sun java version, also checkout the man page of java. Notice that NAME section reads.

java - Java application launcher

If you scroll all the way to the bottom of the man page down to the SEE ALSO section, you see external links to the http://java.sun.com website. So clearly we are dealing with the sun java man pages as well. Like i said if you installed the “java-package” then debian has already setup the alternatives system for you. Let us inspect the java binary.

$ which java
/usr/bin/java
$ ls -l /usr/bin/java
lrwxrwxrwx  /usr/bin/java -> /etc/alternatives/java

As you can see the we tried to locate the java command we found that it was being execute from the /usr/bin path, however as you can see it’s a symbolic link and it links to /etc/alternatives/java. The /etc/alternatives directly is where all the alternatives are configured, let inspect a little further.

$ ls -l /etc/alternatives/java
lrwxrwxrwx  /etc/alternatives/java -> /usr/lib/j2re1.5-sun/bin/java
$ ls -l /usr/lib/j2re1.5-sun/bin/java
-rwxr-xr-x  /usr/lib/j2re1.5-sun/bin/java

There we go, another symbolic link, this time referring to /usr/lib/j2re1.5-sun/bin/java and a little digging afterwards reveals us that this is the real java executable. By now you might know how the alternatives system works or maybe you find it very inconvenient to have all these symlinks pointing to other symlinks and so on. What the alternatives system does is that it will link the executable in the default path to the appropriate file in /etc/alternatives and this will finally link to the executable we actually want to use. Because of this it becomes really easy to just update the symlink in alternatives directory and symlink it to the executable we want to use. We don’t do this linking by hand, there is a nice tool for us that manages the symlinks called “update-alternatives”.

Using update-alternatives we could quickly figure out which alternatives are currently configured for a certain executable. Let us look at the current setup of java.

$ /usr/sbin/update-alternatives --display java
java - status is auto.
link currently points to /usr/lib/j2re1.5-sun/bin/java
/usr/lib/kaffe/bin/java - priority 300
slave java.1.gz: /usr/share/man/man1/java.kaffe.1.gz
/usr/lib/j2re1.5-sun/bin/java - priority 315
slave java.1.gz: /usr/lib/j2re1.5-sun/man/man1/java.1.gz
Current `best' version is /usr/lib/j2re1.5-sun/bin/java.

As you can see the current link point to /usr/lib/j2re1.5-sun/bin/java as we already discovered. There is another alternative which is /usr/lib/kaffe/bin/java and it has a priority of 300, the priority is important we will get to that shortly. The priority of /usr/lib/j2re1.5-sun/bin/java is 315. Did you notice the “java - status is auto.” on top? This means that the alternatives system will look at which package as the highest priority and uses that executable. Because of this the last line says that the current best priority link is /usr/lib/j2re1.5-sun/bin/java.

Now if we want to use kaffe instead, we can tell update-alternatives to use kaffe instead, let us do this by using the interactive mode.

$ sudo /usr/sbin/update-alternatives --config java 

There are 2 alternatives which provide `java'.

Selection    Alternative
-----------------------------------------------
      1        /usr/lib/kaffe/bin/java
*+    2        /usr/lib/j2re1.5-sun/bin/java

Press enter to keep the default[*], or type selection number: 1
Using `/usr/lib/kaffe/bin/java' to provide `java'.

In interactive mode we see al the available alternatives for java. We already knew by looking at the –display output that there where only two option. Now we can select which option we want and as you can see i selected select number 1. No let’s look at the output of display again.

$ /usr/sbin/update-alternatives --display java
java - status is manual.
link currently points to /usr/lib/kaffe/bin/java
/usr/lib/kaffe/bin/java - priority 300
slave java.1.gz: /usr/share/man/man1/java.kaffe.1.gz
/usr/lib/j2re1.5-sun/bin/java - priority 315
slave java.1.gz: /usr/lib/j2re1.5-sun/man/man1/java.1.gz
Current `best' version is /usr/lib/j2re1.5-sun/bin/java.

There we got “link currently points to /usr/lib/kaffe/bin/java” so let’s do the same checking as we did before.

$ which java
/usr/bin/java
$ ls -l /usr/bin/java
lrwxrwxrwx  /usr/bin/java -> /etc/alternatives/java
$ ls -l /etc/alternatives/java
lrwxrwxrwx  /etc/alternatives/java -> /usr/lib/kaffe/bin/java
$ java -version
Kaffe Virtual Machine

Copyright (c) 1996-2004 Kaffe.org project contributors (please see
  the source code for a full list of contributors).  All rights reserved.
Portions Copyright (c) 1996-2002 Transvirtual Technologies, Inc.

The Kaffe virtual machine is free software, licensed under the terms of
the GNU General Public License.  Kaffe.org is a an independent, free software
community project, not directly affiliated with Transvirtual Technologies,
Inc.  Kaffe is a Trademark of Transvirtual Technologies, Inc.  Kaffe comes
with ABSOLUTELY NO WARRANTY.

Engine: Just-in-time v3   Version: 1.1.5   Java Version: 1.1

Only the link within the /etc/alternatives got changed just as we expected. When looking at the java version we see that it’s using the Kaffe Virtual Machine, great!

It gets even better, look at the man page of java. The NAME section now reads “kaffe - a virtual machine to execute Java(tm) bytecode”. So it not only changed the binary executable but also the man page. You might wonder how it knows this, if you paid attention to the output of –display you might have noticed the lines starting with slave.

$ /usr/sbin/update-alternatives --display java
java - status is manual.
link currently points to /usr/lib/kaffe/bin/java
/usr/lib/kaffe/bin/java - priority 300
slave java.1.gz: /usr/share/man/man1/java.kaffe.1.gz
/usr/lib/j2re1.5-sun/bin/java - priority 315
slave java.1.gz: /usr/lib/j2re1.5-sun/man/man1/java.1.gz
Current `best' version is /usr/lib/j2re1.5-sun/bin/java.

With update-alternatives it’s possible to create a group of files which all have a relation with each other, just like our java.kaffe manpages having a relation with the kaffe java executable. This is done with slaves, looking at the –display output you can see that java.1.gz is a slave to the master file /usr/lib/kaffe/bin/java. So when the master file is changed so is the slave file.

You can also setup your own alternatives, let’s do this with our second problem which is having “different ruby interpeters”.

cat

Everyone uses cat, cat stand for catenate or concatenate. www.answers.com says the following about the two words.

1. To connect in a series of ties or links; form into a chain.
2. Computer Science. To arrange (strings of characters) into a chained list.

So we can link or tie files together with cat, away for doing this would be.

$ cat file1 file2
This is file1
This is file2

This would output the two files to standard output, we could also merge/catenate/link them into a single file.

$ cat file1 file2 > bothfiles
$ cat bothfiles
This is file1
This is file2

Lot’s of people use cat just for this. However there are more nice tricks to cat then just catenate. Lets dive into a couple of handy features. Sometimes text files contain a lot of white-lines after another. In order to reduce al those multiple white-lines into a single line you can use the –squeeze-blank option. First we will create a nice playground. As you might know cat can also read from standard input, for instance.

$ echo -e "Hello world,\n\nWelcome to my blog." | cat
Hello world,

Welcome to my blog.

A bit useless you might think however it’s nice for the examples that will get to shortly. I used the -e option to echo which enables interpretation of the backslash-escaped characters, like \n to print a newline.

Line numbers

In order to print the line numbers we can use –number option to cat.

$ echo -e "Hello world,\n\nWelcome to my blog." | cat -n
    1  Hello world,
    2
    3  Welcome to my blog.

Squeeze

I used the short option to –number which is -n. To get back to our –squeeze-blank subject let’s insert an extra \n to our text.

$ echo -e "Hello world,\n\n\nWelcome to my blog." | cat -n
    1  Hello world,
    2
    3
    4  Welcome to my blog.

As you can see line 2 and line 3 both are white-lines. Image we have like 10 or even more consecutive white-lines. Using cat to read the file contents then these white-lines because useless let’s now squeeze them into a singe white-line.

$ echo -e "Hello world,\n\n\nWelcome to my blog." | cat -n -s
    1  Hello world,
    2
    3  Welcome to my blog.

Here i also use the short option -s instead of the long –squeeze-blank. As you can see instead of 4 lines of text we get 3.

Nonblank

The –number option is really nice, it allows you to quickly identify a line with in a file. If your looking at a file with some other people you can easily talk about a certain line. If however you are not interested in all the white-lines you can also use the –number-nonblank option.

$ echo -e "Hello world,\n\n\nWelcome to my blog." | cat -
    1  Hello world,


    2  Welcome to my blog.

Line endings, TAB endings, etc

Yep -b is the short option to –number-nonblank. Sometimes is really handy to know exactly where a line ends. For instance if you dont want any extra blank space at the end of a line –show-ends to the rescue.

$ echo -e "Hello world,\n\n\nWelcome to my blog." | cat -n -E
    1  Hello world,$
    2  $
    3  $
    4  Welcome to my blog.$

Where -E is the short option for –show-ends. The same thing applies to tabs, if you wan to know if a certain piece of text uses tabs you can use the –show-tabs or -T which is the short option.

$ echo -e "Steven\tKroon\n\nWelcome\t\t\t\tTo\nMy    Blog." | cat -n
    1  Steven  Kroon
    2
    3  Welcome                         To
    4  My    Blog.

Now lets look at the tabs.

$ echo -e "Steven\tKroon\n\nWelcome\t\t\t\tTo\nMy    Blog." | cat -n -T
    1  Steven^IKroon
    2
    3  Welcome^I^I^I^ITo
    4  My    Blog.

As you can see the tabs are now replace by ^I, notice that line 4 does not contain a tab, although it looks like it in the output, it just contains 4 spaces instead of a tab. There is also a way to look for non-printable characters this can be done with –show-nonprinting. If you would like to look at all these things at once you can use the –show-all option to enable these flags at once.

$ echo -e "Steven\tKroon\n\nWelcome\t\t\t\tTo\nMy    Blog." | cat -n -A
    1  Steven^IKroon$
    2  $
    3  Welcome^I^I^I^ITo$
    4  My    Blog.$

And -A being the short option. Well thats about it with the handy features for cat, lets get into a more advanced cat called dog.

In the following descriptions, ^X means control-X. ESC stands for the ESCAPE key; for example ESC-v means the two character sequence “ESCAPE”, then “v”.

more

More is a filter for paging through text one screen at a time. There are two ways of using more. You can use it from the command line and you can use more in interactive mode.

A nice feature of more is you can make it search for a certain keyword directly from the command line. Lets say we where looking for the word wowbagger in a text file called h2g2.txt we could do.

more +/wowbagger h2g2.txt

This will directly scroll to the first occurrence of wowbagger.

If would like to start reading at line number 50 we could just say.

more +50 h2g2.txt

This would take us to line number 50.

The most important interactive command is h or ? this displays a summary of interactive commands. You can forget all the other commands, just remember this one.

Most people i know use complain about more because they think it cannot skip backwords, however this possible with the b or ^B command. If you would like to know on which line number you are press =, this can give you a quick identification where you are.

Searching for patterns is probably the most import thing you would like todo, searching with more is really easy. Within interactive mode just type / and then the pattern you would like to search. So lets use our h2g2.txt file again.

more h2g2.txt

Now within interactive mode type

/wowbagger

This will scroll you down to the first occurrence of wowbagger. If you would like to search for the next occurrence just press the n command.

Ever happend that you where in the middle of a more session, searching for patterns etc, then all of a sudden you need to type a command in order to look something up. You can do a couple of things. Press q and do the command from the shell, loosing all your work within your more session. Open a new terminal, but if you are like me you probably all ready have a dozen of other terminals open. There is another alternative, say we would like to list the contents in the directory /etc/apache, within interactive mode just do the following

!ls /etc/apache

This will list the contents from the /etc/apache directory in your more session. Sometimes this leave you with the output of your command scrambled through your more text. You can fix this by doing a ^L, this will redraw the screen.

More is great however more is really primitive. That is why most people these days use less which we will get to right now.