Running GUI programs remotely

It used to be that you would set the DISPLAY variable
and use xhost to run GUI programs on remote machine
and send traffic to your desktop. It seems the modern
way is to use ssh tunneling for X11 traffic.

This is accomplished by simply running:

ssh -fY host program

E.g.:

ssh -fY user@hostname xclock

Internally ssh establishes a connection to remote machine,
starts a proxy server listening to port (e.g. 6010),
sets the display variable (e.g. DISPLAY=localhost:10.0),
and starts the requested program (e.g. xclock).
xclock sends all X11 traffic to the server spawned
by the ssh process, and forwards it to your desktop's X server.

Review of Java CSV libraries

There are many good CSV libraries for Java. Here are the the ones that are open-source and use commercial-friendly licenses like Apache or LGPL:
SuperCSV, OpenCSV, Flatpack, Java CSV

Surprisingly not many of them handle all the quirks of CSV format. A pretty good description of CSV format can be found here:
http://www.creativyst.com/Doc/Articles/CSV/CSV01.htm.
This page also includes an example of CSV data that few libraries will be able to handle:

John,Doe,120 jefferson st.,Riverside, NJ, 08075
Jack,McGinnis,220 hobo Av.,Phila, PA,09119
"John ""Da Man""",Repici,120 Jefferson St.,Riverside, NJ,08075
Stephen,Tyler,"7452 Terrace ""At the Plaza"" road",SomeTown,SD, 91234
,Blankman,,SomeTown, SD, 00298
"Joan ""the bone"", Anne",Jet,"9th, at Terrace plc",Desert City,CO,00123

Comparing the libraries based on functionality, i.e. the ability to handle all the quirks of CSV format, it looks like SuperCSV is the winner.

Super CSV
http://supercsv.sourceforge.net/
Apache 2.0 license

Blocking issues:
None

Minor issues:
Spaces are unconditionally removed from fields.
E.g. if you have a record that looks like this: "a, b, c"
you would get back fields like "a", "b", "c"
instead of "a", " b", " c"

Open CSV
http://opencsv.sourceforge.net/
Apache 2.0 license

Blocking issue:
If fields are quoted, and spaces are used
with delimiters, the resulting fields may include quotes.
E.g. if you have a record that looks like this:
a, "b ""c""", d
your second field may look like
"b "c"
instead of
b "c"


Flatpack
http://flatpack.sourceforge.net/
Apache 2.0 license

Blocking issue:
Unable to handle double quotes followed by separator. E.g.:
"Joan ""the bone"", Anne",Jet,"9th, at Terrace plc",Desert City,CO,00123

Minor issues: It uses just another logging library (slf4j)
Unusual configuration via XML
Steepest learning curve


Java CSV
http://sourceforge.net/projects/javacsv
LGPL

Blocking issues:
Doesn't understand delimiters embedded in quotes, e.g.
"a ""b"", c", "d"
will produce 3 fields instead of 2

Some disadvantages of using an embedded webserver

If using an embedded webserver it may be hard to fine-tune the response details.
The webserver takes the clues on what the response should look like,
based on request, and some of the APIs you called while making the response.
However it may be hard to control the specifics.

For example, let's say you receive a request where the client says it speaks
HTTP 1.1 and requests some data. If the length of the data you're sending
back is not known at the moment when you're starting to send the data,
the 1.1 webserver notices that since you aren't setting the content-length header
in the response you are likely to not know the length, and it goes automatically
into chunked-mode. Both Tomcat and Jetty behave this way. This is quite
inflexible. What if the intention was to reply with a HTTP 1.0 message without
specifying content-length header and without using chunked mode? It is not possible
in Tomcat. It is possible in Jetty by using a trick of adding
Connection: Close header.

These kinds of things make using an embedded webserver feel like
using a black box with knobs that cause unknown and strange effects.
"If you don't use Content-Length, we figure you must want chunked-mode.
But if you set Connection: Close, it looks like you don't want to use
chunked-mode."

The general problem is with a poorly designed abstraction layer that hides
the implementation to the point where it becomes hard to understand
what's going on. The benefits of using a library implementing the HTTP protocols
start to outweigh the disadvantages of not being able to tune specific responses.
A better library is designed in a way that empowers the user but doesn't take
away the flexibility.

How to send back nice REST errors using Tomcat

Tomcat uses "error report valves" which are essentially formatters that convert HTTP error codes and messages into browser-friendly HTML. So if the application calls HttpResponse.sendError(code, message) the error valve would convert the message to HTML, and that's what's going to be send back over the socket as the HTTP response.

That's probably the reasonable thing to do if the client is the web browser, and you want to see HTML in response to errors. What if the client is a program, and the content-type needs to be something other than text/html?

The solution is to write your own error report valve, such as this one that I wrote after looking at the default Tomcat one (org.apache.catalina.valves.ErrorReportValve).

public class MyErrorValve extends ErrorReportValve {
    @Override
    public void invoke(Request request, Response response) throws IOException, ServletException {
        try {
            getNext().invoke(request, response);
            response.setSuspended(false);
            Writer writer = response.getReporter();
            String message = response.getMessage();
            if (writer != null && message != null) {
                writer.write(message);
            } 

        } catch (Throwable e) {
            ...
        }
    }
}

This valve literally prints the error message to the response stream. So if you call HttpResponse.sendError(400, "That's a pretty bad request") what you'll get back from the server should be this:

HTTP/1.0 400 Bad Request

That's a pretty bad request

To install this valve, get ahold of the StandardHost, and make a call like this:

((StandardHost)host).setErrorReportValveClass(mypackage.MyErrorValve.class.getName());

Trapping errors in bash

Bash has functionality similar to exceptions in java.
If we want to handle any sort of error, we can add a handler function
and exit with an error code. This sure beats checking exit code
after every command.

function error {
echo Error
exit 1
}
trap error ERR

Ubuntu gnome session management

Ubuntu 8.04 gnome session manager is very unreliable and unintuitive. You can hardly accomplish anything using the UI. It is amusing that while one can think of it as a helper program to help you restore your programs between system restarts it can also be used to kill programs currently running on your computer, unexpectedly. The author clearly had no idea of how UI should be designed in general, nor a plan for what he was trying to get accomplished with this program specifically. It is the single worst piece of software included with Ubuntu. Oh well one of these days I'll write a better one and get it included with Ubuntu.

However you can partially turn off this useless software by clicking "Automatically remember..." in the Session Options Tab and instead edit the file ~/.gnome2/sessions by hand.

E.g. I added this program to get started automatically with my X session:
16,RestartCommand=/usr/bin/gvim /home/dmitri/Documents/Status.txt /home/dmitri/Misc/GTD.txt

Unfortunately, this displays a strange error message in VIM. The error message only has an error icon and the path to the file. However after I click on it, all the files are opened just fine, except the second one is opened twice. Oh well, it's still 2008 I guess: it's 2009 that will be the year of the Linux desktop.

Compensating for lack of unsigned types in Java

For some reason Java has no unsigned primitive types, which makes it really limiting when writing network code or doing bit manipulation.

One workaround is to cast to a longer-type and mask out significant bits.

E.g.:

byte b = getAByte(); // Get a byte from somewhere
int wrong = (int) b;
int right = (int) b & 0xff;

Int wrong will work most of the time, except when you have most-significant bit of the byte set. Because in Java most-significant bit holds the sign, the following happens:

byte b = getAByte(); // let's say this holds 0xff
int wrong = (int) b; // becomes -127 or -0x7f
int right = (int) b & 0xff; // becomes 255 or 0xff

In fact the Java compiler checks this so the following assignments would show compile errors:

byte wrong = 0xff;
byte wrong = 0x80;
byte right = 0x7f;

Getting complete Java exception stack trace

JDK Throwable.printStackTrace() function attempts to compact the stack trace of an exception by eliminating common parents of all stack elements. However it is often helpful to see the entire stack trace.

E.g. if using JDK you may write this function to get the stack trace of an exception as a String:

    public static String formatStackTrace(Throwable e) {
        StringWriter stringWriter = new StringWriter();
        PrintWriter printWriter = new PrintWriter(stringWriter);
        e.printStackTrace(printWriter);
        return stringWriter.toString();
    }

To obtain the complete stack trace you may use this function:

    private static String formatStackTrace(Throwable e) {
        StringBuffer str = new StringBuffer();
        str.append(e.getClass().getName()).append(": ");
        str.append(e.getMessage());
        StackTraceElement[] stack = e.getStackTrace();
        for (int i = 0; i < stack.length; i++) {
            StackTraceElement element = stack[i];
            str.append("\n\tat ").append(element.getClassName()).
                append(".").append(element.getMethodName()).
                append("(").append(element.getFileName()).append(":").
                append(element.getLineNumber()).append(")");
        }

        Throwable cause = e.getCause();
        if (cause != null) {
            str.append("\nCaused by: ").append(formatStackTrace(cause));
        }
        return str.toString();
    }

Script to reuse Eclipse project when switching code directory underneath

We're using subversion transactions here at Snaplogic: when working on a bug fix or new feature we create a branch off main tree, then after the change is complete we merge it into the main tree. Transaction management can be easily automated using shell scripts: tx-begin, tx-commit, etc.

However if you are using Eclipse there is a problem that each time you create a transaction you have to recreate your project. Even though you're working on the code, and have set up your project paths, and environment, and launch configurations, you have to recreate all that when you switch from one transaction to another, because each transaction resides in a different directory.

You want to have transactions in separate directories to be able to go back and forth between them, if you are working on several at a time, and for historical purposes. It may seem that you can overcome this limitation of Eclipse by using symlinks, and you can, to a degree, however Eclipse converts the symlink to the absolute file system path, and stores that in its project file.

So this is the missing piece of the puzzle of reusing the same Eclipse project when switching project directories underneath.

This python script will edit the .location file stored in the .metadata directory of Eclipse's resources plugin.

#!/usr/bin/python
# Pass destination path as the parameter
import sys

LOCATION = "/home/dmitri/workspace/.metadata/.plugins/org.eclipse.core.resources/.projects/code/.location"
PREFIX = "URI//file:"
replace_path = PREFIX + sys.argv[1]
file = open(LOCATION, "rb")
contents = file.read()
start = contents.index(PREFIX)
length = ord(contents[start-1:start])
sys.stdout.write(contents[0:start-2] + chr(0) + chr(len(replace_path)) + replace_path + contents[start + length:])

You run it as follows:

fix-eclipse /home/dmitri/workspace/all-code/dmitri_snaphome-2/snaplogic/python  > /tmp/.location
cp /tmp/.location /home/dmitri/workspace/.metadata/.plugins/org.eclipse.core.resources/.projects/code/.location

Fixing java disk swapping problem on linux

Running JDK 1.6 on my ubuntu hardy laptop I noticed hard disk drive light flashing every second. Disk activity was happening when running any java program, e.g. Eclipse, or Tomcat. I pinned this down to the JVM's performance files in the /tmp/hsperfdata directory. To turn off disk swapping, and prolong your battery and disk live, run JVM with the following option -XX:-UsePerfData

E.g. for Eclipse you may edit eclipse.ini and add this line at the end:
-XX:-UsePerfData

These were the steps I took to diagnose the issue:

# Become root
sudo su -

# Stop loggers
/etc/init.d/sysklogd stop
/etc/init.d/klogd stop

# Enable block I/O debugging
echo 1 > /proc/sys/vm/block_dump

# Monitor kernel ring buffer
while true; do dmesg -c; sleep 1; done;

...
Examine the output
...
[ 562.539286] java(10323): dirtied inode 8224835 (10310) on sda1
[ 562.539293] java(10323): dirtied inode 8224835 (10310) on sda1
[ 562.545651] eclipse(13917): dirtied inode 8224819 (13898) on sda1
[ 562.545658] eclipse(13917): dirtied inode 8224819 (13898) on sda1

# Find out filename from inode
find / -inum 8224819 2> /dev/null
/tmp/hsperfdata_dmitri/13898

# We're done
# Turn off block I/O debugging
echo 0 > /proc/sys/vm/block_dump

# Start loggers
/etc/init.d/sysklogd start
/etc/init.d/klogd start