Useful wireshark startup options

Sometime when looking at TCP/IP traffic on multiple ports it's useful to start up several wireshark instances and have each capture traffic on distinct port, and give the window a specific title:


sudo wireshark -o "gui.window_title:Data Server on $SERVER_PORT -" -i lo -p -f "port $SERVER_PORT" -k &


-i lo
capture packets on local loopback interface

-p
don't go into promiscuous mode

-f "port $SERVER_PORT"
use the following filter (specific port)

-k
start capture immediately

How to print process environment variables

Here is a script that prints process environment variables given process ID in Linux:


#!/bin/bash
if [ $# -ne 1 ] ; then
echo "Usage: `basename $0` process_id"
echo "Prints process environment variables"
exit 1
fi

sed 's/\o0/\n/g' /proc/$1/environ


Comparison of Python vs Java

Here is my experience using Python and Java for software development.

Overall, Python is best for small projects and rapid prototyping. Python has certain advantages with its dynamic nature, but as the code base grows these advantages become disadvantages. Java is best for larger projects and where performance is important. Python performance is poor: bytecode executes slow, and global interpreter lock prevents threads from completely utilizing multiple CPUs.

Python is pleasant to use with its expressive string operations, e.g. you can write 'a'*80 to make a string of 80 'a' characters. Using operators on collections in Python is so much more intuitive than using Java's method-based operations. It's nice to not have to compile files.

However, in Python you can make a typo, and it can be frustrating to catch those. Something that background compilation would catch right away in a Java IDE and highlight as syntax error in 1 second, takes writing unit-tests, getting back to the code you've written not at this minute, and fixing it. Because Python is so permissive you can easily do this:

  var = method
  for i in var:
    do something else

Note that you intended to write:

  var = method()  

You omitted the braces. Nothing flagged this as a possible error. During execution, the error isn't manifested on the same line, but instead a couple lines after that when we are trying to do
something with a method instead of an object.

Overall it seems that dynamic languages have inherent limitations that limit code discovery. From a practical point of view, modern languages now have huge libraries, and it's impossible for a human to remember even a fraction of library routine names and parameters. So here is where "suggest" or autocomplete functionality comes in nicely. However it's hard to implement in a dynamic language for the simple reason, that you don't know what types objects are. This impacts your productivity severely, as you have to go and check man pages over and over as opposed to having method names pop up as you write code.

In Python writing performant code is hard. Organizing your code into methods has significant performance penalty if these methods are frequently invoked. Python VM doesn't inline. Essentially Python performance trick is making sure you delegate to the C code. Doing work by executing Python code makes things slow. Because of GIL, Python threads don't truly run simultaneously. Instead it is simulated that they run simultaneously. So if you have multiple CPUs and multiple threads you won't utilize all your CPUs.

Click here for the table summarizing Python and Java.
For some reason Blogger puts a large space between the text and the table.

FEATURE	PYTHON	JAVA
Rapid prototyping	Yes	Less so (compile phase)
Suitable for large projects	Less so hard to refactor (dynamic language) unreliable debugger VM crashes	Yes refactoring support good debugging support
Scalability	Limited (runs only one thread at a time because of GIL)	Better threads run simultaneously
Performance	Slower VM doesn't inline methods Performance penalty for method calls.	Faster Straight bytecode execution about 3 times faster than Python.
Language features	Overall better collection operators e.g. brackets expressive string operations unchecked exceptions non-default constr.inheritance default arguments	Overall worse collections use via methods pluses over python: method overloading
Productivity: Can IDE catch typos?	No Typos become runtime errors Most of the time IDE cannot tell an error from valid use	Yes IDE can compile code in background and highlight typos.
Productivity: IDE: Refactoring, Suggestions, Autocomplete, Code analysis	Hardly possible because Python is dynamic	Yes This makes one write code fast. IDE can perform mathematically correct refactoring
Debugging PyDev vs Eclipse	Rudimental Hard to debug multi-threaded programs: cannot pause all threads Cannot break on exception Unreliable	Superior
Business-friendly	Less so Fewer options as many 3d party libraries tend to be GPL. You may have no option but open sourcing your work. Harder to close source, as normally open py files are distributed instead of compiled bytecode.	More so Many 3d party libraries under BSD, Apache, or LGPL licenses.

Running GUI programs remotely

It used to be that you would set the DISPLAY variable
and use xhost to run GUI programs on remote machine
and send traffic to your desktop. It seems the modern
way is to use ssh tunneling for X11 traffic.

This is accomplished by simply running:

ssh -fY host program

E.g.:

ssh -fY user@hostname xclock

Internally ssh establishes a connection to remote machine,
starts a proxy server listening to port (e.g. 6010),
sets the display variable (e.g. DISPLAY=localhost:10.0),
and starts the requested program (e.g. xclock).
xclock sends all X11 traffic to the server spawned
by the ssh process, and forwards it to your desktop's X server.

Review of Java CSV libraries

There are many good CSV libraries for Java. Here are the the ones that are open-source and use commercial-friendly licenses like Apache or LGPL:
SuperCSV, OpenCSV, Flatpack, Java CSV

Surprisingly not many of them handle all the quirks of CSV format. A pretty good description of CSV format can be found here:
http://www.creativyst.com/Doc/Articles/CSV/CSV01.htm.
This page also includes an example of CSV data that few libraries will be able to handle:

John,Doe,120 jefferson st.,Riverside, NJ, 08075
Jack,McGinnis,220 hobo Av.,Phila, PA,09119
"John ""Da Man""",Repici,120 Jefferson St.,Riverside, NJ,08075
Stephen,Tyler,"7452 Terrace ""At the Plaza"" road",SomeTown,SD, 91234
,Blankman,,SomeTown, SD, 00298
"Joan ""the bone"", Anne",Jet,"9th, at Terrace plc",Desert City,CO,00123

Comparing the libraries based on functionality, i.e. the ability to handle all the quirks of CSV format, it looks like SuperCSV is the winner.

Super CSV
http://supercsv.sourceforge.net/
Apache 2.0 license

Blocking issues:
None

Minor issues:
Spaces are unconditionally removed from fields.
E.g. if you have a record that looks like this: "a, b, c"
you would get back fields like "a", "b", "c"
instead of "a", " b", " c"

Open CSV
http://opencsv.sourceforge.net/
Apache 2.0 license

Blocking issue:
If fields are quoted, and spaces are used
with delimiters, the resulting fields may include quotes.
E.g. if you have a record that looks like this:
a, "b ""c""", d
your second field may look like
"b "c"
instead of
b "c"


Flatpack
http://flatpack.sourceforge.net/
Apache 2.0 license

Blocking issue:
Unable to handle double quotes followed by separator. E.g.:
"Joan ""the bone"", Anne",Jet,"9th, at Terrace plc",Desert City,CO,00123

Minor issues: It uses just another logging library (slf4j)
Unusual configuration via XML
Steepest learning curve


Java CSV
http://sourceforge.net/projects/javacsv
LGPL

Blocking issues:
Doesn't understand delimiters embedded in quotes, e.g.
"a ""b"", c", "d"
will produce 3 fields instead of 2

Some disadvantages of using an embedded webserver

If using an embedded webserver it may be hard to fine-tune the response details.
The webserver takes the clues on what the response should look like,
based on request, and some of the APIs you called while making the response.
However it may be hard to control the specifics.

For example, let's say you receive a request where the client says it speaks
HTTP 1.1 and requests some data. If the length of the data you're sending
back is not known at the moment when you're starting to send the data,
the 1.1 webserver notices that since you aren't setting the content-length header
in the response you are likely to not know the length, and it goes automatically
into chunked-mode. Both Tomcat and Jetty behave this way. This is quite
inflexible. What if the intention was to reply with a HTTP 1.0 message without
specifying content-length header and without using chunked mode? It is not possible
in Tomcat. It is possible in Jetty by using a trick of adding
Connection: Close header.

These kinds of things make using an embedded webserver feel like
using a black box with knobs that cause unknown and strange effects.
"If you don't use Content-Length, we figure you must want chunked-mode.
But if you set Connection: Close, it looks like you don't want to use
chunked-mode."

The general problem is with a poorly designed abstraction layer that hides
the implementation to the point where it becomes hard to understand
what's going on. The benefits of using a library implementing the HTTP protocols
start to outweigh the disadvantages of not being able to tune specific responses.
A better library is designed in a way that empowers the user but doesn't take
away the flexibility.

How to send back nice REST errors using Tomcat

Tomcat uses "error report valves" which are essentially formatters that convert HTTP error codes and messages into browser-friendly HTML. So if the application calls HttpResponse.sendError(code, message) the error valve would convert the message to HTML, and that's what's going to be send back over the socket as the HTTP response.

That's probably the reasonable thing to do if the client is the web browser, and you want to see HTML in response to errors. What if the client is a program, and the content-type needs to be something other than text/html?

The solution is to write your own error report valve, such as this one that I wrote after looking at the default Tomcat one (org.apache.catalina.valves.ErrorReportValve).

public class MyErrorValve extends ErrorReportValve {
    @Override
    public void invoke(Request request, Response response) throws IOException, ServletException {
        try {
            getNext().invoke(request, response);
            response.setSuspended(false);
            Writer writer = response.getReporter();
            String message = response.getMessage();
            if (writer != null && message != null) {
                writer.write(message);
            } 

        } catch (Throwable e) {
            ...
        }
    }
}

This valve literally prints the error message to the response stream. So if you call HttpResponse.sendError(400, "That's a pretty bad request") what you'll get back from the server should be this:

HTTP/1.0 400 Bad Request

That's a pretty bad request

To install this valve, get ahold of the StandardHost, and make a call like this:

((StandardHost)host).setErrorReportValveClass(mypackage.MyErrorValve.class.getName());

Trapping errors in bash

Bash has functionality similar to exceptions in java.
If we want to handle any sort of error, we can add a handler function
and exit with an error code. This sure beats checking exit code
after every command.

function error {
echo Error
exit 1
}
trap error ERR