Glen Peterson's Blog

Ubuntu 11.10 "Oneiric" with a Gnome 2 Feel

2012-01-09T08:59:00.000-08:00

I love, and am maybe even a little addicted to the packages Ubuntu includes in their distributions. But like many, I am dissatisfied with the new Unity interface because:

I can't open two copies of KeePassX at once
I hate using text search to find applications
It was too hard to create a custom application launcher

Hopefully all that will change, but until then, it's just not working for me.

I tried the LXDE desktop, but:

I didn't like the default software (Leafpad text editor, Galculator, LX terminal)
Desktop integration is lacking (no screen-shots, limited drag and drop from one application to another)

It's a very new distribution and may need a year or two to catch up. Certainly LXDE is worth keeping an eye on, but it's not there yet.

What's currently working for me is a "retro" basic Gnome session based on OMG Ubuntu's article, Make Ubuntu 11.10 Look and Feel Like GNOME 2. Thanks to "DigalMan" for pointing it out to me. That article has a typo though, it's gnome-session-fallback, not gnome-fallback-session.

Also, when they say, "Add ‘ppa:jconti/gnome3‘ to your Software Sources" their instructions don't include how to add the PPA's key, so that future updates are not be applied properly (or at all). Fortunately, launchpad.net has instructions. The key lines I needed were:

sudo add-apt-repository http://ppa.launchpad.net/jconti/gnome3/ubuntu
sudo apt-get update

It's not perfect (the top-bar color's not quite right and the icons space themselves wider apart when hovered over), but I'm basically back with the Ubuntu system I have loved for years, with the latest versions of all software. If this works in 12.04, I'll probably stick with it. If not, and LXDE has not improved by that time, I'll guess I'll try Mint.

Automatic POJO Generation From a Database

2011-09-14T06:07:00.000-07:00

Like JPA (or presumably Spring), Hibernate "reverse engineering" tools can generate POJOs (Plain Old Java Objects) from database tables and vice-versa. Generating database tables from Java code is probably best used as a one-time short-cut, suitable for rapid prototyping. Because everything in an application is dependent on the database (and not vice-versa), future changes must be made in the database first (and any existing data migrated there first as well), then propagated to all affected parts of the application.

I have found that anything which the "database reverse engineering" process does not generate for me breaks, usually sooner rather than later. Also that the hardest part of managing maintenance of a large system is the constant refactoring. To that end, I have developed 2 goals for the database reverse-engineering process:

It should keep my application in synch with any database changes, automatically updating as much of my application as is practical.
Where #1 is not possible, it should cause any affected areas of my application to generate a compile-time error.

Imagine a database table:

CREATE TABLE `user` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `last_modifier_id` bigint(20) unsigned DEFAULT NULL,
  `company_id` bigint(20) unsigned NOT NULL,
  `identifier` varchar(64) NOT NULL COMMENT 'PIN.  Unique within a company',
  `first_name_c` varchar(40) DEFAULT NULL,
etc.

Database generation tools normally generate the following sort of fields:

public class User implements java.io.Serializable {

private long id;
private User lastModifier;
private Company company;
private String identifier;
private String firstNameC;
etc.

public long getId() { return id; }
public void setId(long x) { id = x; }

public User getLastModifier() { return lastModifier; }
public void setLastModifier(User x) { lastModifier = x; }

public Company getCompany() { return company; }
public void setCompany(Company x) { company = x; }

public String getIdentifier() { return identifier; }
public void setIdentifier(String x) { identifier = x; }

public String getFirstNameC() { return firstNameC; }
public void setFirstNameC(String x) { firstNameC = x; }
etc.

I have modified the generator to also generate the following fields for each column in the table (prefixed with SQL_) and for each field in the Java object (prefixed with HQL_):

public static final String SQL_user = "user"; // table name
public static final String SQL_id = "id";
public static final String SQL_last_modifier_id = "last_modifier_id";
public static final String SQL_company_id = "company_id";
public static final String SQL_identifier = "identifier";
public static final String SQL_first_name_c = "first_name_c";
etc.

public static final String HQL_User = "User"; // class name
public static final String HQL_id = "id";
public static final String HQL_lastModifier = "lastModifier";
public static final String HQL_company = "company";
public static final String HQL_identifier = "identifier";
public static final String HQL_firstNameC = "firstNameC";
etc.

This allows me to write code like the following:

Crit<User> dupIdentCrit = Crit.create(User.class);
dupIdentCrit.add(Restrictions.eq(User.HQL_company, this));
dupIdentCrit.add(Restrictions.eq(User.HQL_identifier, identifier));
List<User> dupIdentifierUsers = dupIdentCrit.list();

Here's an example of building a SQL statement for use with JDBC:

StringBuilder sqlB = new StringBuilder("select * from " + User.SQL_user +
                                       " where " + User.SQL_company_id +
                                       " = ?");
if (!includeDeleted) {
    sqlB.append(" and " + User.SQL_is_deleted + " is false");
}
sqlB.append(" order by " + User.SQL_last_name_c + ", " +
            User.SQL_first_name_c + ", " +
            User.SQL_middle_name_c);

When the database changes and I rebuild my objects, I get compile-time errors, showing me every piece of code I need to fix.

I am working on modifying the generator to also generate the following for any varchar or char fields because their maximum length is defined in the database columns:

public static final int MAX_identifier = 64;
public static final int MAX_firstNameC = 40;
etc.

I am considering parsing comments on each column to look for something like: 'min: 6' and generate the following from it:

public static final int MIN_identifier = 6;

That same technique could be used with int columns to define minimum and maximum values which could be used in validation, in the GUI, and in the help of an appliction. I'm also experimenting with using the HQL_ tokens as the names of the input fields on the GUI screens.

I'm not sure how or even if this would work with stored procedures.

Special thanks to J. Michael Greata, whose interest and suggestions over the past several years have encouraged and shaped the development of this project. Also to Arash Tavakoli whose post in the LinkedIn Java Architects group inspired me to organize my thoughts into this posting.

Java Runtime Exceptions

2011-09-08T04:43:00.000-07:00

I've been enjoying, "Java: The Good Parts" by Jim Waldo and just finished chapter 3: Exceptions. At the end of the chapter, Mr. Waldo takes a humorously firm stance against RuntimeExceptions. Quick review: checked exceptions have to be declared in the method signature and dealt with by the calling code; RuntimeExceptions don't. Indeed, RuntimeExceptions can be evilly misused. Yet I believe there is a time and a place for these exceptions. Knowing when to use them requires an understanding of who is at fault for a particular problem.

The following code sample bit me the other day:

public static void write(String s) {
    try {
        out.write(s);
    } catch (IOException ioe) {
        throw new IllegalStateException();
    }
}

It is evil (as Mr. Waldo says) for several reasons:

This method is not being responsible for handling its own problems. IOExceptions happen for good reasons other than coding errors - users can delete files, network connections can close, etc. It could retry the write() or recover some other way. It could let the original exception bubble up to the caller, or it could report the error some other suitable way (e.g. to the user).
Wrapping a checked exception with a RuntimeException means that this method signature is missing vital information that the caller really needs to know about. It hides critical details (any problem with the write() now causes an unexpected exception).
Absentmindedly wrapping an exception with another exception serves only to complicate the stack trace - it further hides the cause of the problem. It's good to wrap an exception if you have critical information to add (e.g. the method handles two streams and you wrap an exception with the message of which stream it applies to). In the example above, nothing is added. In fact, this code doesn't wrap an exception, it throws it away and substitutes another which is even worse.
The IllegalStateException is blank, giving the unfortunate caller no idea what went wrong.

The following shows proper usage of RuntimeExceptions:

/** Creates or returns an existing immutable YearMonth object.
@param year a valid year
@param month a one-based month between 1-12
@return the relevant YearMonth object
*/
public static YearMonth valueOf(int year, int month) {
    if ( (month < 1) || (month > 12) ) {
        throw new IllegalArgumentException("Month must be a positive integer such that 0 < n < 13");
    }
...

This works because:

Valid input values are documented clearly.
The inputs are checked at the beginning of the method, before any processing is done.
It throws a RuntimeException to inform the caller that they have made a coding error.
The exception provides information about what the caller did wrong.

You wouldn't want to use a checked exception because it would force the responsible caller's code to check their input values twice:

YearMonth ym;
if (m > 12) {
    out.write("Month too big");
} else if (m < 1) {
    out.write("Month too small");
} else {
    try {
        ym = YearMonth.valueOf(y, m);
    } catch (Exception e) {
        out.println("month still invalid: " + e.getMessage());
    }
}

The critical nuance here is that RuntimeExceptions should be used to indicate a programming error on the part of the caller of the function that throws it. They are generally used in the first few lines of the function to check for invalid input values. Each RuntimeException should include a description of the problem (not be blank).

RuntimeExceptions can also be used to catch invalid state as follows:

enum Numb {
    ONE,
    TWO;
}

private Numb num = null;

public void init(Numb n) {
    if (n == null) {
        throw new IllegalArgumentException("init cannot take a null Numb");
    }
    num = n;
}

public void showNum() {
    if (ONE == num) {
        out.println("1");
    } else if (TWO == num) {
        out.println("2");
    } else {
         throw new IllegalStateException("Unhandled value of Numb or called showNum without initializing the num");
    }
}

When some programmer adds THREE to Numb and doesn't account for the new possible value, it fails hard and fast, making the problem easy to find and fix.

The Java compiler and a good IDE can catch many errors for us, but as a second or third line of defense, RuntimeExceptions are a good way to make code fail hard and fast without forcing the caller to check their input values twice. Used properly they can make sneaky coding errors obvious. Used improperly, they can make and obvious errors sneaky.

Object Mutability

2011-08-31T05:55:00.000-07:00

I've read a lot lately about making objects immutable whenever possible. "Programming in Scala" lists Immutable Object Tradeoffs as follows:

Advantages of Immutable Objects

Often easier to reason about because they do not have complex state.

Can be passed freely (without making defensive copies) to things that might try to modify them

Impossible for two threads accessing the same immutable object to corrupt it.

They make safe Hashtable keys (if you put a mutable object in a Hashtable, then change it in a way that changes its hashcode, the Hashtable will no longer be able to use it as a key because it will look for that object in the wrong bucket and not find it).

Disadvantages

Sometimes require a large object graph to be copied (to create a new, modified version of the object). This can cause performance and garbage collection bottlenecks.

For most purposes, an object representing a month can be made immutable - February 2003 will never become anything other than what it is. But a User record is not immutable. People get married or change their name for other reasons. Phone numbers, addresses, hair color, height, weight, and virtually every other aspect of a person can change. Yet the person is still the same person. This is what surrogate keys model in a database - that everything about a record can change, yet it can still be meaningfully the same record.

In order to use an object in a hash-backed Collection (in Java), its hashcode must NOT change. The simplest way to accomplish this is to make the hashcode of a mutable persistent object its surrogate key and to use that key as a primary comparison in the equals method as well (see my older post on Implementing equals(), hashcode(), and compareTo()).

To make an immutable object, you sometimes need a mutable constructor object, like StringBuilder and String. StringBuilder allows you to change your object as many times as you want, then get an immutable version by calling toString(). This is clean and safe, but has some small costs in time and memory (transforming the immutable StringBuilder into a new immutable String object, then throwing away the StringBuilder). An alternative that I have not seen much is to create an immutable interface, extend a mutable interface from it, and then extend your object from that.

Here's an example based on java.util.List. Pretend each interface or class is in its own file:

// All the immutable-friendly methods from java.util.List.
// Interfaces like these could easily be retrofitted into
// the existing Java collections framework
public interface ImmutableList {
    int size();
    boolean isEmpty();
    boolean contains(Object o);
    Iterator<E> iterator();
    Object[] toArray();
    <T> T[] toArray(T[] a);
    boolean containsAll(Collection<?> c);
    boolean equals(Object o);
    int hashCode();
    E get(int index);
    int indexOf(Object o);
    int lastIndexOf(Object o);
    ListIterator<E> listIterator();
    ListIterator<E> listIterator(int index);
    List<E> subList(int fromIndex, int toIndex);
}

// This interface adds the mutators
public interface java.util.List extends ImmutableList {
    boolean add(E e);
    boolean remove(Object o);
    boolean addAll(Collection<? extends E> c);
    boolean addAll(int index, Collection<? extends E> c);
    boolean removeAll(Collection<?> c);
    boolean retainAll(Collection<?> c);
    void clear();
    E set(int index, E element);
    void add(int index, E element);
    E remove(int index);
}

public class java.util.ArrayList implements List {
    // just as it is now...
}

public class MyClass {
    someMethod(ImmutableList<String> ils) {
        // can't change the list
    }

    public static void main(String[] args) {
        List<String> myStrings = new ArrayList<String>();
        myStrings.add("hello");
        myStrings.add("world");

        someMethod(myStrings);

        // Totally safe:
        System.out.println(myStrings.get(1));
    }
}

This doesn't solve the problem of passing a list to existing untrusted code that might try to change it. It also doesn't prevent the calling code from modifying myStrings from a separate thread while someMethod is working on it. But it does provide a way (going forward) for a method like someMethod() to declare that it cannot modify the list. The programmer of someMethod() cannot compile her code if she tries to modify the list (well, short of using reflection).

Guaranteed immutability can be critical in writing concurrent code and for keys in hashtables. Not objects can be made immutable, but many of those objects have immutable surrogate keys that, if used properly, work around the pitfalls of mutability. Limiting mutability and avoiding common mutable object pitfalls can lead to fewer bugs, easier readability, and improved maintainability.

Typesafe List in Java 5+ part two...

2011-08-25T06:00:00.000-07:00

Looks like John's comment yesterday had us both up half the night working on the same thing. I'm posting my version here because it has a parameter that lets the caller determine the trade-off between speed and safety. When debugging, you can use Check.ALL (or leave it null). In performance-critical code, you can later set it to NONE. Check.FIRST is for situations where you just a sanity check - it's fast and better than nothing.

I've also included some test code and a routine that takes advantage of the String.valueOf() methods to coerce each element of the original list to a String, if necessary, much the way that many dynamically typed languages do. I wonder if it would be worthwhile to further optimize this method for long lists by dividing the original list into runs of items that do not need casting and copying those runs with stringList.addAll(list.subList(fromIdx, toIdx)) instead of copying one item at a time?

John's solution uses B.class.isAssignableFrom(a.getClass()) while mine uses a instanceof B. I think either works in this case because a List can only hold objects, not primitives. If we were dealing with primitives, John's solution would handle more cases than mine. Not sure if there are performance considerations, but I doubt they would be significant.

import java.util.ArrayList;
import java.util.List;

public class TypeSafe {

    public enum Check {
        NONE,
        FIRST,
        ALL;
    }

    @SuppressWarnings({"unchecked"})
    public static <T> List<T> typeSafeList(List list,
                                           Class<T> clazz,
                                           Check safety) {
        if (clazz == null) {
            throw new IllegalArgumentException("typeSafeList() requires a non-null class parameter");
        }
        
        // Should we perform any checks?
        if (safety != Check.NONE) {
            if ( (list != null) && (list.size() > 0) ) {
                for (Object item : list) {
                    if ( (item != null) &&
                         !(item instanceof String) ) {
                        throw new ClassCastException(
                                "List contained a(n) " +
                                item.getClass().getCanonicalName() +
                                " which is not a(n) " +
                                clazz.getCanonicalName());
                    }
                    // Should we stop on first success?
                    if (safety == Check.FIRST) {
                        break;
                    }
                    // Default (Check.ALL) checks every item in the list.
                }
            }
        } // end if perform any checks
        return (List<T>) list;
    } // end typeSafeList()

    public static List<String> coerceToStringList(List list) {
        if (list == null) {
            return null;
        }
        try {
            // Return the old list if it's already safe
            return typeSafeList(list, String.class, Check.ALL);
        } catch (ClassCastException cce) {
            // Old list is not safe.  Make new one.
            List<String> stringList = new ArrayList<String>();

            for (Object item : list) {
                if (item == null) {
                    stringList.add(null);
                } else if (item instanceof String) {
                    stringList.add((String) item);
                } else {
                    // If this throws a classCastException, so be it.
                    stringList.add(String.valueOf(item));
                }
            }
            return stringList;
        }
    } // end typeSafeList()

    @SuppressWarnings({"unchecked"})
    private static List makeTestList() {
        List l = new ArrayList();
        l.add("Hello");
        l.add(null);
        l.add(new Integer(3));
        l.add("world");
        return l;
    } // end makeTestList()

    public static void main(String[] args) {
        List unsafeList = makeTestList();
        List<String> stringList = coerceToStringList(unsafeList);

        System.out.println("Coerced strings:");
        for (String s : stringList) {
            System.out.println(s);
        }

        List<String> safeList = typeSafeList(unsafeList,
                                             String.class,
                                             Check.ALL);

        System.out.println("Safe-casted list:");
        for (String s : safeList) {
            System.out.println(s);
        }
    } // end main()
}

Checking an Unchecked Cast in Java 5 or Later

2011-08-24T08:13:00.000-07:00

If query.list() returns an untyped List, then the following code will produce an ugly warning: "Unchecked Cast: 'java.util.List' to 'java.util.List<String>'

List<String> tags = null;

try {

    ...

    tags = (List<String>) query.list();

    ...

Unfortunately, you can only use @SuppressWarnings on a declaration. I don't want to add this annotation to my method declaration because my method does a lot more than this one cast and this is a useful warning to detect in the rest of the method. One solution is to add a new variable to make a declaration to use @SuppressWarnings on:

List<String> tags = null;

try {

    ...

    @SuppressWarnings("unchecked")

    List<String> tempTags = (List<String> query.list();

    tags = tempTags;

    ...

Now it produces an inspection warning in IDEA 10.5, "Local variable tempTags is redundant." I should probably just ignore this warning or turn it off altogether, but I wasn't quite comfortable with that either. In situations like these, I naturally turn to Joshua Bloch for guidance and his item #77 - Eliminate Unchecked Warnings has this advice in bold:

If you can't eliminate a warning, and you can prove that the code that provoked the warning is typesafe, then (and only then) suppress the warning with an @SuppressWarnings("unchecked") annotation.

I could probably prove that my query is only going to return Strings, but it occurred to me that there is a more general way to satisfy all the above requirements:

@SuppressWarnings({"unchecked"})

private List<String> getListOfStringsFromList(List list) {

    if ( (list != null) && (list.size() > 0) ) {

        for (Object item : list) {

            if ( (item != null) && !(item instanceof String) ) {

                throw new ClassCastException("List contained non-strings Elements: " + item.getClass().getCanonicalName());

            }

        }

    }

    return (List<String>) list;

}

The method does nothing but make a typesafe cast on a list, so I can @SuppressWarnings on the whole thing. It throws an undeclared runtime ClassCastException, but only if the caller makes a programming error and this exception would get thrown anyway wherever the list was eventually used if I didn't throw it here. In short, this method takes a programming error that cannot be caught by the compiler and does the next best thing: fail-fast, making the error easy to find.

Using the above solution hides all that complexity and preserves the intent of the original line of code:

List<String> tags = null;

try {

    ...

    tags = getListOfStringsFromList(query.list());

    ...

I thought this was interesting enough to post. Well, at least interesting enough for people who are considering a Java logo tattoo.

Question about MySQL Update speed

2011-03-09T06:24:00.000-08:00

I originally posted this as a question, but Eric Wood helped me solve it in email, so I've added the solution below.

The minimum time MySQL with InnoDB tables takes to do an update on a 3Ghz Core 2 Duo runing 64-bit Ubuntu 10.10 is somewhere around 0.06 seconds, though I wonder if hard drive speed could be the gating factor? My average time using Hibernate is 0.15 seconds. I think JDBC would approach 0.06 seconds. This was done using the mysql command line client. Any thoughts would be appreciated. Also timings vs. other databases if you have a sense.

mysql> CREATE TABLE `test` (

    ->   `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,

    ->   `last_click_date` datetime DEFAULT NULL,

    ->   `is_active` bit(1) NOT NULL DEFAULT b'0' COMMENT 'True if user is still logged in.',

    ->   PRIMARY KEY (`id`),

    ->   UNIQUE KEY `id` (`id`)

    -> ) ENGINE=InnoDB;

Query OK, 0 rows affected (0.21 sec)



mysql> insert into test (last_click_date, is_active) values ('2011-03-07 14:18:16', b'1');

Query OK, 1 row affected (0.11 sec)



mysql> insert into test (last_click_date, is_active) values ('2011-03-07 14:18:16', b'1');

Query OK, 1 row affected (0.11 sec)



mysql> insert into test (last_click_date, is_active) values ('2011-03-07 14:18:16', b'1');

Query OK, 1 row affected (0.06 sec)



mysql> update test set last_click_date = '2011-03-07 14:18:16' where id = 1;

Query OK, 0 rows affected (0.07 sec)

Rows matched: 1  Changed: 0  Warnings: 0



mysql> update test set last_click_date = now() where id = 1;

Query OK, 1 row affected (0.06 sec)

Rows matched: 1  Changed: 1  Warnings: 0



mysql> update test set is_active = b'0' where id = 1;

Query OK, 1 row affected (0.07 sec)

Rows matched: 1  Changed: 1  Warnings: 0



mysql> update test set is_active = b'0' where id = 1;

Query OK, 0 rows affected (0.02 sec)

Rows matched: 1  Changed: 0  Warnings: 0



mysql> update test set is_active = b'0' where id = 1;

Query OK, 0 rows affected (0.10 sec)

Rows matched: 1  Changed: 0  Warnings: 0



mysql> update test set last_click_date = 20110307141816 where id = 1;

Query OK, 0 rows affected (0.06 sec)

Rows matched: 1  Changed: 0  Warnings: 0

It turns out that if I change the table to use the MyISAM engine, all the updates take 0.00 seconds.

Top 10 Things a Password Manager Must Have

2011-01-06T08:06:00.000-08:00

Maybe 1 in 60 of my accounts reports their passwords stolen every year. For every site that reports a break-in, a few others are probably broken into and don't know it or don't report it. I would guess that if you have accounts at 30 different sites, you should probably assume that one of them gets broken into every year. You can't stop people from discovering passwords this way, but if you use a unique, strong password, you can contain the damage so that a hacker cannot leverage the knowledge of one of your passwords to break into your other accounts.

I just watched How to choose a strong password and while that's good advice, most people can neither remember nor type a good password, or at least not more than one or two good passwords. The only practical way to use a unique, strong password for every site is to use a good password manager. As such, I'm proposing a Password Manager Feature Manifesto for people to use to compare password managers and decide which one is best for them.

Password Manager Feature Manifesto

A password manager needs to do certain things to be worthwhile:

1.) Store passwords securely, in one place so you can find them, change them, secure them as a unit. It always seemed to me that storing your passwords in your browser was a little bit like taping your wallet to the outside of your front door - you are putting your valuables in the most vulnerable place. KeePass (without any plug-ins) is completely separate from your browser. Browser integration is not necessarily bad, but I think it loose some points from a security perspective. In any case, the passwords must be secured by a strong master password and encrypted on disk (and maybe in memory when possible too).

2.) Generate random passwords - people don't manually make strong passwords. Collecting entropy for the randomness is a huge plus. KeePass and LastPass both generate strong passwords for you.

3.) Make it equally easy to store your credit-card number or license key for Photoshop as to store a password to a web site.

4.) Must be backed up every time you make a change. LastPass has this built-in. KeePass must be used with Dropbox and set to save automatically after every change.

5.) To be shared between multiple computers, e.g. LastPass or KeyPass/Dropbox

6.) Needs to be relatively easy to use

7.) To work on all major operating systems (Windows, OS-X, Linux). I look for this every time I choose software. I hate being tied to one vendor's operating system or browser.

If a password manager doesn't do all of those things, I'm not really
interested in it. One thing that's not important yet, but I bet it
will become critical for most people in the next few years:

8.) To work on your phone or other mobile device. Here is where
LastPass may move ahead of KeePass.

9.) Popular OpenSource software is recommended for security

And finally, not critical, but the icing on the cake:

10.) It's free, or at least a reasonable price.

That leaves KeePass the clear #1 for me and LastPass #2. LastPass could threaten KeePass if they keep improving on #6 and #8 - specifically, it is very hard to log into sites with LastPass that have the user ID on one screen and password on the next.

Sadly, no password manager can remember your operating-system login when you boot up your computer, so you have to remember that password yourself. Also, the master password for your manager. But for most people that's just 2 passwords to remember and type, and that's fairly do-able.

I have to thank DigitalMan for his contributions to this article by talking about this with me and sending articles about break-ins, security, and passwords for the past few years, and for encouraging me to improve my own password practices.

Using Java Collections Effectively by Implementing equals(), hashCode(), and compareTo()

2010-12-29T06:00:00.000-08:00

Thanks to attending the local Java Users Group "Oracle® Certified Java™ Programmer Certification Boot Camp" on and off for several years, I've entered the 21st century and begun using Java Collections the way they were intended. Especially with regard to sorting and eliminating duplicate records, these classes are a god-send in terms of coding-time and run-time efficiency and arguably in terms of readability as well. But bug-free usage of these classes requires strict adherence to the contract of equals(), hashCode(), and often compareTo().

Even though popular IDEs automatically generate stubs of some of these methods for you, you should still understand how they work, particularly how the three methods work together because I don't see many IDE's writing meaningful compareTo() methods yet. Here's what I've learned in the past few years of implementing these methods. For much of what follows, I am endebted to Joshua Bloch and his book, Effective Java. Buy it, read it, live it.

In short, make the behavior of equals(), hashCode(), and compareTo() consistent. I'll cover each in order.

UPDATE 2011-08-31: As much as possible, you must base these methods on immutable fields. If you store an object in a collection (e.g. as a key in a HashMap) and it's hashcode changes, you probably won't be able to retrieve it from that hash table anymore! Thanks to "Programming in Scala" Section 30.2 p. 657. See also my later post on Object Mutability. END-UPDATE

equals()

a.equals(b) should return true when a and b represent the same object. Not that a and b are both pigs, or even that they are both pigs named Wilbur, but that they both represent the pig named Wilbur in the story, Charlotte's Web. Bloch (Item 8) says that the equals() method must be reflexive, symmetric, transitive and a few other things as well which I won't cover here. For any non-null value:

x.equals(x) must be true.
If x.equals(y) then y.equals(x) must be true.
If x.equals(y) and y.equals(z) then x.equals(z) must also be true.

The following should get you off to a good start in writing an equals method that is all of the above.

@Override
public boolean equals(Object other) {
    // First, the obvious...
    if (this == other) {
        return true;
    }
    if (other == null) {
         return false;
    }
    if ( !(other instanceof MyClass) ) {
        return false;
    }
    // Details...
    final MyClass that = (MyClass) other;

    // If this is a database object and both have the same surrogate key (id),
    // they are the same.
    if ( (this.getId() != 0) && (that.getId() != 0) ) {
        return (this.getId() == that.getId());
    }

    // If this is not a database object, compare "significant" field here.
    try {
        if ( this.item.equals(that.item) &&
             this.startDate.equals(that.startDate) &&
             this.endDate.equals(that.endDate) ) {
            return true;
        }
    } catch (Exception e) {
        // You may not want to catch exceptions here if that exception
        // indicates an invalid object.  See below.
    }

    // Not proven equal
    return false;
}

Be careful not to compare objects that may be invalid. Consider a rental system where no two people can have overlapping date values for the same rental item, but a user enters the wrong date values and the system builds an object with those values to compare it against the existing objects from the database (to check the validity). Is the new object the same as the database object whose dates it conflicts with? You can't make that comparison when one of the objects is invalid. The coding error here is not that the equals() method is flawed, but rather that you are trying to compare an object before it is valid.

Your equals() method should either compare "significant" fields (the ones that uniquely identify the object) OR surrogate keys - not both! The danger of providing a field-by-field equals comparison for a database object is that it will work in some cases with invalid objects, but it will not always work, thus I'd rather have it fail fast, than leave me scratching my head when an intermittent bug crops up in production. For database objects I always use surrogate keys because doing so acknowledges that everything about an object can change over time, yet it is still essentially the same object (The Artist Formerly Known as Prince). For non-database objects, you must compare the fields that uniquely identify a valid object.

hashCode()

Bloch's Item 9 states, "Always override hashCode() when you override equals()". Giving your hashCode() function an even distribution of values makes the Hashtable-based collections work better. The following are specifically required (see: Object.hashCode()):

x.hashCode() must always equal y.hashCode() when x.equals(y). If you break this nothing will work.
It's OK for x.hashCode() to equal y.hashCode() when x.equals(y) is false, but it's good to minimize this.

Once again, surrogate keys can come to your rescue here. Truncating the row number from the database from a long to an int is a great way to ensure equal distribution of hash buckets in a hash table. Until you have 2 billion records (2^31 for a signed 32-bit int), they each get their own hash code. Up to about 4 billion records, you only have 2 records in each hash bucket, and on and on as you add records. If you don't use surrogate keys, you need to construct an int from the same "significant" fields you used in the above equals method to keep them consistent. Good luck with that. Here's the simple solution:

@Override
public int hashCode() {
    if (this.getId() == 0) {
        return System.identityHashCode(this);
    } else {
        // return (possibly truncated) surrogate key
        return (int) this.getId();
    }
}

If you don't have a surrogate key, sometimes you can fit the other fields into a single int. For instance, in a date class, you might use the following which gives each day a unique, even human-readable integer representation:return (this.year * 10000) + (this.month * 100) + this.day;For a class with a lot of booleans you can/should represent them all losslessly in the hashcode:

int ret = (isHappy ? 1 : 0); // 2^0 = 1 = b0001
if (isHungry) {
    ret |= 2; // 2^1 = 2 = b0010
}
if (isSleepy) {
    ret |= 4; // 2^2 = 4 = b0100
}
if (isAntsy) {
    ret |= 8; // 2^3 = 8 = b1000
}
return ret;

compareTo()

Bloch's Item 12 states that if you are writing a class with an obvious natural ordering (alphabetical, numerical, chronological, etc.) consider implementing Comparable. I have personally gotten into the most trouble when my compareTo() method was not 100% compatible with my equals() method. So I added one rule (and it's converse) to Bloch's others:

If this.equals(that) == true, then this.compareTo(that) must return 0.
If this.equals(that) == false, then this.compareTo(that) must NOT return 0.

The example below should be fairly self-explanatory and sorts alphabetically by the description field.

/**
 Returns a negative integer, zero, or a positive integer as this object is
 less than, equal to, or greater than the specified object (respectively).

 @param that the object to compare to ("that")

 @return -1 means they are in order (this comes before that), 0 means they are equal
 (and Sets will eliminate one of them), +1 means to swap them (that comes
 before this)

 */
@Override
public int compareTo(MyClass that) {

    // ensures consistency with equals()
    if (this.equals(that)) {
        return 0;
    }

    // compare parents first in a hierarchical structure.
    // If this class can be it's own parent (e.g. a self-joining
    // table) this is much more complicated.  I'll have to
    // make another article on that.  You may also have to
    // check for a null parent, but here's the simplest case:
    int ret = this.getParent().compareTo(that.getParent());
    if (ret != 0) {
        return ret;
    }

    if (this.getDescription() == null) {
        if (that.getDescription() != null) {
            // sorts nulls first
            return -1;
        }
    } else if (that.getDescription() == null) {
        // sorts nulls first
        return 1;
    } else {
        // For each test, check and only return a non-zero result
        ret = this.getDescription().compareToIgnoreCase(that.getDescription());
        if (ret != 0) {
            return ret;
        }
        ret = this.getDescription().compareTo(that.getDescription());
    }

    if (ret != 0) {
        return ret;
    }

    // If you can't tell, return -1 which means that they are in the right
    // order but unequal - ensures consistency with equals()
    return -1;
}

Note: I have not verified this, but it stands to reason that if you change hashCode() and your object might be used in a Set or Hashtable, you probably need to change SerialVersionUID. Since a set only calls equals() for objects in the same bucket (ones that have similar hashcodes), you may end up with two of the same objects in the set (one with the old hashCode and one with the new). I'm not sure if this can happen in practice or not. Maybe someone will post test code in the comments that proves it one way or the other?

Software Development has only One Metric that Matters

2010-12-12T07:27:00.000-08:00

Having read and thoroughly enjoyed More Joel on Software I bought myself Joel on Software this week and find it to be similarly wonderful. Both books are basically just hard-copies of his blog and make for entertaining reading even though they are packed with knowledge from decades of successful software development.

Joel's Measurement article from 2002 is not his best article, but reading Joel helped me crystallize some vague notions that have been bumping around my head for years. The aspects of software that are easiest to measure are generally the least valuable measurements. For instance: lines of code. The more lines of code, generally the worse your software is; it's bloated and complicated. In general, the fewer lines of code for the same functionality, the better, though taken to an extreme, you can make something completely illegible and impossible to change without throwing it out and starting over. How many lines of code are appropriate for the problem you are trying to solve?

Similarly, increasing complexity usually makes a product buggy, unusable, or both. But decreasing complexity, taken to an extreme, can make a product useless (it doesn't do what it needs to). But how do you measure the level of complexity that is "just right" for the problem you are trying to solve? Bugs is an interesting metric because even though fewer bugs is better, the Heisenberg principle comes into play such that there's no way of measuring bugs without skewing your results. Scott Adams sums it up beautifully:
http://www.joeindie.com/images/dilbert-minivan.gif

But there is one metric that combines and trumps all others in just about every meaningful way: Customer satisfaction. Does the software solve the real-world problem it was intended to for the people who need it most? That's the only thing that matters. Not cyclomatic complexity, efferent coupling, or any other measurement that a computer can make on the code directly. It has meet someone's needs.

I recently saw Objectified which was an interesting film. But I didn't know whether to laugh out loud or stare in horror at one artist who made a robot that required human attention to do what it needed. The clip shows some woman dressed like a flight attendant leaning down so that this thing could whisper in her ear so that she would know to move it to the other side of the room. This is exactly how much of our software fails. The technology we create is supposed to make our lives easier, better, more enriching. Not make us it's slave.

How often is the new version of a product a step backward from the old? I remember one person I worked with actually advertising that with the new version of his architectural component, what used to take you one click now takes you several and makes you wait longer. How is that supposed to be a good thing? It's slower and more difficult... why?

Rackspace is on to something, realizing that what you get from a hosting company is not servERS, but servICE. Anyone can set up a few servers. But the first time your server goes down and your hosting company doesn't or can't respond, you realize that the service is what counts. Maybe I'm pushing this a little too far here, but I think software development has more in common with a hosting company than a discount store. That meeting the customer's needs, providing excellent SERVICE is more important than the implementation details of the product. The software is more like an extension of that service (it serves the customer instead of a human serving the customer) than like a shrink-wrapped product.

Providing an effective autonomous electronic servant means understanding the customer's need and designing something that meets that need, then communicating that understanding to the people who actually have to build the software. Get them excited about, or at least involved in solving the customer's actual problem instead of just thinking about some architectural detail or slavishly following a spec.

Obviously, there are pitfalls. In The Iceberg Secret Revealed, Joel says that "Customers don't know what they want." And it's true. In Make Users Happy by Ignoring Requirements I discuss what I should have called the "Excel Syndrome" where users describe the problem as if Excel were the solution. It's not. If it were, they would have made a spreadsheet instead of hiring you.

One last thing... When I say, "customer" I don't mean just the people your company serves. I mean the target audience for your software, which may be inside your company instead of outside it. When I worked for Fidelity, I worked for a little group called FMTC (now Pyramis) that handled retirement plans for large organizations. I think the minimum amount to open an account was over a million dollars. After years of working on the "customer-facing web-site" I learned that the primary users of the site were a handful of customer service people within Fidelity. Customers would call them up, ask a question, and the internal rep would use the web site to find the answer. Had we known this up front, we might have designed it very differently. That was years ago and most people are comfortable logging in and accessing their own account nowadays, but if you are in charge of billions of dollars, you may still have your secretary call the investment company and hand you the phone to get the answer to your question verbally. No password, no logging in, just "Yes Mr. Big-Wig. It's at 42 billion and change Mr. Big-Wig. I'd be happy to explain that for you..."

In short:
1.) Find out the real need.
2.) Meet it.
3.) Measure your success by asking your customers.
4.) Do it better next time (PDCA).

Disposing of Computers Responsibly

2010-04-20T05:53:00.000-07:00

You don't fall for con games, you use a router with WPA2 wireless, store your passwords securely, keep your operating system and applications updated. You're safe, right?

That depends on how you dispose of old hard drives and computers. If you run Windows, Microsoft recommends cleaning the hard drive with the secure delete command-line application, sdelete. Whatever operating system you run, you may prefer to Use an Ubuntu Live CD to securely wipe your PC's hard drive.

The Frontline video, Ghana: Digital Dumping Ground not only exposes the human health and environmental costs of disposing of computers, but about 8 minutes in, shows data recovery on those same computers that should make the hair on the back of your neck stand up.

One of the reasons I got into computer programming is that it eliminates paper, saves trees, and eliminates paper waste. Electronic devices require electricity, but more importantly, they break or become obsolete very quickly and we replace them. The impact of our obsession with newer, faster computers and other electronics is astounding. But even if we think we are recycling, we might be creating new waste problems in far-away lands as the 60 Minutes Electronic Wasteland video shows (also shows data recovery).

Anyone who reads this blog knows that when my computer became unacceptably slow running Microsoft Windows XP about 9 months ago and I switched to Ubuntu Linux and have been delighted with that decision. File operations that took hours on Windows take minutes on Linux. Everything else seems to run about twice as fast. So if you care about the impact of disposing of your computer (of if you want to save money), install Linux and keep your hardware twice as long. My laptop was bought in 1999 and I run Xubuntu on it (a lightweight version of Ubuntu). It's great for email and web browsing and with a little patience, I can even run a database and a web server on it, even though it only has 500MB memory a 500Ghz processor, and an incredibly slow disk drive.

Thanks to my occasional involvement with UCLUG, I learned of Free Linux PC. They are a fantastic organization that takes donations of old computers, installs Linux on them, and gives them to people who don't have a computer! I've given them a laptop and some memory and I've volunteered at a giveaway at the Greenville Public Library which was a great time. The recipients were pinching themselves.

Here's an old-fashioned recipe for improving environmental, human, and security impacts of your computer usage:

Reduce

Buy new computers (cell phones, TVs, etc) less often. Use your old ones longer.

Reuse

Installing Linux can double the useful life of a Windows computer. If that's not for you, then donate any still-working equipment to FreeLinuxPC and they will do it for you.

Recycle

Recycling sometimes works and is probably better than burning your computer on your front lawn (though it might still be burned in Ghana, Taiwan, or China), certainly better than throwing it in the trash. Even if you don't recycle, if you can find recycled gold watches or other products made from recycled computers, you are supporting an industry that badly needs encouragement.

In any case, make sure to wipe the hard drive before disposing of it!

POST vs. GET for HTML Form Security (and The Back Button)

2010-04-15T04:35:00.000-07:00

At the last GreenJUG meeting, we talked about how you should always use POST instead of GET for any secure web sites. But when you use a POST form on an HTTPS site, navigate away, and click the Back button (on Internet Explorer), you get "The web page you requested is no longer available - try refreshing..." It essentially breaks the back button which UI designers will tell you never to do.

So for me, the rule is, for any view-forms (where the form sends selection criteria for what the user wants to view), make them GET. Such forms tend to submit only the ID number of whatever they are pulling from the database, usually do not involve entering sensitive information, and should work when the users navigates away and uses the back-button. For update forms (where the user submits new data or changes old data) make them POST because it's good to prevent accidental resubmittion and because people tend to enter private and proprietary information on such forms.

Passwords Don't Matter

2010-04-13T07:40:00.000-07:00

Why Passwords Don't Matter

I set out to write an article about tools to easily manage passwords securely, but when I looked for data on computer crime to encourage people to use better passwords, I discovered a very different story. Most "computer crime" (according to the FBI) is various forms of scams and con games that used to be carried out in person, over the phone, or through the mail, but are now done through online auctions or email. Nothing to do with passwords. This 25-page 2008 Internet Crime Report by the FBI only uses the word "password" twice.

At least for corporations, the big problem seems to be people using the access they were given to do bad things. That happens much more often than people hacking into other accounts.

Computer attacks tend to target applications and the operating system. If you don't keep up with patches, your password won't matter. Source: The Top Cyber Security Risks.

Contrary to the title of this posting, good password practices are important. But what's even more important is to:

1.) Keep your wits about you and cultivate a healthy skepticism before downloading a free game, clicking on an advertisement, or buying something from someone you don't know (e.g. eBay).

2.) Keep your operating system and applications updated. Always choose, "Yes, apply updates right now" and "Of course I'll reboot." Manually check for updates periodically just in case.

3.) Use a tool like Revo Uninstaller to remove applications you are no longer using. Especially anything by Adobe, RealPlayer, toolbars (e.g. Yahoo!), and the Microsoft .NET framework.

When Do Passwords Matter?

I got an email today saying that a web application I used a single time eight years ago had suffered a break-in and warning me that if I used that password for multiple accounts, I should change the passwords to all those accounts. I have over 120 personal accounts, and God knows how many at my various jobs over the last 8 years - how many of those applications have been compromised? Kudos to the organization who discovered the break-in AND alerted me. I think it's safe to assume this is not the only break-in among those 120 applications, nor the only one discovered.

Minimum Effort Password Management

I just read a wonderful article in the Boston Globe Online about the time-wasting, annoying, and mostly useless advice security experts have given us about passwords. So if you want to be secure with the minimum amount of effort, what is the most important thing?

I believe using a different unguessable password for every account is the most important password practice because doing so means that all of your other accounts are safe whenever one of them is compromised - and if you use a computer long enough, accounts WILL be compromised. Some have suggested using X9$bFacebook, X9$bTwitter, X9$bMySpace, but schemes that use the application name, even if it's altered in various ways, are still guessable.

To manage different passwords for every account, you need a password manager. Many people use the "remember passwords" feature of their favorite browser. This is a terrible idea because:

It means you are storing your most secure data (your passwords) in your least secure application (your browser)

You are going to need to enter activation keys or passwords into software installed on your machine at some point, and you cannot store that in your browser

When you go to another computer, or try to switch to another brand of browser, you don't have your passwords.

When your hard drive dies, so do all your passwords

When you die, so does access to your computer and all your passwords.

So the web browser is not such a good solution. The best I've found (and thanks to a good friend for pointing me to it) is a free, open-source password manager called KeePass which is available for Windows and KeePassX for Linux, Mac, and Windows. I use it with a strong master-password and a tool called DropBox to synch it across my computers. Lifehacker has an article on how to use them together. Once a year, I recommend printing out your KeePass database, writing your master password on it (your Dropbox password will be in your KeePass database), sealing the list in a tamper-evident security envelope, and putting it in your safe deposit box. Then burn last years list (you know, with a match). When your hard drive dies, you have a backup immediately available on your other computers via Dropbox. When you die, there are legal proceedings for your next of kin to access your safe deposit box.

For a less secure, less robust, but easier to use password manager, look at LastPass.

Grep: Yet Another Reason I Love Bash (the GNU/Linux Born Again Shell)

2010-02-03T06:55:00.001-08:00

I have to say how incredibly awesome it is to be able to issue a one-liner like the following:

egrep -w --color --exclude-dir='.svn' '[Bb]usiness [Uu]nits?|BUs?' WEB-INF/jsp/*

Everywhere the expression Business Unit, business unit, or BU appears in the JSP pages is now listed on my screen in color.

How does it work?

egrep: Run grep (the GNU "find" utility) using Extended (that's where the E comes from) regular expressions.
-w: I was looking up the word boundary expression in the man page for grep and found that the -w switch forces the match to occur within word boundaries.
--color: Show the file names and matches in a different color (red) from the other text.
--exclude-dir='.svn': Prevents grep from looking in subversion (source control) directories
'[Bb]usiness [Uu]nits?|BUs?': Match "business unit" with or without initial caps and with or without an "s" at the end, or match "BU". These matches have to occur on word-boundaries (because of the -w above) so that ABUSE does not match.
WEB-INF/jsp/*: Run the match against all the files in the WEB-INF/jsp directory.

I pay about $150/year for an individual license for InteliJ IDEA because of it's refactoring, but it still can't do this. You can pry bash from my cold, dead hands.

My Favorite Free Windows Applications

2010-01-05T05:50:00.000-08:00

I rely on a small set of free tools (suggested on Lifehacker) to keep my Windows (virtual) machines (and relative's Windows machines) running smoothly. They are:

Revo Uninstaller: To remove programs, but perhaps more importantly, to nuke junk that auto-starts (click Tools, then the traffic-light icon). Does a nice job of removing the extra junk that inconsiderate programs leave behind.

Ccleaner: To clean up trash on the hard drive and registry. I make sure to uncheck cookies for the cleanup in all my browsers. Longer-term, I will probably mark the cookies I care about as safe and let it nuke the rest. Then I run the registry cleanup repeatedly until it doesn't find anything any more.

Microsoft Security Essentials (antivirus): Free, highly rated, low-resource, no hassle. Time will still tell, but so far, so good. I've always thought that Microsoft should be the one responsible for protecting their own operating system.

MyDefrag: The "Weekly" script rocks. I don't run it weekly, but it's amazing what a mess Windows makes of its drive. It's like an animal soiling it's cage. MyDefrag really helps boot times. I'm very impressed. Note: I do not like the "Monthly" script at all. Sorting files in name and directory order seems like a total waste of time and disk-life to me.

Honorable Mention:

sdelete: A command-line utility from Microsoft that zeros out unused disk space. This is nice for security reasons, but I use it to keep my virtual disk images really small. I run "sdelete -c C:" Then in Linux, "VBoxManage modifyhd --compact"

cygwin: A Linux command-line emulator for Windows. I wouldn't live without it, but people wouldn't have a need for it.

The above programs have helped me tame the most clogged-with-crapware systems and make them secure and responsive again.

If someone told me that once I ran Linux, I'd think nothing of running several Microsoft operating systems, I would have laughed. But Windows works great in a VM. It's a fun toy when you don't have to rely on it to do anything useful. I use one VM for IE6 (for testing) and the other for IE8 (for programs that don't have Linux equivalents). Both are running Windows XP SP3. A Windows 7 VM is probably in my future.

What is Linux?

2009-10-30T06:48:00.000-07:00

For those of you who don't know already, Linux is an operating system (like Windows or Mac OS-X), that is built by the public and it's free.* If you know how to use Windows or OS-X, You can install Linux and be using it with no training in about 30 minutes. It's made entirely of programs donated to the public good and many distributions come with office applications, games, and web tools already installed.

There are hundreds of companies, causes, and individuals who package Linux into a "distribution" consisting of the Linux kernel (the guts of the operating system) and a set of software designed for a specific purpose. For example:

- Easy-to use desktop: Ubuntu
- Corporate web-server: RedHat
- Free (GNU) software purity: Debian or gNewSense
- Minimal hardware: Puppy
- Run from a CD: PCLinuxOS
- Repair your computer after a crash: SystemRescueCD
- To be just like another distribution: CentOS (like RedHat)
- To protect networks: IPCop
- Breaking into other computers: PHLAK (Professional Hackers Linux Assault Kit)
- Christianity, bible study, and parental control: Ubuntu Christian
- Online learning environment, suitable for classroom use: Edubuntu

That's a random sampling taken mostly from Distrowatch. There is a lot of crossover between the goals and included packages of various distributions, but this gives you a sense of what's out there. Because the source code for the entire operating system and most of the software generally distributed with it is free and/or open-source, a programmer can compile their own version and alter it to work with custom hardware such as a new phone, or a robot.

Interestingly, no-one told anyone to create Linux. In 1989, Richard Stallman, a visionary programmer who could be called the world's first software activist, created a new kind of software license - the GNU Public License, GPL, or "copyleft. The GPL guarantees everyone's right to copy, share, modify, and (re-)distribute software. In 1991, A programmer named Linus Torvalds started working on a version of Unix that he could use on his own computer. He published it under the GPL and released it on the web and soon many other programmers (or "hackers") contributed changes and improvements. The work of untold numbers of programmers, writers, graphic artists, testers, and other largely self-selected volunteers, under the GPL and other "open source" licenses, has created Linux in all its flavors - and it continues to evolve.

Who uses Linux? We all do. At least half of all web sites run on Linux servers. Mobile phones such as the Android run Linux. 443 of the top 500 supercomputers run Linux. Increasingly, personal desktop computers run Linux. If you aren't familiar with Linux, I encourage you to get yourself a LiveCD of any distribution and give it a try. Ubuntu is currently the most popular for non-techies.

For your entertainment, here's a brief history of Linux through advertising (and one spoof):

Mac Spoof: Upgrading

I'm a Linux

The Heist

* Thanks to DigitalMan for helping me come up with a one-line explanation of Linux.

Settling in to Ubuntu Linux

2009-10-29T11:11:00.000-07:00

I'm surprised at how much I like Ubuntu Linux better than Windows! The UI is basically the same, only everything is faster. Especially when you do anything to a file (write to it, read from it, move it, or delete it), Linux turns hours into Minutes. When you consider that doing anything useful on a computer generally involves changing a file, that says a lot!

Printing to a network printer was actually easier on Linux. Our vendor-supplied proprietary Windows driver hangs the printer every second or third time you try to print a PDF! I've got my complaints (a few sound limitations and a few software packages that I still have to run in a Windows VM). But whenever I have to fix someone's Windows computer I pity them. Even Windows runs better on Linux (in a VM). That way you can keep working while you reboot, reboot, reboot...

As I'm writing this, I can hear my wife downstairs using Windows... "Network Problems!?! I can't have network problems right now! Operation failed???" I feel sorry for her, I've been there myself, but I have to laugh at the background to this blog post.

I encouraged some people I know who are absolute beginners with computers to buy Macs. But now I'm not so sure. Ubuntu Linux is actually easier to navigate in some ways than a Mac.

Update 2010-04-01: I spent 2 hours trying to copy some pictures from a camera to the hard drive on a Mac and it was awful. In Ubuntu, your camera gets mounted as a drive and you can drag and drop the pictures wherever you want them.

The only drawback to running Linux is that some programs only run on Windows. Screen sharing programs (except for TeamViewer) all require Windows or Mac. The latest version of Microsoft Office only runs on Windows, Photoshop only runs on Windows/Mac, and Proprietary VPNs only run on Windows. Most people don't need those things, but I do, so I have to run Windows in a Virtual Machine. But I only have my VM running about 25% of the time.

Windows in a Virtual Machine

2009-10-22T04:27:00.000-07:00

My Windows XP SP3 CD came in the mail yesterday. This time around I did a more-or-less default install in VirtualBox with 20GB expanding drive, 1.2GB memory, 24M video memory, and audio disabled. Instead of Symantec Antivirus, I tried ESET NOD32. I also disabled Adobe Drive this time because I got some weird messages. The install and all setup took about 8 hours, much of which was waiting, which is much more reasonable than the 3 days my previous attempt took.

The new image uses only about 6GB of disk space instead of 15, presumably because it doesn't contain any junk from my old hard drive. It is immeasurably faster than my 140GB image of my previous hard drive, half-of which was unreadable. Not sure how much was VM settings, Symantec, or other junk, but I think everything helped.

I deleted the old virtual hard drive image. It took hours because I had a dozen huge "snapshots" I had taken in VirtualBox, each of which took 1-20 minutes to delete. Before I started, my new 1TB drive was 60% full. Now it's only 7% full! What a disaster that was.

I currently use Windows in VirtualBox a few times a week for the following, but I work mostly in Linux:

Photoshop CS4
I use this a lot and it requires 1024x768 resolution for some of the dialogs to work properly (to show the Save button). Takes a lot of resources, but as much as I love GIMP, it's not a substitute for Photoshop for professional work.: A couple of proprietary VPNs
The nice thing about running these in a virtual machine is that it lets me check email and access the web from my primary OS while the VPN is connected.: Citrix GoToMeeting
Works great: Remote Desktop Connection
Necessary for work.: Internet Explorer
Necessary for work (for testing): Microsoft Office
Open Office is a fantastic substitute. It changes the layout only slightly I think because the available fonts are different on Linux. I still need the real thing for presentations and testing for work.: Firefox HTML Validator plug-in
I really wish there was a Linux version of this!

Hopefully, over time, things like GoToMeeting and the Firefox HTML Validator will work on Linux. I'm still looking for a secure backup strategy for my entire drive.

Alternative for spacer.gif in cross-browser standards-compliant html

2009-09-25T13:22:00.000-07:00

Instead of spacer.gif I use:

CSS:
.spacer2px{font-size:2px;line-height:2px;}

HTML:
<div class="spacer2px"> </div>

Works with <td> as well. We have spacer1px, spacer4px, spacer6px, etc. as needed. Works the same on all browsers, so long as each HTML page starts with the following, which keeps IE from entering quirks mode and makes it as standards compliant as it gets (requires no blank lines until after the opening HTML tag):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">



    ... headers ...



<body>

    <div id="page">



    ... content goes here ...



    </div>

</body>

We currently have two lines of browser-specific code in the entire application, and they are both in the CSS file:

/* This sets defaults for all elements and helps overcome browser differences */

* {

 margin: 0px;

 padding: 0px;

 box-sizing: border-box;

 /* -moz-box-sizing: border-box; Firefox used to need this */

}



/* Add space for scrollbar on IE so it doesn't obscure the page.

This uses the Commented Backslash Hack so only IE can see this: \*/

#page { margin-right: 16px; }

/* End Hack */

Well, that's my recipe that I've been using for 2-3 years now. Tested on IE 6, 7, 8, Firefox 2, 3, Safari 3, 4, Chrome, Opera 9, Epiphany, Konqueror and on Windows XP, Mac OS-X, the iPhone, and Linux (Ubuntu/Xubuntu).

Special thanks to quirksmode.org for probably being the best resource for these kind of details. The O'Reilly CSS: The Definitive Guide and HTML & XHTML: The Definitive Guide books are great for the basic rules.

Moving from Windows XP to Ubuntu Linux this "Weekend"

2009-09-23T08:11:00.000-07:00

Sat, Sep 19, 2009

6 PM

Bought a Western Digital 1TB Caviar (Green) drive and 4GB memory. I wanted to keep the old drive untouched in case there were issues with the install.

10:02 PM

new hard drive installed, Ubuntu install begun. I almost went with Xubuntu, but the display got messed up in VirtualBox when I applied patches. I think Ubuntu is going to be more reliable/compatible and that's even more important to me than performance.

11:53 PM

sent first email from new system using Google Chrome! I spent one of those 2 hours googling and thinking about partitioning before deciding to let the installer do what it wanted. Finally I clicked "OK" about 5 times and in 20 minutes it was done. Ran the updater, rebooted, added and removed some programs; another 20 minutes total. Really, it couldn't have been easier.

Sun, Sep 20, 2009

10:13 AM

Massive frustration. So far, it looks like installing Windows is going to be the hardest part of installing Linux. All I can find is an XP Home edition upgrade CD from 2004.

Microsoft doesn't sell XP any more. The Windows 7 beta download program has ended and you can't buy it until October 22nd (or thereabouts). What you can buy (for $300) is Windows Vista Professional Non-Upgrade with a "free" download of Windows 7 when it's released so you don't have to miss out on upgrading, patching, and being the first to find bugs. Even if that sounded good, I can't buy it now because the Windows store doesn't open until noon. Tech support isn't open either.

I decided to follow the instructions here:
http://www.virtualbox.org/wiki/Migrate_Windows

It looked good, but it's not the easiest, or necessarily the best way. Read on...

4:01 PM

Booting a 150GB image of my old Windows hard drive. Woohoo!

4:06 PM

Windows asks for a key code. I copy it off the sticker on the side of my computer. It doesn't take it. I call in and punch the numbers over the pone. No luck.

It turns out that it wouldn't let me re-register because I have an OEM version that was pre-installed on the drive and the virtual machine had different "hardware" then it was expecting. Maybe I could have faked the bios, the MAC address, the hardware, and I'm not sure what else in order to satisfy XP that I wasn't trying to install it on a different PC. I'm installing it on the same PC, so it's legal, but it's in a virtual machine, so I couldn't be sure the hardware would ever match. At this point I gave up and installed my XP Home Upgrade from 2004 over the XP Pro image because the upgrade has to be installed over an existing version of Windows (apparently it doesn't have to be a validated version).

The install kept the files on the C:\ drive, but it overwrote the My Documents folder and left my old home directory inaccessible. Using CACLs showed it to be owned by WINDOWS NT/my-old-id. I was able to make it readible, but couldn't delete anything. I began deleting my now useless applications but it was taking hours. The solution? Boot the VM into Ubuntu and delete them from Nautilus. It took minutes. How can Microsoft charge $300 for an operating system when a free one has a 100 times faster file system?

8:01 PM

Windows XP home is installed and works. Beginning "Windows Update."

8:29 PM

Installed vpnc, which connects to a Cisco VPN from the Linux command line. Nice! XP is still applying updates.

9:10PM

Windows updates, updates, updates. 50 at once this time.

Mon, Sep 21, 2009

8:24 AM

Windows XP home was finally upgraded to SP3 around midnight last night after many reboots, and lots of clicking and waiting. Installed VirtualBox Guest Additions this morning which was quite easy.

Noonish

Had a work call that wanted to use Citrix GoToMeeting. I accidentally laughed out loud at the suggestion (it was a little akward). But within 5 minutes I had the Citrix client installed on Windows and thanks to VirtualBox GuestAdditions I was looking at a full-screen screen-share without any noticeable lag or any issue whatsoever. Nice!

At some point I called Microsoft to find out my options. This took about 4 calls. One of which was a voice recognition system that couldn't recognize a thing I said. later I played "20 questions" with an automated voice response system that put me on hold for 10 minutes. When I finally got a human, the response was, "If you got your operating system with a computer, call that computer's manufacturer, not us." and "We don't sell Windows XP. Want to buy Vista?"

I called Lenovo and they could only offer a CD of the original drive image for $45, but I would have had the same hardware vs. validation issues in VirtualBox.

Tue, Sep 22, 2009

4:35 PM

I'm basically Done. XP upgraded so SP3, Office 2003 installed and upgraded to SP3, Norton Internet Security installed and upgraded, 2 VPNs installed on Windows. I tried to compress the virtual drive image to no avail. They aren't really designed to have their size changed. There's inaccessible or otherwise useless little bits of junk spread out on the drive image (some are unmovable by a defragment) which probably kept it from compressing. At one point, I ran some program on my XP image that was supposed to fill the drive with zeros (to make the image compress better) by making a big enough text file with each character being the null character. It was supposed to delete the file when finished, but presumably your computer would crash about the same time once the drive was full, but it was running so slowly that I killed it and never got to see.

I thought I'd just write a little script that piped the small file of zeros many times into a new file until the machine crashed. But you can't pipe like that in DOS. I downloaded sdelete instead. It took about 5 hours for it to write 140GB of zeros.

While I was waiting for Windows/Office to install, I installed my development environment and got all my Linux apps up and running.

Mon, Oct 5, 2009

11:27 AM

I've installed Photoshop and Cygwin on Windows in the last week. They work fine, but Photoshop really needs the full-screen 1024x768. I might have to get a real graphics card and bigger monitor.

Had to share encrypted WinZip archives with Windows users. Installed PeaZip and it encrypted/decrypted zipped/unzipped WinZip files perfectly. Everything works so much faster than on Windows. There was definitely pain involved in making the switch, but it's also definitely paying off.

Lessons Learned:

A new hard drive was a great idea. I could pop the old one out of the machine and be sure I wasn't deleting anything important by accident. I wouldn't try an upgrade like this without a blank drive.

A Windows XP Pro SP3 CD would have been a godsend. I still might get one and reinstall to bring the disk image size down to about 10GB instead of almost 150 and to be rid of any lingering junk on the drive.

Dropbox and KeePass make managing online accounts across machines really, really easy.

I did the right thing. I was really unsure about it. But consider:
1. I was able to be up and running on Linux FOR FREE in 2 hours (would have been 1 if I didn't second-guess the installer for an hour). The XP install took 2 days (would have been 1 if I paid $130 for a disk which Amazon, not Microsoft sells).
2. Deleting big files on Windows was incredibly slow. Rebooting into Linux running off a CD (inside the same VirtualBox VM) turned hours into minutes or less.
3. Support: If I actually had a support question (not a pre-sales question), I would have had to pay Microsoft $45 to ask it. The Mac Ads are very funny, but they are accurate in this case. I can get on Freenode IRC for free and have virtually any (polite, pre-researched, well worded) question answered in about 10 minutes, and probably learn something else interesting while I'm there.
4. My whole system is much, much faster. Startup is in about 20 seconds as opposed to 3 minutes. Suspend (sleep) takes about 2 seconds, waking up takes 5-10. The wake-up on Windows often took 5 minutes!
5. In this process I expected to discover that Linux wasn't ready for prime time. Instead I discovered a lot of negative things about Windows and a lot of positive things about Linux!

Conclusion

I'm installing Photoshop now on Windows so a Windows VM is probably here to stay, but I would be very surprised if I ever switched back.

If Google comes out with a killer Netbook operating system this fall, more people will use Linux. If Windows 7 forces developers to rewrite their apps, and if people have to run XP in a virtual machine to use older applications, they can do that just as well on Linux (or Mac) and have a better, faster, more secure host operating system. All this adds up to Windows having a smaller market share and developers having more reason to come out with a OS-X and/or Linux port of their software. If they use a toolkit like GTK+ (free) instead of Microsoft Visual Studio (very expensive), it shouldn't be such a big deal to run on all three operating systems. At least for me, the future looks a lot more like Linux and less like Microsoft.

Free (GPL) Software Restricts Your Rights!

2009-06-05T09:14:00.001-07:00

When I say Free Software in this article, I am referring specifically to software released under the GNU Public License (GPL), a license written by Richard Stallman that has revolutionized the software world. Revolutionized in the sense of completely and indelibly changing it, but also in the sense of starting a movement of people. I owe great thanks to RMS, his copyleft, and the ideas he set in motion. I use emacs and cygwin so much every day that I don't even notice when I dream in emacs. I have created or contributed to a small number of Free or Open-Source Software (F/OSS) projects, but like most professional programmers, I have mostly just helped myself of the bounty of fabulous F/OSS software that exists.

But for all it's talk of freedom, free as in "Free Speech" and free as in "Free Beer", Free (GPL) Software still restricts your rights as a programmer in one very important way that some other Open Source software does not; it cannot be bundled with "non-free" or proprietary software for distribution because the act of doing so turns the proprietary software into free software. Most professionals get around this either by violating the terms of the GPL (intentionally or through ignorance) or by adopting the Software as a Service (SaaS) model of hosting web applications themselves so that the software does not need to be distributed in order to be used. Other strategies for making money with Free Software include selling support and packaging (as RedHat does with Linux) or dual-licensing (as Oracle does with MySQL).

At my current job we pursued the SaaS model as far as it would take us, but in the end we found we had to distribute our product. We were unwilling to license our company's only asset (its software) under the GPL, so we removed all GPL code instead. That was a sad day indeed and it has since been a struggle to keep up with the sheer number of Licenses that F/OSS developers use for their products. For some minor components we have had to choose tools based more on the license than the quality.

The saving grace for us has been the Apache Software Foundation and their license. For F/OSS developers on the fence, software released under the Apache License v2.0 can be converted or combined with GPL 3.0 software at any time, but the reverse is not true.

The trouble with the GPL is a very specific, technical detail which requires some background in order to explain. Modern computer programs tend to be composed of a number of proprietary and/or open-source components "linked" together to form a whole. The definition of "linking" varies somewhat between programming languages and platforms and is somewhat disputed, but it basically means what it sounds like - components are developed separately then combined into a single whole. The trick the GPL plays on programmers is that merely linking to GPL code (without changing it in any way) allows the GPL to extend to all linked code and effectively puts all of it in the public domain. The Free Software Foundation will tell you that it's fine to link GPL code to proprietary code and it is, so long as you have the right and the desire to change that code into GPL code.

I have made my living writing software and charging for it since 1992. I am happy to contribute bug fixes and non-core components of software I have written to the Open Source community, but I want the freedom to distribute critical components under the license of my choosing (generally a proprietary license, granting ownership of the code to my employer). The Apache License (and other Open Source licenses) allow me to do that. The GPL does not.

RMS wrote an article in the latest CACM entitled, "Why 'Open Source' Misses the Point of Free Software." With all due respect to RMS, I have to say I'm glad that it does and I'd like to encourage anyone familiarizing themselves with F/OSS to read several licenses very carefully before choosing one. The GPL is a classic, but I'm afraid history will show its application is limited to two scenarios:

People who believe that proprietary software is fundamentally wrong and should be eliminated.
Companies who write software which is much more useful when linked as a component of other software than it is as a stand-alone product. These companies will pursue a strategy like MySQL where they dual-license their products under the GPL so that the world is free to try, examine, change, and distribute their software for non-commercial purposes. But those who wish to use the software commercially will have to pay for another version of what might be the same code released under a different license.

I predict that the GPL will see the most usage by the second camp. For the rest of us, there are a few hundred other F/OSS licenses to choose from. A great source for learning about these licenses is the Open Source Initiative. Their approval process is very instructive for anyone thinking they want to write their own license. The proliferation of essentially duplicate licenses has run increasingly rampant as more programmers have gotten excited about F/OSS, mostly through the writer's ignorance of an existing license that already does exactly what they want.

Using @Deprecated more liberally

2009-03-09T10:13:00.000-07:00

I recently created a class and an interface for managing dates as just a month and year (ignoring the day) and called them YearMonth and YearMonthInterface. I then added comparison methods to YearMonth: greaterThan(YearMonthInterface x), lessThan(YearMonthInterface x), and equals(YearMonthInterface x). Do you see the problem?

equals() overrides Object.equals() and has a special significance for the Java Collections framework. According to the Java SE 6 spec, equals() must be reflexive, symmetric, transitive, and consistent, and x.equals(null) should return false. I had no way of knowing that any class that implemented the YearMonthInterface could satisfy those conditions. In fact, I already had one implementing class that could never satisfy symmetry: myYearMonth.equals(myOtherClass) would often not equal myOtherClass.equals(myYearMonth) because my other class was dependent on another object and could only be equal when its associated objects were also equal.

I decided to keep the YearMonth.equals(Object o) method as overriding Object.equals() so that YearMonth would work correctly with collections, but create a new method, YearMonth.equalTo(YearMonthInterface x), for manual comparisons. My concern was that I never wanted to mistakenly use equals() instead of equalTo(). The solution? Deprecate YearMonth.equals() and provide a comment suggesting the use of equalTo() instead. The compiler will now catch any attempt to use the wrong method and the collections framework can still use the equals() method without problem.

/**

This is deprecated because it does not compare YearMonthInterface objects,

rather it overrides Object's equals method.  It still works in

collections, but there should be no reason to ever call it directly.

Use equalTo() instead which really compares two YearMonthInterfaces as

you would expect, but doesn't need to be symmetric.



@param other another YearMonth (or it will return false).

@return true of the YearMonths are equal.

*/

@Deprecated

@Override

public boolean equals(Object other) {

   // Check address and null first for speedy return

   if (this == other) {

       return true;

   }

   if (other == null) {

        return false;

   }

   if ( !(other instanceof YearMonth) ) {

       return false;

   }

   final YearMonth that = (YearMonth) other;

   return (this.year == that.year) && (this.month == that.month);

}



public boolean equalTo(YearMonthInterface that) {

   // Check address and null first for speedy return

   if (this == that) {

       return true;

   }

   if (that == null) {

        return false;

   }

   return (this.year == that.getYear()) && (this.month == that.getMonth());

}

This technique would not work if I were to release this class to the public, but for a utility class with limited application, it can prevent me and my coworkers from making a simple but costly mistake.

For a full explanation of how to write a good equals() method, see Effective Java Second Edition by Joshua Bloch Chapter 3 (pp 35-36 is specifically about symmetry). I'm about half way through this book and really enjoying it.

New Favorite Find/Replace (bash) shell commands

2009-02-03T08:38:00.000-08:00

These commands took a while for me to find and I now use them very, very often. One is to find files in all subdirectories excluding certain directories (in this case, the subversion ".svn" directories). The second expression replaces regular expressions in all the files that match the find in the first expression:

# To grep for things excluding subversion .svn directories

find . -path '*/.svn' -prune -o -type f -print | xargs -e fgrep -I PATTERN



# To replace the same (using sed), only in files that grep matched, only in non .svn files.

find . -path '*/.svn' -prune -o -type f -print | xargs -e fgrep -l -I 'OLD' | xargs sed -i -e 's/OLD/NEW/g'

UPDATE 2009-02-09
I was browsing at Barnes and Noble over the weekend and the O'Reilly Pocket Guide to Grep mentioned an --exclude-dir option. I looked at the man page again and it's there too. I even googled again and found it right away. How did I miss it? That makes things much easier.

# To grep for things excluding subversion .svn directories

fgrep --exclude-dir='.svn' -rI PATTERN *



# To replace the same (using sed), only in files that grep matched, only in non .svn files.

fgrep --exclude-dir='.svn' -rIl 'OLD' * | xargs sed -i -e 's/OLD/NEW/g'

UPDATE 2010-03-31

# To find whole words, using an even briefer syntax for

# chaining the output of grep into the input for sed.

# Taken from a shell script that sets oldWord and

# newWord, but you get the idea:

sed -i -e "s/\<$oldWord\>/$newWord/g" $(fgrep --exclude-dir='.svn' -wrIl "$oldWord" *)

Hibernate automatic POJO and mapping file generation from MySQL

2008-07-30T07:54:00.000-07:00

Let's face it. Updating Hibernate .hbm.xml and .java files for every database change is phenomenally boring and a waste of time. It's the sort of thing that makes people switch to Ruby/ActiveRecord. I have recently undergone an effort to get the Hibernate reverse engineering tools to generate .hbm.xml and .java files as close as possible to what I want. In theory that should allow me to add new methods to the files in my source tree and still generally get hands-free updates by running the hibernate updater and diffing/patching. In practice, getting Hibernate to give me exactly what I want is very hard, so in the first week I moved from 3% of my changes being automatically applied to 80%. In the next month, I moved up to about 95% and it will probably slowly continue to improve.

The basic outline of the process is simple:

Save the generated files from the last time I ran the generator: mv generatedSchema generatedSchema.1

run the generator again and diff the files it generates this time against the ones it generated previously

patch those changes to my source tree.

I use the ant task, not the Eclipse integration because I'm in love with the command line. Assuming that you have read and absorbed all the documentation about hibernate reverse-engineering, I'll list a few key points from my hibernate.reveng.xml.

hibernate.reveng.xml

For some reason BIGINT was being reverse-engineered as Long (the object) instead of long (the primitive). I'm not so sure whether my VARCHAR mapping is a good idea...

I included an example of renaming a column, and a much more useful example of naming a parent-child relationship. The syntax of the foreign-key mapping was not obvious to me, so I hope you find this example helpful.

<hibernate-reverse-engineering>

 <type-mapping>

  <sql-type jdbc-type="BIGINT" hibernate-type="long" not-null="true"></sql-type>

  <sql-type jdbc-type="VARCHAR" length="1" hibernate-type="char" not-null="true"></sql-type>

 </type-mapping>

 <table-filter match-catalog="mydb" match-name="temp_user" exclude="true"/>

 <table catalog="mydb" name="company"> 

  <column name="identifier_c" property="coIdentifierC" type="string" />

 </table>

 <table catalog="mydb" name="tree"> 

  <foreign-key constraint-name="tree_ibfk_parent">

   <many-to-one property="parent" />

   <set property="children" />

  </foreign-key>

 </table>

</hibernate-reverse-engineering>

pojo/*.ftl Files

In addition to these customizations I've made extensive use of the Freemarker POJO templates. I made a number of very minor changes for personal preference, but one very important change for diff and patch to work. I added a two-line comment before and after the fields (in PojoFields.ftl) and the get/sets (in PojoPropertyAccessors.ftl):

// Begin Auto-generated Accessors (by myGenerator project)

// Manual editing is STRONGLY DISCOURAGED below this point!



...



// End Auto-generated Accessors (by myGenerator project)

// Manual editing is OK below this point

Having two lines of comments is very important because:

Hibernate reverse-engineering tools like to shuffle the field order even when you changed one of the related tables

Diff uses the previous 2-3 lines for comparison to confirm it's found the right place in the file and a two-line comment plus one blank line makes it very happy.

diff and patch

After running the hibernate ant task (and tweaking the output with some canned perl regular expression replacements), I run diff (telling it to ignore the timestamped Hibernate comment):

diff -I '^\(//\|<!--\) Generated ' -druN generatedSchema.1/ generatedSchema/ >patch.txt

Often I check what changes were made:

less patch.txt

From my source tree, I do a dry-run of patching the differences.

cd WEB-INF/src

patch --dry-run -l -p1 <../../../../../scorecardSchemaGen/patch.txt

If there are any failures:

patch those files manually performing any cleanup necessary to make patches more likely to succeed automatically next time

copy the new generated version of the manually patched file from generatedSchema to generatedSchema.1

Re-run the diff command so that it will find no differences in the generated version(s) of the manually patched file(s).

Perform another dry-run patch

Repeat until there are no errors in the dry-run

Now there will be no failures when you patch for real:

cd WEB-INF/src

patch -l -p1 <../../../../../scorecardSchemaGen/patch.txt

Finally, clean up the backup files made by patch:

rm company/project/section/*.orig

MySQL Conventions

When choosing optimal types and variable names for the columns in MySQL, one should take into account how Hibernate will reverse-engineer it so as to make sure the resulting files are as close as possible to what you want. In addition, MySQL has no "check" function and the MySQL BOOLEAN becomes TINYINT(1) which allows values from -128 to 127. Here are the subset of data types that are really easy to use with all tools and provide some data integrity by eliminating possibilities for meaningless values.

Boolean: Start each name with is_ whenever it makes sense - the code reads more like English that way. Use positive column/variable names. E.g. instead of preventUpdate, isNotDirty, or doesntHaveErrors, use allowUpdate, isClean, or isErrorFree. Declare in MySQL as:
```
is_something bit(1) NOT NULL DEFAULT FALSE
```
Note: Java/Hibernate like this but PHP and some other tools may not. Here are two work-arounds for other tools:
```
select if(is_something, 1, 0) from my_table;
```
or
```
select if(is_something, true, false) from my_table;
```

Integers:Try to only use MySQL (Java) types: INT (int), BIGINT (long), and maybe TINYINT (byte). The size (the number in parenthesis after the type in MySQL) has no effect that I can see. The SMALLINT (short) data type requires a trade-off between saving space and limiting possible values and ease of use in Java. It requires too much type casting for my taste when working with it in Java. You even have to use casts for situations like:
```
i = (short) (i + 1);

myObject.setShort( (short) 0 );
```
Unless you require a smaller data type for data integrity or for space considerations, It's easiest to stick with INT. Stay away from MySQL's MEDIUMINT: It's a 3-byte integer which is probably why Hibernate does not recognize it. 64-bit (8-byte) integers like BIGINT (long) didn't used to even be available on hardware. The manuals don't make any note of whether they are supported by hardware or not. Anyone know?

Strings: Use VARCHAR with the appropriate length. That way, it's easier to support utf-8 or other variable-width character encodings if you want to.

Naming: I started off naming my columns description_c and year_i, but ended up dropping the type-specific suffix because Java and MySQL are both type-safe anyway and the extra letter made the code slightly harder to read. I still use the word _date for dates because that seems to make the code easier to read (to my eye).

Foreign Keys: MySQL uses a separate constraint name and foreign key name. The constraint name has to be unique within the database so it's automatically assigned the tablename_ibfk_1 where the "1" is a sequence number incremented for each new constraint. If I drop and recreate a table in development, but merely add a new column in test, then the sequence numbers could be different in different environments and issuing a command to change a constraint in one environment will change a *different* constraint in another environment. This would be really bad.

To avoid this, I use the following template to declare a foreign key with MySQL InnoDB (the default database, MYISAM ignores the "on-delete cascade" clause):
```
CREATE TABLE process ( 

    ... 

    company_id BIGINT UNSIGNED NOT NULL, 

    constraint `process_fk_company` FOREIGN KEY (company_id) 

        REFERENCES company(id) on delete cascade, 

    ... 

) ENGINE=InnoDB;
```
That way I'm sure that the constraint name (which is what I have to reference in the Hibernate file) will not change. It's confusing because MySQL syntax mixes foreign key column names and constraint names. E.g. DROP FOREIGN KEY constraint-name. To find what constraint name is used for a given foreign key, use the SHOW CREATE TABLE command. Constraint names in mysql look like `tablename_ibfk_1`

RESTful Form Beans with Struts 1

2008-03-22T07:37:00.000-07:00

What Struts is Good At

Struts works great for view-only pages, particularly when you need to build those pages from the database. But there are a number of subtle issues with update pages which can probably best be solved with a RESTful solution.

REST and state

The database holds persistent state. My understanding is that one of the concepts of REST is that there is rarely a reason to cache that state on the server and/or the browser/client when it's already stored semi-permanently in the database. It seems that the session on the server only needs to hold state that doesn't belong in the database. That would be data specific to what the user is currently viewing or doing. Some examples are:

Logging in - that is usually (and probably best) accomplished by establishing a session on the server and storing a cookie on the client containing a token that the client's browser sends with every request instead of resending their user-id and password. The token tells the server which session is associated with that user. That way you aren't repeatedly sending the user ID and password over the wire. If someone eavesdrops on the session and steals the token, it expires after a certain period so that an attacker will (theoretically) not be able to log back in.

Other session-specific state. This is for global-controls analogous to the caps-lock key on the keyboard. All the other controls (keys) are still available to you, but their behavior is slightly different because the state (caps-lock) has changed.

Session Timeout

Sessions need to time out eventually (usually in 30 minutes or so for security reasons). End-users find this incredibly annoying. A little JavaScript can soften the blow:

setTimeout(alert('Your session will expire (silently) in 2 minutes...'), 1680000);

This is still annoying, and not very secure, since the point of having sessions expire is to prevent an unauthorized person from accessing a still-valid session. To a hacker, this is essentially a pop-up, advertising: "You can hijack someone's session if you act now!".

If you're application is mostly RESTful, storing only the login information on the server, you can pop up a login window to allow them to log-back in without loosing whatever they are working on. This is secure because the user ID and password are required. Also, the user does not have to respond within a certain period because they are effectively logging in and creating a new session.

if (confirm('Your session has expired.  Log back in to continue your work?')) {

    loginWin = window.open("pre_login.do");

}

Since you do not rely on server side state or cookies for anything but the login, the user can get a form and fill it out in one session, then submit it in a completely different session, so the login pop-up works. If you have a little bit of state on the server and you are clever, you can enhance this solution to send a setting or two from your server-side state from the old session to the login page so that the new session will match the old.

Struts Problem 1: Clearing session-scoped Form Beans

Form beans are session scoped by default. A user GETting the form in preparation for POSTing a new record will see whatever data was left over in the form bean from the last record they edited. You need to work around this by figuring out when the user is going to add a new record and clearing the fields in the form.

Struts Problem 2: Session expiration with session-scoped form beans

Sooner or later the session will time out and the form bean will be completely lost. Writing an Action and pairing it with a log-back-in JavaScript function becomes prohibitively ugly from a maintenance and security standpoint if even a few different fields are required for a few different screens. As far as I can see, there is no reasonable work-around for this scenario, which is why REST is so useful.

Struts Problem 3: Request-scoped Form Beans and the Struts validator

When there is a failed validation on a form bean, the model isn't invoked, so only the fields that the HTML form posted are filled in from the request. Any extra data (breadcrumbs, related records from the database, or other variable text that provides context for the update taking place) is lost unless that data was also built into fields in the form <input type="hidden"... />. But REST is supposed to make things simpler and more lightweight. Building breadcrumbs into URLs requires escaping any special characters, increases both the request and response sizes, and is generally a very ugly solution. Storing these fields in client side cookies is possible too, but very complicated.

A RESTful solution

REST precludes any unnecessary server state and as mentioned above, the validator is not RESTful. So you'll need to declare actions:

<action ... validate="false" scope="request" />

All validation must take place on the action, not the form bean (see Problem 3, above).

Query the database for all the data to build the screen with each request. Any extra load on the database is small because this amounts to making extra queries only for failed validations. The catch is that you need to keep track of whether any data on the form bean is dirty (user has changed it) or clean. Remember, a user can clear a field, so just checking for null or 0 doesn't mean that the data is unchanged.

Fortunately, we already know whether the form data is dirty or not by the method the user used on the request.

GET without a database record number: Clean: Show the user a blank form so that they can add (POST) a new record.
GET with a database record number: Clean: Show the user the current values of the record from the database so that they can update (PUT) the existing record.
PUT: Dirty. If input passes validation, update existing record with changes. If it doesn't pass, return the dirty form plus any "extra" data from the DB for building the screen.
POST: Dirty. If input passes validation, check that the record doesn't exist already as the user can resubmit a post by pressing the back button. This resubmitted post should either be ignored or treated as a put (see above) since the record already exists. If it is a valid POST add a new record to the database. If the input is invalid, return the dirty form plus any "extra" data from the DB for building the screen.

Note: HTML only supports GET (all links) and POST (only in <form method="post"...>). Not surprisingly, most browsers don't support the other methods mentioned in the HTTP spec either. I'm sure people are signing petitions calling for an overhaul of HTML and all browsers, but it's really not an issue to send an extra parameter to designate the "method" of each request. Your application may require some methods that HTTP does not provide, in which case you'd have to send a "method" parameter anyway.

In order for the extra data from the DB to be refreshed on a request-scoped form, the action still needs to be invoked. Something like the following would probably be a good template for Actions:

Check that the user is logged in. This might better be done in a servlet, but it needs to be done.

Check for required parameters (like the unique identifier of the record you are editing). Send failures to a Not Authorized page.

Check that the user is authorized to do what they are trying to do. Send failures to a Not Authorized page.

Validate any other parameters that would make the request just invalid - essentially assertions that your UI is working properly go here. Send failures to a Not Authorized page.

Create your ActionMessages object to hold user-friendly validation errors

User input validation. Accumulate friendly error messages in your errors object.

If there are no errors and the method is POST or PUT, perform update.

If the form is not dirty - fill form fields with data from database. Build or rebuild other data for the screen to show the user and put it on the form bean. Or you might create a new form bean and forward to a different screen after a successful update.

That list is a starting point, but I always code Actions to minimize the number and extent of database queries, so that sometimes shuffles the later stages in that list.

Advantages:

You don't have to worry about clearing old data from the form bean (you get a new form bean with each request).

You don't need to encode any "extra" data (breadcrumbs, etc.) into the form to make it get included in the request (it's reread from the database).

You can log back in elegantly, without loosing what you were working on (all necessary data is reread from the database or carried as form data in the request).

You can use any objects you want for building the screen - you are not limited to Strings and other primitive types.

This requires less code than any of the other ways I tried solving these issues.

Disadvantages

You can't use the Struts validator. All your validation must be done in the action.

Conclusion

So far, this is the neatest solution I can come up with to all of the above issues. And it can be as RESTful as the design of your application allows (uses a minimum of server-side state). It fails to leverage the struts validator, but I was never eager to perform that much XML configuration anyway. Struts 1 has been around for a long time. People ask much more of a web application now than they used to. Maybe Struts 2 has some easier shortcuts for these same issues?

Converting a Web Application to UTF-8

2008-01-25T12:17:00.000-08:00

Overview:

Java-Web-Application Character Encoding Overview - this is a great overview. It answers the questions, "why bother" and "what's involved".
UTF-8 and Unicode FAQ for Unix/Linux - A more detailed overview of Unicode in general.
The Unicode Standard: Conformance (PDF) - See section 3.9 Unicode Encoding Forms. Table 3-5 explains UTF-8 encoding at a binary level, which is really where you have to go to understand it. Table 3-4 is UTF-16.
UTF-8 Sampler a great collection of text in many different languages.
Phonecian (Range 10900-1091F) (PDF) - Some high sample-characters

MySQL

MySQL supports only characters from the Basic Multilingual Plane (BMP or Plane 0) which is U+0000 - U+FFFF. The Plane 0 characters all take 2-bytes in UCS-2 or 1-3 bytes in UTF-8.

Source: MySQL 5.0 Reference Manual 8.1.8. Unicode Support.

IMPORTANT: MySQL's default collation for each characterset (not just utf8) ends with a "_ci" which stands for Case Insensitive. This is probably just what you want for everything *except* passwords digests (I'm assuming no-one stores plaintext passwords in databases any more). If a 43 character Base64 digest were stored in a case-insensitive column, it would be roughly 5 billion times more vulnerable to hash collisions. That's bad. Using utf8 for user IDs increases security by allowing tens of thousands of characters in the user ID instead of the mere 100 or so safe ASCII characters. Decide whether you want johnDoe and JohnDoe to log into the same account and choose your collation for the user ID column accordingly.

As of this writing, the only case-sensitive utf8 collation is utf8_bin. To find out what collations your installation supports issue:

show collation like 'utf8%';

Don't use utf8_bin for fields that you will sort for the user because letters A-Z come before a-z in this collation, but it is the only collation to use for utf8 case sensitivity. Specify character set and collation explicitly on your password hash column (and optionally the user ID column) so that you are free to change the character set of the database and tables in the future without accidentally locking any users out of your system:

user_identifier varchar(30) character set 'utf8' not null;

password_hash varchar(44) character set 'utf8' collate 'utf8_bin' not null;

More information on character sets and collations is available on the Character Sets and Collations in General page of the MySQL manual.

When you change the character set of a table in MySQL, every column is explicitly set to take the *old* character set - even if the columns in your tables all previously used whatever the default was! You will need to issue ALTER TABLE statements for every column, or dump the database to file and replace all the character set references. I did the latter with three text replacements:

mysqldump -p -u root --single-transaction my_db > ~/backup_$(date --iso-8601).sql

Change table-specific character sets (always present)
Replace:" DEFAULT CHARSET=latin1"
With:" DEFAULT CHARSET=utf8"

Remove any column-specific character sets (less common)
Replace:" character set latin1"
With:""

Change all char's to varchar's because char has to store 3 bytes/character in utf8 while varchar can store 1 for short characters
Replace:" char("
With:" varchar("

You might want to search for "latin1" and see if anything else needs to be changed. Then follow the instructions above and everything will be fine. Hopefully this will save someone else a day's worth of trouble.

Change configuration settings in the system-wide /etc/mysql/my.cnf file (or .my.cnf or my.ini file on Windows):

[client]

default-character-set=utf8

[mysql]

default-character-set=utf8

[mysqld]

default-character-set=utf8

Restart MySQL.

Login to MySQL and use the database in question. Enter the following:

alter database db_name character set utf8;

Test the settings:

mysql>show variables;



character_set_client            | utf8

character_set_connection        | utf8

character_set_database          | utf8

character_set_filesystem        | binary

character_set_results           | utf8

character_set_server            | utf8

character_set_system            | utf8

character_sets_dir              | C:\Program File

collation_connection            | utf8_general_ci

collation_database              | utf8_general_ci

collation_server                | utf8_general_ci

Log out and re-load the database:
mysql -p -u root my_db <~/backup_$(date --iso-8601).sql

Cygwin

UTF-8 Cygwin seems to add support for Japanese characters, but I can't tell if it supports UTF-8 up to U-FFFF or higher. Maybe I just don't have the fonts on my system to see it!

Log4J

log4j.appender.appenderName.encoding=UTF-8

Source: a comment in email fromDon Dwoske

Hibernate:

In hibernate.cfg.xml, sethibernate.connection.url to:
jdbc:mysql://host:port/db?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8

Source:More UTF-8 head-thumping with Hibernate 3

Struts:

http://ianpurton.com/struts-utf-8-and-form-submissions/

JSP:

Add the following attributes to the<%@page %> tag:
pageEncoding="UTF-8" contentType="text/html; charset=UTF-8"

Source:Tomcat/UTF-8

Tomcat:

In$CATALINA_HOME/conf/server.xml add the following attribute to the<Connector> tag:
URIEncoding="UTF-8"

Source:Tomcat/UTF-8

Java

In your Java code you need to replace:
s.getBytes()
with:
s.getBytes("UTF-8")

Source: a comment in email fromDon Dwoske

IMPORTANT: You *especially* need to convert bytes that are being sent to your SHA message digest algorithm because otherwise any non-latin characters will come out as "invalid", and they will all have the same hash! Perform the UTF-8 conversion above, and you will have no problems. All the ASCII characters keep their old encoding, so existing users' passwords still work. Not sure what happens to high ASCII characters. If that's an issue for you, you'll have to check.

Java seems to support the 16 Supplimentary Planes that take an extra 16-bit character to represent.
Source:JSR 204: Unicode Supplementary Character Support

MySQL only supports the BMP (Basic Multilingual Plane or "Plane 0") so I'm going to have to intercept these "high" UTF-8 characters in Java and put the replacement character in the database instead:

safeString = inString.replaceAll("[^\u0000-\uFFFF]", "\uFFFD");

Note: U+FFFD is � - the Replacement Character "used to represent an incoming character whose value is unknown or unrepresentable in Unicode". Source:Specials Range FFF0-FFFF (PDF). This shows up as a blank box in Internet Explorer (it's supposed to be a white question-mark in a black diamond). I'm sure IE will support it eventually.

I made a routine to clean all my string inputs from the user:

private static final Pattern highUnicodePattern = Pattern.compile("[^\u0000-\uFFFF]");



public static String cleanInputStr(String inStr) {

    if (inStr == null) {

        return null;

    }

    String trimmedStr = inStr.trim();

    if (trimmedStr.length()< 1) {

  return null;

 }



 Matcher highUnicodeMatcher = highUnicodePattern.matcher(trimmedStr);

 return highUnicodeMatcher.replaceAll("\uFFFD");

}

As just shown, I filter any characters above U+FFFF, so I don't have to worry about characters that take more than 2 bytes of storage. I know what you are saying: "But, U+FFFF takes THREE bytes to store in UTF-8, wouldn't it take more than 2 bytes in UTF-16?" Maybe, but Java stores it as 2 bytes, so the Java String.length() still works. If you support characters above U+FFFF, you'll need to replace all your string-length checks:

int charCount = testString.length();
with this:

int oldCount = testString.length();

int charCount = testString.codePointCount(0, oldCount);

Source:Strings - Core Java Technologies Technical Tips

Notes:

There are also some characters that are affected by the neighboring characters (accents and such). I'm not thoroughly convinced that I can count characters by counting the number of times I press the arrow keys.

Even with the Java character-counting hack above, I thought for a while that JavaScript in FireFox and Safari counted more characters than Java in certain circumstances. Then I discovered that I had line-feeds in my test data, and they were being counted by Java, but not by JavaScript (probably because they wouldn't render in the HTML) - so this "problem" wasn't related to UTF-8 after all!

Emacs

On Windows, GNU Emacs supports all the Unicode characters I could test on it of the box and XEmacs doesn't support unicode at all. For the past 8 years I've been using XEmacs because that's what I started with and I never saw a significant enough difference between GNU and X to bother switching. Ladies and Gentlemen, I am making the switch!

Note: Once you get your .emacs file switched over, make a backup or be careful not to open XEmacs, because it will *nuke* your old .emacs file and leave a new version converted to XEmacs in its place! That alone is so annoying that I don't ever want to switch back.

describe-current-coding-system is a handy function to see what emacs thinks it's doing! Emacs UTF-8 support is not 100% at the time of this writing (4-byte characters sometimes flicker and change), but it does pretty well. The GNU Emacs site says that even better UTF-8 support is planned for the next release.

Fixed-width Windows Fonts

I downloaded Everson Mono Unicode and it really supports every character, but I found it unacceptably hard to read the English/code in Emacs when I made it small enough for coding.

Courier New and Lucida Console seem the best of the ones that come with Windows XP English. They are both missing big chunks of UTF-8, but they have better support than any of the others I found (at least they show some Greek, Russian, Turkish, Arabic, Hebrew, Chinese, Japanese, and Korean). I picked Courier to use with Emacs (does anything else really matter?) because it's smaller and seems a little easier to read on the screen to me. Maybe I'm just used to it.

Humor

From xkcd: Emoticon (read the mouse-over)

Make Users Happy by Ignoring Requirements

2007-12-12T08:35:00.000-08:00

People talk about the system they want you to build in the language of what they have now. It is natural to not want to give anything up. Writing web applications, I hear over and over that people want to be able to sort by every column, have column headings print out on every page, filter by any value... That's Microsoft Excel - and it is a truly incredible tool for doing all of those things.

On the other hand, most business systems (as opposed to software systems) are dependent on communication to make people with different skills work together. In general, if you can meet those needs, you can always add a CSV download later if people still want to "slice and dice" data in Excel. I have had consistently good results from ignoring the "Excel on the Web" requirements and asking my clients instead to describe the people, the tasks, and the communication involved in their processes.

I try to break a process into the sets of tasks that each person (or type of user) needs to perform. The usual goal is to show someone only the data they really need to perform their task(s), and give them the tools they need to accomplish that task on the same screen. Sometimes, similar tasks can share a screen or multiple screens might be required for a task, but the goal is to compartmentalize and specialize the business system into units that use common sets of data and functionality.

Example 1: Ignoring User Requirements

Below are the requirements I was given for a year-long project:

A list of the documents
Columns showing which one's are ready, which ones are late, and who is responsible for each
Sort by any column
Filter by any column
Print out with column-headings on the top of every page...

Sounds like Excel, no? I only actually met requirement #1, yet the client was delighted. Why? Because the list of Excel features I was given tipped me off that the requirements were not well thought-through.

Here is a list of real requirements that a team of us had to dig for:

A draft document needs to be written by an analyst
The document must be reviewed and approved
Financial reports need to be produced by a system
Financial reports need to be approved by a person
The documents and financials need to be combined into various client reports
The client reports need to be printed
The client reports need to be mailed
A manager needs to keep track of all of the above

Developers understand systems and the benefits and limitations of your technology. Businesspeople know what they need to accomplish, but are often dimly aware of the business systems they use to achieve their goals. Bridging that communication gap is the hardest part of writing good software.

Example 2: Ignoring Systems Requirements

Stated requirements for a reporting web site that I worked on (on-and-off) for 6 years:

high availability
Load balancing
portal
pluggable, snappable, replaceable software components
Thin, rich client
Fully denormalized reusable data source
Dashboarding
3-tiered communication layer

Today, you could add to those requirements:

stateless (REST-ful)
SOA (Service-oriented architecture)
AJAX
Web 2.0 (or 3.0, or whatever)

What the business actually needed was:

Show quarterly reports as soon as the data was approved to show
Secure
Support a maximum of 20 simultaneous users
Available 8AM-8PM Eastern (US) time
Show some other marketing information

One web server would have been more than adequate, but we had 2. The slow part was actually some of the queries/reports in the database. The only part of the site we ever plugged/snapped/reused were some hand-coded HTML pages and the login code. If we had known enough to discover the actual needs of the users, we could have saved at least 50% of the effort and used some of that time to further optimize the long-running queries, or even redesign the database to make the queries simple enough that they wouldn't need optimization.

Well, 20/20 hindsight is easy. That's how you learn...

Conclusion

When requirements read like a winning "Buzzword Bingo" card, it's almost a sure sign they weren't thought out very carefully. Time spent digging for real requirements pays off in both medium and long-term system usefulness and cost savings.

On Generating Random Keys for Use in Cryptography

2007-09-30T18:45:00.000-07:00

Computers are deterministic state machines - totally incapable of producing anything random. They employ Pseudo-Random Number Generators: functions that appear to produce random output with respect to their input. Good Pseudo-Random Number Generators (PRNGs) can produce thousands of values a second with a surprising degree of entropy. But even the best PRNG is limited by the variability of its “seed” or input values, and none have perfectly even distribution or an absence of patterns in their output.

Bits of Entropy:
In cryptography, key size, measured in bits (or powers of 2), represents the number of keys which must be tested to break an encrypted message by brute force. But encryption algorithms are generally “lossy” because, being deterministic, they have some degree of predictability and/or collisions of different keys producing the same output values. Some algorithms are “lossier” than others and there are many statements in the literature to the effect that a 1024-bit key using one algorithm is just as secure as an 80-bit key using another. That means one algorithm is 1.49×10²⁴⁸ times more secure than the other. Despite common misuse in advertising claims, bits are the "standard" unit of entropy. In this writing, they represent a mathematically derived number of equally likely possible combinations, expressed in powers of 2. Not the computer space used to store a value generated with a PRNG.

Dirty Secrets of Pseudo-Random Number Generators:
Given the same initial input, a PRNG will not only produce the same output value, but will produce the same sequence of values every time, and that sequence is generally your automatically generated cryptographic key or password! The "moving parts" inside most PRNGs are 64-bit and 32-bit registers and this limits their entropy. But PRNGs are often more limited by the precision of their input or seed. Since the few somewhat random processes in the computer tend to produce clusters of values, they are unsuitable as random number seeds. The only part of the computer that produces an even dispersion of values is the system clock, which measures milliseconds.

There are only 8.64×10⁷ milliseconds in a day. That's about 26 bits of entropy. Using the 95 characters on a US-keyboard, a 5-character truly-random password is more secure! So why not use a longer time frame? There are 3.1×10¹¹ milliseconds in 10 years which is only 38 bits of entropy – and who has a password older than 10 years? Most people generate their keys at work, between the hours of 9-5 M-F which is only one fourth of the hours in the week, so you might be losing about 2 bits of entropy there.

So even if you have a 64-bit random number generator, unless you can get a seed with more than 36 bits of entropy, you only get 36 bits of entropy in your key. That’s the same number of significant digits in a truly random 6-character key (95⁶). How many significant digits of randomness does any computer generated encryption key contain? No more than the significant digits of entropy used to create it times the number of PRNG algorithms that could have been used.

Links:

How We Learned to Cheat at Online Poker: A Study in Software Security - If you liked my article here, you will probably like this article even more. One of my favorite reads on the web!

The Memorability and Security of Passwords - Some Empirical Results - Using initial letters and punctuation from pass-phrases is as easy to remember as the most commonly hacked English words, but is as hard to crack as truly random characters... until cracking tools learn what to look for. In the meantime, this is the best advice I've seen on telling an end-user how to construct a secure password.

Entropy Gathering Daemon - A great way to seed a random-number generator. The only complaint I've heard is that EGD can block, waiting for more entropy if you use it heavily in software such as a web server.

KeePass password store - has an entropy-gatherer built-in for generating random passwords... Nice!

RANDU Wikipedia article about an infamous random number generator.

Humor from xkcd: Random Number, Code Talkers